Johannesburg, 07 Nov 2019
The South African Mint Company (South African Mint) is an organisation responsible for minting South African currency on behalf of its owner, the South African Reserve Bank (SARB). It is the only export mint in Africa with over 100 years’ experience producing all South African coins as well as other African countries.
Data is critical to the everyday running of the South African Mint business, with its customer experience and technology key areas in addressing business objectives. When the South African Mint started to run out of capacity on its existing storage and replication started to fail intermittently, a request for proposal (RFP) was issued to address these challenges.
After First Technology National conducted an in-depth assessment of the environment, it became clear that the RFP requirements would not resolve the issues. As a result, First Technology National responded with an innovative proposal that included a federated cluster design that would revolutionise the storage architecture at the South African Mint.
The challenge
“The existing storage environment at the South African Mint had no room for growth as additional storage and servers could not be added. Data and system growth had caused stress on the virtual environment, resulting in increased risk of host failure and critical applications experiencing downtime. In addition, the hosts had far more servers than the storage architecture originally planned for,” explains Bianca Ackerman, Senior System Administrator at the Mint.
Drives began to report faults, including insufficient space and disc errors, which ran the risk of total data loss on these failing drives. Backups were also being negatively affected due to lack of available capacity. While a second storage server had been implemented with full replication set-up between server rooms, auto-failure between the two was not fully implemented. It was clear that a comprehensive solution was needed in order to resolve these challenges before it could cause further harmful impact.
The request
The South African Mint’s RFP detailed the requirements for improvements to processing power and storage capacity, to cater for both the existing needs and future growth as well as to prevent system failures, improve the virtual environment and ensure reliable and effective failover. This included the upgrade of one of the existing admin server hosts and the ability to failover any virtual machine running on the host.
In addition, hosts in the admin server room and in the factory server rooms needed to failover bidirectionally. The South African Mint also requested upgraded storage capacity through the purchase of additional disks, including upgrading the virtual environment.
Locating the problem
“While other RFP participants advised the South African Mint to upgrade capacity to resolve the failures and errors it was experiencing, our assessment revealed a number of pain points that a capacity upgrade would not resolve. We took a different approach and proposed a solution we knew would enhance the South African Mint’s business and deliver on requirements; and as a result, the South African Mint saw the value in the solution and selected First Technology National as its technology partner,” says Soon Broodryk, External Account Manager at First Technology National.
First Technology National ran a comprehensive monitoring tool on the network to collect detailed information and a comprehensive view of the current infrastructure. As a result, the following additional areas of concern were identified:
- Under-performing replication between the admin and factory server room.
- The current replication only allowed for limited volumes to be replicated due to the capacity constraints, risking intermittent unavailability of critical data and systems.
- Replication from slow spinning disks to fast solid state disks (SSDs) caused poor performance.
- The quorum witness resided on one of the production sites. However, according to best practices, the quorum witness should be located on a third site to ensure optimum availability of data and systems in case an issue occurred on another site.
- Backups existed on the production storage, which undermined recovery time objectives (RTOs) because of the single point of failure.
- An upgrade of the existing storage for which a large investment was already made, would require more capital. Consequently, there was a need for a solution that could leverage the current investment.
- The factory storage unit was nearing end of life; upgrading the unit would not allow for SSDs to be added, which would continue to impact performance.
Subsequently, upgrading the existing storage solution would not improve performance as it would not address the issues of controller speeds and increased workloads.
Innovative solution
“We took an innovative, ‘out of the box’ approach and proposed a solution that would address all the pain points the South African Mint was faced with, planning ahead for the next five years and fully utilising the previous investments made. We proposed to replace both the admin and factory server room storage with All Flash Arrays (AFAs) and configured the backup to run on the existing storage solution. The other existing storage solution would be moved to the disaster recovery (DR) site, to enable the replication of business-critical systems backups. We also proposed to offer a three-year, 20 hours per month support service level agreement to proactively support the storage environment. With the proposed solution being completely out of scope, we had to be 100% factual in our approach,” explains Broodryk.
The two AFAs would also be configured as a federated cluster, a revolutionary approach for the South African Mint that would deliver a number of additional benefits. The proposed federated cluster would run two instances of a virtualised storage server synchronously on a live volume. This means that resources would be able to be synchronised across clusters, for load balancing between the servers, which would increase performance. It would also allow for seamless automatic failover in the case of critical hardware failure or disaster without affecting the user experience. This level of high availability is essential for an organisation such as the South African Mint, which cannot afford downtime.
“As a forward-thinking, technology-driven organisation, First Technology National was chosen due to its excellent, modern proposal. The company took the time to fully understand the South African Mint’s business requirement, not simply supplying the hardware requested, but actually proposing a solution to challenges. While First Technology National proposed a product that the South African Mint was unfamiliar with, it provided both the IT team and non-technical committee members with a full explanation and breakdown of the environment and its architecture that gave complete confidence in what the company had to offer,” says Ackermann.
Seamless implementation
Implementation was conducted in line with the requirements and guidelines provided by the South African Mint, in a manner designed to minimise disruption and impact on normal business functions. During the entire engagement, knowledge transfer was encouraged and executed to ensure resources at the South African Mint were equipped with the skills they needed on the solution implemented. Once the storage architecture was in place, the service level agreement came into effect with an established escalation process to ensure efficient resolution of any challenges or issues.
“The implementation was seamless and First Technology National delivered the project within the scope and timelines assigned as per the project plan. There was no downtime required during the project and no issues with the implementation of the solution. Everything was implemented as planned, and since then has been up and running without a single issue. The support and constant maintenance has been reduced significantly due to First Technology National fully managing the environment, ensuring it actions any alerts and keeps the South African Mint informed,” says Ackermann
Benefits
Since the implementation of the federated cluster storage solution, the South African Mint has realised a number of benefits, not least of which is increased storage capacity in both server rooms catering for at least five years of growth. In addition, as the storage is active, live volumes that are completely synchronous, they offer fully automatic and instantaneous failover between server rooms. High availability ensures the user experience is never affected, as there is no downtime. Backups have been separated from production storage to reduce risk, and business continuity has been improved by ensuring critical systems are replicated to the DR site.
“First Technology National has reformed the South African Mint’s storage architecture. Technology underpins the South African Mint’s ability to do business and effectively serve the customers and continually evolve to ensure changing requirements are met. This solution and the support received from First Technology National have been instrumental in allowing the South African Mint to achieve this goal,” Ackermann concludes.
Share