First National Bank's (FNB's) power outage earlier this week affected customers across the country and abroad, and has raised questions about City Power's ability to service the bank.
Raj Makanjee, CIO at FNB, confirms the bank successfully transitioned to its alternate data centre today, and that all systems were back at full capacity.
Makanjee says the bank can handle power cuts, but this week's outage was caused by a power spike greater than anything the bank's systems could handle.
“Since 2008, we've had frequent power outages, and invested heavily in infrastructure to keep running when that happens,” Makanjee notes. “We've performed well in the past to ensure there's no impact to our customers throughout a cut. But this was something very different. We've never seen anything like this.
“We're still waiting for explanations from City Power, but the estimates suggest a surge of around 12 000kVA - equivalent to about eight simultaneous lightning strikes. We use two different substations, and they were both affected. Because we're close to both substations, we took the brunt of it. We've heard other businesses nearby were affected, but not to the same extent.”
Although the bank managed to keep critical operations going, albeit at a reduced capacity, the incident has caused alarm about City Power's ability to provide a stable electricity supply to the bank's data centre, Makanjee says. “We're considering our options and looking at options for alternate power sources.”
Although the outage affected customers across the country and even abroad, Makanjee says the incident response went by the book. “We have disaster management processes, and they're fully tested. I'm not happy about the incident, but I am happy that our processes worked according to plan.”
Reduced capacity
The normal procedure following a complete failure is to switch to a reduced-capacity standby system first, before cutting fully across to the alternate data centre, Makanjee says. “We have a standing system which operates as a backup for critical transactions like card swipes and in-branch deposits. That system was invoked immediately. Our data centre normally processes 800 to 1 000 transactions per second; the standby system can manage about 200 per second.”
That reduced capacity meant frustration for thousands of customers, many of whom took to Twitter to vent their frustration on the bank. CEO Michael Jordaan took the brunt of it, apologising to customers and promising extended banking hours to help catch up.
Besides the reduced capacity, “there were some problems in business and online transaction systems, but they were resolved quickly,” Makanjee says. “At 10am [following the incident] we officially declared a disaster and cut fully across to the alternate data centre.”
The alternate data centre will be the primary for the near term, he says, since a power spike of that magnitude will have caused lasting damage to the bank's electrical systems. “There's definitely going to be permanent damage. We're currently assessing the damage, but we won't know for a couple of days how extensive it is, and how long it will take to fix. Until then, we'll be running out of the alternate data centre.”
City Power could not be reached for comment at the time of publication.
Share