eBucks learns its lesson

First National Bank`s (FNB) Internet banking division eBucks.com says valuable lessons have been learned from the downtime experienced by the Web site in September.

The unscheduled downtime and two weeks of intermittent service at the end of September angered and frustrated many eBucks customers, prompting the bank to repay all service charges for the following month.

Despite the apologies and financial compensation, customers were dissatisfied with the explanation that the system had failed due to unprecedented transaction volumes.

At the time, FNB Internet Banking head Roland le Sueur said a number of technological and business issues would be reviewed and a report issued. True to his word, Le Sueur has released the report`s conclusions and recommendations.

The report says extensive investigations revealed that a number of factors were responsible for the initial downtime and following instability of the eBucks Web site.

It was found that the monthly build on 21 September generated significant database fragmentation, resulting in below optimum performance of the eBucks infrastructure.

"Due to the fixed disk and CPU infrastructure on which this database operates, the capacity of the eBucks system was put under strain. Complicating this was the large amount of 'garbage` on the database."

Ironically, the report continues: " Due to risks of downtime, it was decided in mid-2003 to conduct a last database optimisation operation in mid-October, rather than weekly. This would have thoroughly reorganised the database. The combined effect of all these issues meant that, the infrastructure did not have the capacity to deal with the normal load volumes of 22 September."

The report says when instability was identified it was decided to take the system off-line to clean the database, but the planned performance increase was insufficient to cope with the increased load as a result of the downtime.

"The intermittent downtime from the 22^nd to the 26^th generated significantly increased demand for the service. Even completely optimising the database structure [over the weekend of the 27^th] did not improve system performance enough to deal with the increased load."

The service was restored to standard service levels only once technicians had improved the disk throughput by optimising disk controller and operating system settings.

eBucks says steps and strategies to safeguard against a recurrence of the problem include the following:

* A new Sun Fire server has been installed with 64 CPUs enabling partitioning into various domains and dynamic allocation of CPUs and memory across domains.

*High-level load indicators will be identified, and trends monitored more closely to enable load forecasting and capacity planning.

* Capacity planning is to be based around peak volume demands.

Le Sueur says continued activity on the site and increasing registrations is most encouraging. "There is an average of 7 500 registrations a month and the base is currently at 210 000 Internet bankers performing an average of 1.3 million financial transactions, which shows that despite the inconvenience caused, customers continue to utilise and benefit from the service."

He says eBucks.com remains committed to improving its service. "The lessons learned through this difficult period will facilitate even better, faster and more efficient Internet banking."