Counting the cost of recovery

The shocking and unexpected attack on the World Trade Centre two weeks ago has very dramatically brought into focus the necessity for companies to have solid and effective disaster recovery plans in place. It has highlighted the vital need for a business continuity plan that embraces not only IT disasters, but also all aspects of the business, and accounts for every possible eventuality.

However, the World Trade Centre attack is an extreme case of an unplanned disaster, and the majority of companies will find that their investment in backup, disaster recovery and business continuity may well go unused for years. Some may never use the facilities they have paid so much for. Much like insurance, disaster recovery and business continuity services are there to protect against the unexpected, and as such, the usefulness of these services cannot be underestimated.

The consequences of not having a plan in place can be disastrous.

Dave Linacre, MD, IBM Business Continuity and Recovery Services

The fact that most organisations are unlikely to ever use the full extent of the services they have paid for has in the past made disaster recovery something of a "grudge buy" and not something that most companies are eager to spend money on.

"Recovery services," says IBM`s Business Continuity and Recovery Services` MD Dave Linacre, "don`t add anything to the bottom line, but the consequences of not having a plan in place can be disastrous. The impact of a loss of business momentum, especially in today`s Internet-based economy, is compounded in the case of a disaster."

Jorgen Nielsen, MGX Holdings` Business Continuity Solutions director, illustrates the point further with a topical example: A Gartner report on the effect of the previous bombing of the World Trade Centre found that of the 300 companies affected by the tragedy, 260 did not have a disaster recovery plan in place. Of those 260 companies, 150 were out of business within a year - a fact Nielsen ascribes directly to the failure to implement a business continuity plan.

However, as companies, both large and small, begin to adapt to being electronic-based organisations, many are starting to see the value in protecting the data and information upon which their business is built.

Linacre says that in today`s connected economy, the downtime tolerance level for most companies has shrunk substantially over previous years, with most organisations unable to weather any more than a few hours of downtime. He notes that for some businesses, the tolerance level is even smaller and often requires an immediate switchover to a duplicate system.

Disaster recovery services are not just an IT issue, says Renita K"orner, CEO of IQ Solutions, a division of the IQ Business Group. She explains that effective disaster planning is as important a part of good business practice as is effective financial control. "It is a business process."

Bernard Vertenten, business manager for SafeGuardIT, says companies have an obligation to clients and shareholders to ensure that electronic data is protected. "Organisations should be proud of the fact that they have a disaster recovery plan. A proper disaster recovery plan gives you a competitive-edge."

"Disaster recovery" is not a term that is popular in the industry, as it has negative connotations and is not a particularly descriptive term. In fact, disaster recovery is just one aspect of what is most widely termed "business continuity", a more descriptive term that embodies the entire planning process that goes into ensuring that no matter how big or small a company, the effects of unforeseen circumstances are kept to an absolute minimum.

Business continuity doesn`t come cheap, and for most companies it is something of a juggling act in which cost is played off against benefit. And the costs can be high. If you are after synchronous mirroring with a guarantee that not a second`s data is lost, says Nielsen, then expect to pay handsomely. However, not all companies require this type of service, and as a result, the costs of shared recovery services are substantially reduced.

Typically a company will spend between 5% and 10% of its IT budget on recovery services.

Bernard Vertenten, business manager, SafeGuardIT

There are those who argue that the costs of disaster recovery services are negligible when compared with the losses that would be incurred in the event of a major data loss. Typically, says Vertenten, an average company will need to spend between 5% and 10% of its IT budget on recovery and continuity services - an outlay that he says more than pays for itself in the case of a disaster.

The sell, however, is still a very hard one as most customers would much rather delay the process and save a little bit here and there. Vertenten says most companies are aware of the need to establish services but shy off at the last moment, tending to delay the process. Most companies, he notes, believe recovery centres "will be there when they face a disaster". Not so, points out Vertenten. "It is a lot like insurance. If you aren`t a subscriber, it means that you don`t have access to the services."

He adds that many companies tend to underestimate their needs in order to cut prices, hoping that in the case of disaster they can squeeze the needed extra capacity out of their continuity provider. Again, says Vertenten, this is not the case.

This is, after all, a capital-intensive industry; a fact that is brought home very quickly by walking around the fully networked and equipped "recovery" offices hosted by most continuity providers. Looking around the empty and waiting "offices", it is difficult to calculate the investment required to maintain, upgrade and purchase the equipment. MGX`s Midrand complex, for example, has place for up to 800 seats, each equipped with workstations, telephones and the standard office requirements.

A parallel system in the same building is a redundant solution.

Bernard Vertenten, business manager, SafeGuardIT

Workstations aside, the most significant costs come in the form of the servers and mainframes that all these companies maintain on an ongoing basis. Without the regular subscription fees paid by companies, few of these centres would be in existence as the overheads are too high.

Many organisations, such as MGX, SafeGuardIT and IBM, also offer full trading room facilities, including satellite subscriptions to Reuters and Bloomberg, as well as fully equipped trading room floor environments. Most of this equipment stands unused, apart from the regular testing procedures that continuity companies build into their services.

Despite the availability of these facilities, most of the "disaster" cases encountered by service providers are hardware and software failures. Nielsen says MGX`s experience is that around 70% to 75% of all disasters are hardware- or software-related. Vertenten concurs, saying that experience has shown that hardware failure heads the SafeGuardIT list, with software corruption a close second, together accounting for 70% of disasters. While most companies see these types of failures as "minor", says Linacre, they can prove to be just as much a disaster as a flooded data centre or power outage.

So where does one start with a business plan? In most cases, companies tend to think of making backups and storing them in a safe, or running two servers in parallel to ensure uninterrupted service. This, notes Vertenten, is the most critical mistake that most firms make as duplicating machines on the same premises is a redundant solution. "That is not a disaster recovery solution," he comments, pointing out that in the case of fire, the entire system could be damaged.

Deciding on a "real" continuity plan involves a number of steps, including assessments and planning procedures. The first step, says K"orner, is to conduct a business impact assessment and risk analysis by assessing the potential damage that can be caused by unplanned downtime and the tolerance levels the business can withstand. K"orner explains that various business models can absorb differing levels of downtime. A brokerage, for example, may well only have a tolerance of a couple of minutes, while a smaller business may be able to weather half a day or more.

The second step, says Nielsen, is to develop a strategy that takes into account cost and benefit factors. This, says Nielsen, can be something of a balancing act, with companies being forced to decide what they can afford to lose in the worst possible case scenario and what their business cannot survive without.

The centre includes stationery as well as spare company chequebooks.

Jorgen Nielsen, director, MGX Business Continuity Solutions

According to Craig Heafield, principal of Organisational Performance Solutions, the cost/benefit ratio is particularly important when deciding on a backup method. Heafield says that for smaller companies, the benefits of live online backups are often well outside their budget constraints. However, he believes that the need for live online storage is becoming increasingly important, even for an SME market that now generates substantial amounts of data that is critical to their business continuity.

"The volume of data managed today by companies is growing exponentially ... and for many companies, high-availability systems are needed to take over within a defined period of time following a failure or interruption," notes Heafield.

Deciding on a continuity plan necessarily involves more than just the safe and effective restoration of lost data. K"orner points out that it is just as important to guarantee that the business processes around the restoration of data and a working environment are in place. It is a problem most prevalent in smaller companies where one person is responsible for various tasks. To this end, a comprehensive continuity plan will include a process for documenting details of all tasks in the business. This will ensure that if the member of staff usually in charge of the affected department is away or, at worst, incapacitated by the disaster, a replacement can easily undertake the required duties.

It is often the more seemingly "mundane", such as stationery and office equipment, which are overlooked in a recovery plan. Nielsen says these types of requirements are built into the service plan at MGX. Its centres have items such as stationery, fax machines and spare company chequebooks on hand.

Although only a small percentage of the disasters reported involve the relocation of staff and equipment, it is in these circumstances that the full extent of a recovery subscription comes into play. When personnel must be relocated, most of the centres can host many hundreds of staff members as well as offering facilities such as conference rooms, video conferencing facilities and management offices. Already established and configured networks are brought into play, giving companies access to their data and working environment within hours or even minutes.

However, in the case of a fire or flood, for example, staff will have to be relocated and space will be needed to rebuild computer equipment as the damaged equipment is replaced. IBM`s BCRS centre, for example, includes access to existing equipment as well as space to accommodate new equipment ready for relocation.

The most critical aspect of the process, all agree, is the regular testing of the recovery environment. Nielsen says that all too often, companies implement a plan that is too difficult or time-consuming to test; for example, internal processes that involve replacing one department with another, which cannot be tested without major downtime and revenue loss. With a disaster recovery centre available, firms are able to test their readiness as often as they require, and generally most are advised to do so twice a year.

Tolerance for downtime is shrinking rapidly.

Dave Linacre, MD, IBM Business Continuity and Recovery Services

So who uses disaster recovery services? In reality, probably everyone should, but in truth a small percentage of businesses have embraced the benefits. Larger corporations, say most of the service providers, have already accepted the need for a comprehensive recovery and continuity plan.

It is often the smaller businesses that still believe a disaster of any magnitude is unlikely to affect them. In reality, says Nielsen, it is these companies that are most likely to be put out of business by a disaster.