Good governance issues make disaster recovery a good buy

Paying out money every month or year to a service provider for a service you hope you`ll never make use of doesn`t make a lot of sense in common business terms. And yet, this is exactly what more and more companies, both big and small, are starting to do.

Disaster recovery doesn`t add to your bottom line, says IBM`s Business Continuity and Recovery Services MD Dave Linacre, "but what are the costs going to be if you don`t have a recovery plan in place and something does go wrong?"

It is not an easy sell, says Linacre, and relatively few companies are willing to pay for disaster recovery services.

The problem, says Bernard Vertenten, business manager of SafeGuardIT, is that most people imagine a disaster to be an earth-shattering event of the magnitude of an earthquake, fire, storm or even a terrorist attack. In reality, says Vertenten, a disaster could be caused by something as simple as a failed hard disk, corrupted backups or theft. The true "disasters" do happen, say service providers, but they account for only a small part of the contracts invoked.

Vertenten says that of the approximately 2% of clients that invoke their contracts over any given period of time, the vast majority are hardware-related issues.

Unless your organisation is world-class, you won`t get the full benefits of high-availability systems.

Allen Smith, MD, MGX Business Continuity Solutions

Allen Smith, MD of MGX Business Continuity Solutions, says that of the 11 invocations in the past two months at the company, only two included physical re-location, with the balance being hardware and software failure issues. Some 90% of failures are the result of IT system failures, says Vertenten.

Glen McDonald, software division manager for Drive Control Corporation, agrees. "Data loss can happen through numerous causes from hardware failure, application failure and viruses. In SA we have some unique threats such as lockouts and rioting. Theft is obviously a real threat in SA."

Despite the fact that hardware and software failure is the biggest cause of data loss and "disaster", the push towards disaster recovery is not driven as much by the IT department as it is by company management. This is very different to a few years ago when it was the IT department that spearheaded disaster recovery with backup procedures, failover systems and redundancy.

Today, issues of disaster recovery, or rather business continuity, are high on boardroom agendas around the country. So much so that disaster recovery is largely a dying terminology, being replaced by the all-embracing term "business continuity".

"The question now being asked is this: is the overall business protected and not just the IT infrastructure?" says Linacre. IT is still an essential component of this, he points out, because more and more companies rely on IT infrastructure for their competitive advantage in the marketplace. But the question is an extensive one that includes all elements of business operation from IT infrastructure to personnel.

The wider concerns of business continuity are driven, says Smith, by the greater awareness of good corporate governance within the business sector. He notes that issues such as the King commissions and the auditing scandals in the US are making business managers increasingly aware of the responsibilities towards company shareholders and employees, and the threat of liability in the case of a disaster. Pin this together with an increasingly reliance on IT infrastructure, and it is no wonder that disaster recovery and business continuity are regular features of boardroom meetings.

"Businesses are more and more dealing with this as risk management," says Smith. "The modern MD is aware of the value of IT and the issue is discussed at boardroom level."

Bandwidth is still the great inhibitor in this industry.

Dave Linacre, MD, IBM Business Continuity and Recovery Services

Shoden Data Systems MD Fanie van Rensburg agrees: "Big businesses have decided the time has arrived to be serious about disaster recovery."

"Disaster recovery is back on the agenda," says Linacre. "But not in its traditional sense of disaster recovery but as business continuity. It is evolving into a business process ... and growing awareness of good corporate governance is creating greater awareness of disaster recovery."

No one is immune to the trend, says Linacre, not even smaller businesses. He explains that while many businesses may not have considered a disaster recovery plan, they are often part of a bigger supply chain and many of their trading partners may demand that they are adequately protected against failure and downtime. Similarly, Smith says that for many listed companies, disaster recovery and business continuity plans are of strategic value to organisations that are able to inform potential investors and shareholders that they are doing what they can to ensure they are adequately protected.

Disaster recovery may be gaining greater mind-share and more companies are becoming aware of the value of their data, but one thing that hasn`t changed much is the cost of recovery services. In fact, with increasing reliance on high availability systems, larger volumes of data and limited bandwidth, disaster recovery services are in some cases even more expensive than ever before. Whereas many businesses in the past were capable of handling downtimes of 12 and 24 or even more hours, today`s online world has reduced that to anywhere in the region of 6 to 12 hours, and for some even less. "Tolerance for downtime today is significantly less than a few years ago when periods of 24 to 48 hours were acceptable. Now that tolerance level is down to 12 to 6 hours and even less," says Linacre.

"If you lose 24 hours today," adds Van Rensburg, "you can recover to some extent ... but can your business afford it?" The impact of downtime is exacerbated in the case of institutions such as banks and financial systems, he notes. "A bank experiencing downtime, for example, could have an impact on a country`s economy."

Vertenten describes the new world of business as a "smaller, faster and more dangerous place than ever before". As systems become more capable and pervasive, they in turn become more complex. And maintaining a disaster recovery system across distributed systems such as these is overly complex and expensive to achieve.

And yet, despite the complexity of building a good disaster recovery plan, many companies still do it themselves, something that Smith argues is counter-productive and often more expensive than opting to contract the services of a disaster recovery facility. His main argument against the "self-help" approach is that disaster recovery is not a core competency of most companies. Nor do they have the requisite staff to design and maintain the system.

Some 90% of disasters happen as a result of IT systems faults.

Bernard Vertenten, business manager, SafeGuardIT

More importantly, he says that business that do it themselves typically repeat the errors of those that went before. These include a failure to position a recovery facility at a suitable distance from the production environment and a failure to maintain the facility. But the biggest faults, comments Smith, are the failure to test and the tendency for companies to use their recovery facilities for production activities as soon as they experience a growth curve.

"Business continuity and disaster recovery are all about testing. Without testing, it is just a theoretical exercise."

He notes that a business continuity plan and facilities should be tested every few months. In reality, however, many firms leave the testing periods far longer than that and some hardly ever test. "We can`t force clients to test often, because it is their own business, but we do advise them to do so."

On the cost of companies doing it themselves, or contracting to an outside service provider, Smith is adamant. "The costs of doing it yourself could well exceed the costs of paying for a service." Particularly considering the self-help option may be flawed and incomplete.

Not everyone can afford to mirror their data in real-time in two or more different locations. And for some, doing so is not necessary and adds no value to the company. So what do you save and protect?

Linacre says it is a risk management decision. "A decision needs to be made on what businesses need to protect. They need to decide what business process they need to stay available to their users."

McDonald offers an example: "Choosing a disaster recovery solution really depends on the type of business involved. A trading desk stands to lose millions if they are unable to trade for an hour, but McDonald`s would still be able to make hamburgers even if they did not have their computers for a week."

It is important to decide on the levels of tolerance within the business. E-mail is another good example, says McDonald. "If the mail server crashes, how long is acceptable before mail is restored?"

"The trick," says Vertenten, "is to balance your needs."

For financial institutions, the demands are significantly bigger and typically they are forced to pay for dedicated facilities with high availability mirrored solutions. Similarly, financial trading desks require immediate switchover, at least enough of a time gain to close investments and avoid significant financial loss.

For the average business, bandwidth, or rather the lack of it, is the biggest hurdle to affordable mirrored off-site systems. "At the bottom-end, the bandwidth available is enough so that they are able to achieve a backup copy that is true at the end of the day`s trading," says Van Rensburg. "At the top end, however, it is very expensive to have full mirroring systems."

Linacre agrees that bandwidth is "the great inhibitor". "We are looking forward to the possibility of increased bandwidth because more and more companies are looking at going the whole way to high-availability solutions that allow for quick switchover."

Understanding the capacity you have to cater for is the most difficult to decide and the hardest to fix.

Fanie van Rensburg, MD, Shoden Data Systems

But disaster recovery is not only about mirroring data to another location. For many it is as simple as ensuring backups are made regularly and tested frequently to ensure copies. There are numerous disaster recovery technologies available to businesses, says McDonald, including "clustering, replication, mirroring, hot standby, cold standby, near line storage, online backups, full daily backups, incremental/ differential backups, snap-shot backups. The options are numerous and varied; each have advantages and disadvantages and each come with their own price tag."

While many companies opt for the relatively simple backup process, Van Rensburg highlights a number of issues that are often overlooked by businesses when purchasing a backup solution. The first is related to the amount of data that needs to be backed up, as administrators often forget that the backup windows are a lot shorter today than they were a few years ago. Online trading and 24-hour presence make effective backing up a difficult task to manage.

Similarly, he says, there are companies that are feeling the pinch of bandwidth constraints because they are unable to complete the previous day`s backup to remote locations before the new day resumes, meaning that many businesses are never fully backed up during normal trading. An equally important issue, says Van Rensburg, is that administrators forget that a single change in a system of data often has a far bigger knock-on impact and these also need to be backed up.

Van Rensburg believes there are two main mistakes that administrators make when backing up. The first is that they underestimate capacity. "The capacity you have to cater for is often the most difficult to decide and the most difficult to fix." The second is the nature of data changes. "The changes made to a system are a significant factor that is larger than the growth factor." It is necessary to backup the changes as well as the new files. Similarly, he says, a decision has to be made on what is backed up. "Policy-based storage management is key in disaster recovery."

There are three main players in the local market that offer disaster recovery facilities: MGX`s Business Continuity Solutions Division, SafeGuardIT and IBM Business Continuity Solutions Division. Together they provide services for many of the biggest institutions in the country, as well as a growing portion of the lower end market. All three offer a full range of recovery facilities, including syndicated and dedicated hardware, call centre capabilities as well as financial trading desks. The services provided range from backups to full office environments.

Typically the dedicated equipment, which comes with a hefty price tag, is taken up by financial organisations that require immediately available hardware in the case of a disaster. For other clients the best option is to syndicate the equipment which lowers the overall cost for each subscriber. Hardware is typically syndicated at around 20 users per box, says Smith. By limiting the number of users per box, they can all but guarantee that equipment will be available in the case of multiple clients declaring a disaster. For the majority of time, however, the equipment stands eerily quiet with only the occasional testing procedure occupying the servers and desktop machines.

Smith points out that having hardware on standby is only part of the equation. He says that much is made of the speed at which recovery can be achieved - anywhere from a few hours, to a day or so. However, he warns that this is only the case if the business is correctly prepared.

Disaster recovery is like insurance: if companies do purchase it, they do so grudgingly and if they don`t and something happens, they rue their stupidity.

Glen McDonald, software division manager, Drive Control Corporation

"People expect mirrored systems to give them 99.9% uptime and availability. Unless your organisation is world-class, you`re not getting that," so don`t expect it. A sentiment echoed by everyone in the industry: disaster recovery is about planning and testing, and then hoping that nothing goes wrong. And when it does and you don`t have a plan? "Simple, you go out of business," says McDonald.

Disaster recovery is not cheap. It is also not simple. As business comes to rely on more extensive and complex IT infrastructure for business advantage, the process of managing disaster recovery becomes an equally complex one. But with a good business continuity plan that encompasses all areas of business and not just IT infrastructure failure, businesses are better able to take advantage of the many technologies available on the market and from service providers.

If you lose 24 hours today, you can recover to some extent ... but can your business afford it?

Fanie van Rensburg, MD, Shoden Data Systems

After all, disaster recovery is about risk management. It may not add to your bottom line in profitability, but the costs of not having a plan could be enormous, perhaps even disastrous to the business.

There is no rule that adequate disaster recovery can only be achieved by business continuity service providers, but what is clear is that there are many pitfalls on the way to a comprehensive in-house solution. And skills are just a portion of this. Bigger conflicts such as inter-departmental interests and expansion plans often hinder or damage the best-laid plans, warns Smith.

And if it is decided that the best solution is a simple backup plan, then it would be wise to heed Van Rensburg`s warning that most administrators underestimate their capacity needs and this is often the most difficult to fix after the fact. He also cautions against proprietary systems lock-in that makes expansion difficult or sometimes nearly impossible.

Is mirroring right for your business? Unless you`re a trading house that stands to lose millions for each second of lost trade, then probably not yet. More people are considering the move to mirrored systems, says Linacre, but bandwidth is still the big inhibitor. Until remote backup is possible at speeds that meet the volume needs of current businesses, the mirroring option is likely to remain illusive. And as trading hours extend and volumes increase, the backup window closes, placing extra financial and logistical strain on organisations.

Signing up for a syndicated solution is probably the best bet, particularly if the service provider offers consultancy services to draw up a realistic plan of what can be achieved and at what cost. However, as Smith warns, you can spend all the money in the world on hardware but if you don`t have a well-structured continuity plan, then your efforts will fail.

And don`t only plan for a typhoon, flood or war. As Vertenten says, 90% of "disasters" are hardware and software failures, but they can do as much damage to an organisation as a full-blown natural disaster.