IN CASE OF EMERGENCY
.New Technology – and new threats – have businesses reexamining how they cope with disaster.
By John Goff

UMention Calgary, the city of 400,000 in western Canada, and civil unrest doesn’t spring to mind. This quiet outpost in Alberta is better known for the Saddledome, all-season skiing, and the annual rodeo roundup called the Stampede.

But in 2002, the prospect of civil unrest was worrying managers at Calgary-based Canadian Pacific Railway. At the time, Calgary was getting ready to host a Group of 8 summit, and reports began circulating that protesters were going to try to shut down parts of the city. Recalls Paul Cammack, a manager of the railroad’s contingency-planning management group: “We were concerned employees might not be able to get into the building.”

The civil disobedience never materialized, and the railroad kept running. But other dangers remain, including fires and car or truck accidents in the city center. “A main line runs right by the office,” explains Cammack.

To keep its operations center operating, the US$3.7 billion company has invested considerable resources in disaster recovery. In 1999, it constructed a state-of-the-art, remote hot-site. Interestingly, the company has also poured money into an empty lot that abuts the site.

Rocky Mountain fever? Hardly. In case of a catastrophe, the company plans to park two large trailers on the lot. The trailers are deployed complete with computers, desks, and telephones, courtesy of Agility Recovery Solutions, in Ontario. They are connected via “hitching post” to the hot-site for instant connectivity. During an emergency, the railroad’s management plans to house up to 80 additional workers in the trailers, mostly to handle customer inquiries.

Until the recent train bombing in Spain, a mobile site next to a hot-site might have qualified as disaster-recovery overkill. But such a view ignores the herculean coordination necessary to run a transcontinental railroad. “If we can’t throw switches,” says Cammack matter-of-factly, “we’re out of business.”

The phrase paradigm shift aptly describes what’s going on in the world of disaster recovery these days. Spurred on initially by Y2K and, more recently, 9/11 and the blackout in the northeast US in 2003, corporate executives are focusing on data protection like never before. According to US research firm Meta Group, companies spent just 3.2 percent of their IT budgets on security (employee education, business continuity, and disaster recovery) in 2001. Last year, the outlay was more like 8.2 percent – a dramatic increase.

This newfound interest in security goes beyond increased spending. Advances in technology – and a wider array of threats – have corporate executives rethinking their whole approach to disaster recovery. The days of the onsite, raised-floor room, with rows of clunky tape machines and droning cooling units, are fast disappearing. In their place: remote hot-sites, fail-over systems (backup networks that can be brought on-line instantly), and Web-based file storage and retrieval. Says Gregg Therkalsen, vice president of business continuity at US-based vendor EMC: “The idea of backing up info on tape, having human beings put that in a truck, and driving it away ... Well, every customer wants that to go away.”

Not Every One Gets Corrupted

Ellen Christy can attest to that. Christy, director of information technology at US-based private-equity specialist HarbourVest Partners, says the company used to back up its data to tape onsite. Then, at the end of each day, an employee would lug the tape home and store it on some high shelf. “But small companies grow,” she notes, “and one tape becomes two, two become three ...”

In early September 2001, after a six-month process, management at HarbourVest ditched its in-house tape backup, choosing instead to send the data from its 15 servers over the internet to a remote site. The company stores 100 gigabytes of data at the site, and about half of that is base data; that is, financial records, agreements, and the like. The information, which is backed up nightly, is retained for several months. In case of an outage necessitating a massive restore, Christy says vendor AmeriVault could cut the data to tape and get it to her in around two hours.
The real selling point of the web-based service, however, is that employees can retrieve lost or zapped data simply by going online. “The most common problem we have is people deleting files,” explains Christy. “It takes 50 percent longer to restore a file using the tape-backup method.”

Scrambling through reams of old tape can certainly be a laborious process. Worse, tapes and other backup media are notoriously unreliable. Experts say data gets easily corrupted, and often tape backups just plan fail. “Half the time, zip drives and tapes don’t restore,” insists Wally Beddoe, vice president of operations in the US office of Telekurs Financial. “They can be a big waste of time.”

To address that issue, management at the Swiss-owned supplier of financial data hired a US vendor called Connected. It provides a back-up service that safeguards Telekurs Financial’s distributed data – information not stored on network servers. While Telekurs does back up its commercial data to a remote site in Connecticut, employees rely on their PC hard drives to store tons of information – contracts, e-mail, even application code. “All the stuff to support our business is on PCs,” says CFO Mike Stisi. Moreover, Telekurs has an increasing number of employees, including programmers, who work remotely. “The stuff they have on their PCs is scary,” notesStisi. “It’s hundreds of man-hours’ worth of work.”

The finance chief can attest to just how valuable the company’s new retrieval system is. In December 2002, Stisi came to the office only to discover that the hard drive on his computer had failed. “When my hard drive died, I almost had a heart attack,” he recalls. “Duplicating the information, including customers and contracts, would have been a huge headache.”

Using the retrieval service, Stisi recovered his files in a matter of minutes. Since signing on with Connected, he says he hasn’t had to worry about failed hard drives and flat-line laptops. This, of course, raises the obvious question: Why is the company finance chief involved in such mundane matters as lost Excel files – matters usually left to CIOs and system administrators? “As the CFO, I’m responsible for protecting our assets,” explains Stisi. “My neck and the CEO’s are on the line.”

Didn’t Think of That One

Before Y2K and 9/11, most finance chiefs were woefully ignorant about the value of digital assets. Even today, few know the difference between a Bernoulli Box and a Bento Box. But with new threats – including terrorist attacks, computer viruses, and infrastructure failures – many CFOs are beginning to at least sit in on disaster-recovery meetings. “Some CFOs perceive disaster recovery as a sunk cost,” says Gary Foster, CTO at US trade-management services provider Omgeo. “But you have to think worst-case once in a while.”

At US-based Edgar Online, a US$15 million (in revenues) supplier of public-company data, CFO and COO Greg Adams is doing more than that. He reviews his company’s written disaster-recovery plan in detail each year. Adams is also apprised of changes in the plan before he files the company’s 10-Qs. “Disaster recovery is critical for us,” notes Adams. “If we’re down, a lot of money is lost.”

After the events of September 11, management at the Connecticut-based company decided to construct a remote hot-site in the state of Maryland. The site, which has a backup generator, can restore the company’s main systems in a matter of hours. Edgar Online also maintains a New York–based fail-over for its website (as the name implies, the fail-over immediately kicks in if the website fails).
The system was put to the test last August, when the huge power outage knocked out the electricity at Edgar Online’s Rockville office. “During the blackout,” recalls Adams, “we had no downtime.”

Other companies were not as fortunate. Atlanta-based Delta Air Lines, which maintains an extensive disaster-recovery and business-continuity plan (including backup generators for its main and remote sites), was able to keep its planes running and its ticket systems operational during the power outage. But according to Keith Hansen, manager of emergency-response and business-continuity planning at the airline, Delta passengers at a number of airports couldn’t board their flights after the power went out. The reason? Unlike the well-prepared Delta, some airport security systems didn’t have backup generators. “We’re now looking at hub and major airports,” notes Hansen. “If they don’t have a backup [power] system for security, we try to convince them to get one.”

The summer blackout exposed shortcomings in other disaster-recovery plans, as well. Many businesses, for example, discovered that their remote sites simply weren’t remote enough. “It’s all right to have a backup center,” says Lance Travis, vice president of core research at Boston-based consultancy AMR Research. “But if you’re in the same power grid, it doesn’t do you any good.” Moreover, a fair number of companies found that their uninterrupted power sources were designed to run for only a few hours. Now, says Travis, some corporations are looking for remote sites that are so far away they can avoid almost any blackout.

Such a strategy, while prudent, can constrain the amount of data that gets backed up. Delta, for one, performs synchronous backups from a mainframe to a remote site. That’s a massive dumping of data – and one that limits the distance between the company’s remote site and its main data center. As Ray Shepherd, coordinator for business-continuity planning at Delta, explains: “You can push that amount of data only so far.”

Coming: More Bad Stuff

Experts believe that increased bandwidth and better compression technology will ease the problem. Already, Connected can shoehorn the information from 15,000 PCs onto one NT server, a fairly remarkable achievement. But supply is barely keeping up with demand. The fact is, companies are producing prodigious amounts of data these days, a trend that shows no sign of abating. “Ten years ago, people were running businesses off what you can get in a laptop today,” says Omgeo’s Foster. “Now we’ve got terabytes of data.”

And while the price of storage technology has come down in recent years, backing up mountains of data can be an expensive proposition. Some companies, in fact, are choosing to discard data after a short time. Haynsworth Baldwin Johnson & Greaves, a US law firm, sends new or changed files to a co-location site each night. After 14 days, earlier versions of files get deleted. Skip Lohmeyer, information systems director at the firm, says he’s able to retrieve files, which come across in an encrypted format, using a password and a built-in decryption code.

The cost: US$4,400 a month to store 140 gigabytes of compressed data. “When you look at it from a mid- to small-company perspective, it may seem expensive,” grants Lohmeyer. “But [the reality is], you’re going to have a disaster.”

John Goff is technology editor at CFO in the US.