May 2, 2002 -- It is the hypothetical "what if" moment that justifies the existence of data centers. You never know when it may arrive. But it's the reason behind the installation of huge generators, expensive UPS systems and banks of batteries.
What if the power goes out or there's a natural disaster? Will my company's data be safe?
"What if" arrived Monday afternoon in Jacksonville, Fla., when a system failure left 355,000 customers of the Jacksonville Electric Authority without electricity. For Peak 10 Technology Gateways, it was the moment when all the contingency planning came off the drawing board and into the data center.
It all worked as planned and as practiced. But that's not always the case.
In the past 18 months, data centers' reliability claims have been tested by a series of disasters, including a 6.8 magnitude earthquake in Seattle in Feb. 2001, flooding from Tropical Storm Allison in Houston last June, and the Sept. 11 terrorist attacks in New York and Washington.
On balance, the industry has performed exceptionally. Several Houston data centers were flooded, and one NYC carrier hotel experienced sustained outages in September. But the vast majority of facilities have stayed online, and that was the case this week in Jacksonville.
It started innocently enough. At 3:50 p.m., a lightning arrestor - a small device designed to protect transmission lines from lightning strikes - failed near from the Pickettville Substation on the west side of Jacksonville. Within minutes, that single failure had triggered a cascading series of equipment problems that overwhelmed the JEA's entire grid.
By 4:15 pm, many residents and businesses in Jacksonville were without power.
At Peak 10's Jacksonville facility at 4905 Belfort Road, monitoring equipment detected the loss of commercial power and switched to battery power. Within seconds, the building's 750 KVa Caterpillar diesel generator came to life, keeping the web servers up and running.
Meanwhile, the power supply fluctuated across the city as JEA engineers sought to get the system back online. The first outage at Peak 10 lasted just four minutes, followed by a two-hour period of normal operation, and then a second outage that lasted 20 minutes.
"Our facilities are set up to stay on the generator until the power has been restored and has stabilized for 30 minutes," said John Proctor, a senior engineer at Peak 10. "All in all, we ran off the generator for about an hour and a half. We keep 2,900 gallons of fuel on site, which will allow the facility to run for 48 to 72 hours, depending on the load."
In addition, Peak 10’s service agreements from local fuel vendors guarantee priority refueling in case of long-term power outages.
Things were worse elsewhere, as some areas of Jacksonville were without power for 12 hours. City residents had to boil water for several days afterward due to water pressure problems related to the blackout.
While other Jacksonville businesses lost hours of productivity and sales, Peak 10 customers remained online. For data centers, that's business as usual.
"The power is always on at Peak 10,” said executive vice president and COO David Jones. “Once a week, we simulate a power failure and run all facilities off of our diesel generators to ensure that our power systems will continue to operate seamlessly.”
"We're in the business of keeping customer networks and systems up and running all of the time - even during major commercial power failures," said Mike Raines, general manager of Peak 10's Jacksonville site. “This is exactly why customers use Peak 10.”
Some members of the local business community have apparently noticed.
"We have entered into negotiations with several new prospects since the outage," noted Proctor.
.