Verizon Generator Failure

New York City's 911 emergency system was knocked offline briefly several times during last week's blackout by a generator failure at a facility in Brooklyn, and city officials aren't happy about it. "Verizon officials said the failures took place between 10 p.m. Thursday and 6 a.m. Friday at the Bridge Street office, which operates the generator," according to one news report, which added that Verizon "acknowledged the system did not work perfectly, but added that the city has not purchased a backup plan offered by Verizon." Hey, there's class - seeking to shift blame for a failure that cripples a critical public safety system by blaming the customer for not spending enough money.

Geographic Diversity

Last week's blackout once again has the words "geographic diversity" on the tips of the tongues of Wall Street executives. Federal regulators had used the phrase in their draft white paper outlining business continuity guidelines for the financial industry, designed to ensure that the markets could get back online quickly after a "regional disaster." For such a plan to work, backup data centers would need to be far away from a firm's primary data center - far enough that a major disaster couldn't take out both facilities. Regulators eventually decided not to impose a minimum distance between data centers. But the Blackout of 2003 is causing some companies to once again rethink their disaster recovery plans, according to a story in today's New York Times (free registration required).

"The geographic dispersion of this event really causes some rethinking for companies," said Jim Simmons, the chief executive of Sungard who said customers told him: "Gee, I never thought it would be out on Long Island and out in New Jersey."

The Blackout of 2003

Early reports suggest that the Internet and phone systems performed well during the Blackout of 2003, with reports of one possible carrier hotel outage and a small decrease in the number of hosts on the 'Net.

Some providers at Toronto's primary carrier hotel at 151 Front Street apparently experienced air conditioning problems from Thursday night into Friday morning, according to reports from the North American Network Operators Group (NANOG) mailing list. It was not immediately clear whether the issues were localized or building-wide or related to a generator failure, but some providers in the building were offline. NANOG posts said the building's AC was on the mend as of about 5 am Friday. Some problems may be lingering. As of 10 am Eastern Time Friday, I wasn't able to reach the Toronto Internet Exchange via the Web.

Network operators reported that BGP routing tables showed a drop of between 2,600 and 3,510 prefixes, a drop of between 1 and 2 percent from optimum traffic levels.

Major news coverage of telecom and Internet infrastructure was largely upbeat. A PC World story noted that Internet was largely untouched by the power outages. The only problems were with certain news Web sites that were slowed by high volumes of traffic, according to Keynote Systems, whose Internet Health Report showed business as usual on the major backbones.

Volume was also the largest problem for phone companies in the Northeast. While most systems were working, the outages sparked call volume from those coping with the power outage, especially on cell phones. There's additional coverage of the blackout's impact on telecom in the Washington Post and New York Times.

A small number of companies activated their disaster recovery plans, according to Sungard, but most did so primarily to gain access to the powered office space at Sungard facilities.

Primer on Colo Security

What are the security risks faced by colocation and data center operators, and how do you spot them? Jonathan A. Zdziarski of Network Dweebs has posted a secuity paper outlining the ways terrorists might try to get around data center security, and possible solutions. Much of what he discusses will be familiar territory to security-minded providers. Attention is paid to attacks from within the data center, rather than external threats.

Zdziarski emphasizes the importance of knowing your customers, and sees risk in some providers' limited scrutiny of new customers and the equipment they are installing. "The bottom line is: how difficult is it for anybody out there to sneak their way into your facility as a tech or a customer, and sneak in a dangerous device?," he writes. "If the answer is 'not very difficult' then you have some vulnerability in your policies that could potentially expose you to an attack of any type."

Breathing Room for Cogent

Light Reading has an update on the restructuring at Cogent , which leaves the company's backers with 96.6 percent of the common shares in exchange for $41 million in fresh investment. About $20 million of that cash went to Cisco Systems, the company's largest trade debtor. Cogent had fallen short of revenue targets, placing it in technical default of its $262 million credit line with Cisco. Rather than forcing Cogent into bankruptcy, Cisco opted for some cash and a significant equity stake. The deal dramatically slashed Cogent's debt burden, reducing its debt to Cisco to just $17 million - just 6.4 percent of what it previously owed the equipment maker.

The wisdom of Cogent's business model has been widely debated on industry mailing lists, especially the ISP-Colo list, where it seems you can't mention the company's name without setting off an extended flame war. The company's focus on cost management has allowed it to survive its ambitions and pricing, which have both been disruptive to competing bandwidth providers. But the pain in Cogent's restructuring has been borne by its investors rather than its customers. On balance, that's probably a good thing for data center operators, whose tenants include hosting companies that rely on Cogent's cheap bandwidth to squeeze out profits in an increasingly price-sensitive business. If Cogent had failed, the ripples would have been significant. Instead, pundits can now debate whether a largely debt-free Cogent can make the model work.

Turmoil at HavenCo

Following a wave of publicity upon its launch in 2000, HavenCo became the most famous (or infamous) colocation facility in the world. Now the company, formed to host secure sites on an old gun platform in the English Channel, may be facing a financial crisis. Ryan Lackey, the former chief technical officer for HavenCo, says the company is nearing financial ruin and has just six remaining customers. Representatives of HavenCo dispute Lackey's description of the company's status.

Lackey attributed HavenCo's troubles to difficulties with the "ruling family" of the Principality of Sealand, which claims to be a sovereign nation even though the platform lies within British territorial waters. Lackey told an audience at the DefCon hacker convention that Sealand was nervous about hosting controversial clients, and caused network outages by "tinkering" with the company's network connections.

An unnamed representative of HavenCo painted a different story in an e-mail response to an inquiry from C/Net's Declan McCullagh. "We have a moderate-sized installation which is growing monthly, very many more than the alleged six customers and their servers in operation, and in the last eight months or so have been able completely to re-engineer our network and its international connectivity arrangements," the representative said Monday.

Lackey is still listed as the administrative and technical contacts for HavenCo in domain records. If HavenCo is really so secure, why does a disgruntled former employee who says he is owed $220,000 still have management control over their primary business domains? One wonders if Lackey is still the technical contact on customer domains as well. Security consultants consistently cite failure to limit ex-employees' access to key business data as a failing of many corporations.

Explosion at Reno Facility

A transformer explosion this week caused the evacuation of 200 South Virginia (The Wells Fargo Building) in Reno, a carrier hotel that houses numerous telecom and Internet companies. A transformer exploded about 4 p.m. Tuesday, sending flames shooting up the side of the eight-story building. Oil used from the transformer caught fire and ran down the street. A passerby was slightly burned in the explosion. "I was walking along looking at the big building when suddenly there was an eruption from behind the brick area," Chris Christiansen told the AP. "It was a big explosion, and a fireball shot across at me burning me on my face and neck and my ears are rumbling from the noise. The fire came down the curb and glass started shattering. Papers were flying out of the broken windows. The trees were on fire and were burning pretty good."

The building lost grid power for about 10 hours, but several larger telecom tenants, including Advanced Telecom Group and MCI/WorldCom, continued to operate as their generators took over, according to posts on the mailing list for the the North American Network Operators Group (NANOG). There were reports that some smaller telecom tenants were knocked offline during the incident. The building regained power at about 1:30 am Wednesday after the transformer was replaced.