170 likes | 386 Views
Super Storm Sandy Case Study. Sandy – what’s special?. Major metro area Data center consequences Was it foreseeable? Lessons learned. New York City. Data Center Impacts. 75 Broad Street Internap Peer 1 33 Whitehall Datagram 60 Hudson Street Big APE NYIIX EQIX-NYC.
E N D
Sandy – what’s special? • Major metro area • Data center consequences • Was it foreseeable? • Lessons learned
Data Center Impacts • 75 Broad Street • Internap • Peer 1 • 33 Whitehall • Datagram • 60 Hudson Street • Big APE • NYIIX • EQIX-NYC
Customers impacted Major outages at Internap, Peer 1 and Datagram. 60 Hudson Street - Homeland Security
Other Comments • AtlanticMetro: LGA4 at 121 Varick St., which suffered a generator failure Oct. 31. • Datagram: Via Twitter, “A 2 megawatt mobile diesel generator is confirmed to arrive tomorrow (Nov. 1) before 5 PM,” the company added. “There is a chance that power will be restored before 5PM.” • Equinix: Equinix appears to be stable, at about the same status as on Oct. 31. • Ezzi: Ezzi operates three facilities: at 882 Third Ave. in Brooklyn, and at 75 Broad St. and 25 Broadway. The company has not published any updates on its Web site or Twitter feed. When Slashdot called the support line for an update, we were told that one data center remained open and one remained down. • Internap: Internap operates two facilities, one at 75 Broad St., and one at 111 8th Ave. The Broad St. facility was flooded with up to three feet of water and shut down, and the company said late on Oct. 30 that it was “working as quickly as possible to implement a workaround for the fuel system that will allow us to bring the generator farm back into operation.” • Peer1: Peer1’s operations at 75 Broad became one of the Sandy stories that made for good press: a bucket brigade that carried diesel fuel up 18 flights of stairs to the generators that kept operations running. The hosting company enlisted its own customers to help out. But there’s bad news. Manpower appears to be running short. • Telx: All sites are running, although the company advised customers to power down where they could. • Verizon: On Nov. 1, the company said little about its wired infrastructure services, choosing to focus on how it was erecting temporary wireless towers for customers, and rescue and recovery efforts.
Data Center Flooding Verizon
Diesel Became the Issue Datagram – Fog Creek Bucket Brigade
How Much Diesel? • 1 rack = 5 to 10 KW including HVAC • 20,000 SqFt= about 600 racks • 5 megawatts (MW) of power • A reasonably efficient generator needs 200 gallons of diesel per hour to push out 5 megawatts. • = Just over 3 gallons per minute • Bucket brigade is hopeless and …
Typical Tracks Based on typical track, October is the period of greatest risk.
Storm Origin Activity is less in October, but it’s hardly as dormant as it is a few weeks later:
Actual Track October 29, 2012
Lessons “Everybody’s got a plan ‘til they get hit.” – Mike Tyson