Crisis Management The New Role of the CIO/CTOs & IT staff

1. Crisis Management The New Role of the CIO/CTOs & IT staff

2. The City of New York • Resident population of over 8 million; daytime population of 10 million • Over 350,000 City employees, 300,000 retirees • New York City Government includes 5 counties • The 1 million student school system reports to the Mayor • Annual budget exceeds $54 billion dollars • Trails only the Federal Government, and the states of California & New York • If New York City was a private sector corporation, it would be in the Top 30 of the Fortune 500 companies • Over 120 agencies, offices, and organizations make up “The City”

3. New York City’s Public Safety Record In December 2005, out of the nation's 10 and 25 largest cities, New York City ranked the safest with the lowest overall crime rate. Among cities with 100,000 residents or more, New York City ranked 211th out of 227 cities.

4. New York City’s Major Incidents - the past 5 years • Terrorist Attack on World Trade Center – September 11th 2001 • Crash of Flight 587 - November 12th, 2001 • Northeastern U. S. Blackout – August 14th & 15th, 2003 • Staten Island Ferry crash – October 15th , 2003 • One and a half hour partial 911 outage – March 26th, 2004 • Major Transit Strike - December 20th – 23rd 2005 • Blizzard, largest recorded snowfall in NYC- February 12th 2006

5. NYC’s Major Incidents –Effect on Telecomm (1) • Land line & cellular phone utilization peaks, exceeding network capacities. • Phone service spotty and extremely limited. • Challenge for critical communication for command and control. • 911 may not be accessible. • Government Emergency Telephone Service (GETS) may be inaccessible via cell phone.

6. NYC’s Major Incidents –Effect on Telecomm (2) • A prolonged power outage, further erosion of phone service occurs. • Cell sites lose power and fail over time 3-4 hours (100%UPS, 50%Generator Couplings, 10%Generators) • Public Safety Radio infrastructure fails (unless supported by UPS) • Non-copper based telephone infrastructure fails. Fiber optics premise based multiplexers • Cable television service fails. Radio stations w/o UPS go off the air. • VoIP service dependent upon operating non-copper based infrastructure fails • Premise equipment w/o battery backup fails – wireless phones, cable modems, routers.

7. The role of the CIO/CTO during power outages and events impacting infrastructure • What is the impact on public safety communications? • What is the impact on public telecommunications? • What do the city’s web site, 311, & 911 need to do? • What IT support needs to be provided for the emergency response? • What needs to be done to keep Municipal IT functioning?

8. Telecommunications - What is the impact, what is the status, what type of volume are is being handled, are there any issues/problems, any developing? • Are 911 systems operating normally? • Are the telephone systems operating? • Are cellular networks operating? • Are our public safety radio systems operating normally? • Have any critical Public Safety sites been effected? • Police/Fire/EMS sites, Command and Control – Cityhall, OEM, Hospitals, Key agency buildings and operational facilities • Have any sensitive locations/installations been effected? • Nuclear power plants, Chemical plants, refineries….. • Have any critical commercial sites been effected? • Exchanges, strategic industries, banks, alarms, surveillance, monitoring and security systems • Are cable systems up and operating?

9. Public Facing IT Utilities – What needs to be done? • 911, 311, 211 • How high is call volume, do we have enough call takers, is the next shift going to be able to get in to work? Supplement call takers? • What is the message, “What do we tell people?” • What are people telling us? Who needs to know? • Are there any new 311 functions that need to be brought up quickly? How quickly? What information do we need from callers, what is the process? • Web site • How is the web site performing? • What is the message/content? • Do we post an emergency home page (OEM) • Municipal Television • Is there information to go out on our crawls. • Do we want to facilitate press coverage by providing a pool feeds, where are the press conferences going to be held?

10. Municipal IT infrastructure What is the impact, what is the effect, how big a problem is it, should DR procedures be implemented? • Are data centers effected? Is there potential for integrity to be compromised? • Are networks operational? Is there a security threat? • Are any non-data center based systems adversely effected? • Are systems critical to support emergency operations up and running properly? • Web sites, GIS, E-Teams, CRM (311), Email, BES servers Have other technologies been adversely effected? • PBXs, Traffic Pattern Control systems, Building Control Systems

11. Providing IT support for the emergency - Wireless • Wireless • Does anyone need radios, satellite phones, cell phones, blackberries, EVDO? • Do we need to deploy additional batteries, chargers, accessories? • Is the radio infrastructure stable, deploy back up systems? • Are there Interoperability requirements? Deploy ACU 1000s, system level interoperability? • Do we need to deploy Cells on Wheels (COWS), Cells on Light Trucks (COLTS)? • Do we need to deploy temporary public pay phone banks? • How is the infrastructure holding up, any systems failing, Batteries, UPSs, Generators Operating, is fuel required?

12. Providing IT support for the emergency – General Technology • Is there a need for new office environments to be established, an EOC, field command centers, Family Assistance Centers? • Is there a need for credential systems? • Does anyone need laptops? • Is there mapping and GIS support required for press announcements, traffic control and logistical support? • Are there new applications that need to be brought up quickly? • Are there IT support staff in EOC’s, public safety agencies that need to be relieved?

13. Aftermath of the Attack on the World Trade Center – September 11th 2001 • Needed to Coordinate IT equipment requests, triage, stage, inventory, distribute, manage FEMA reimbursement applications, pay vendors. (coordinate and track donations) Lesson Learned – Develop procedures, establish Emergency IT Contract • Needed to build facilities rapidly • OEM Emergency Operation Center & Family Assistance Center • Backup Emergency Operation Center & Executive Offices Lessons Learned - LAN in a box. (Network equipment and Notebooks ready to deploy), Emergency IT Contract, pre-configure • Needed to coordinate the restoration of telephone service Lessons Learned – Strengthened Mutual Aide Restoration Consortium Agreements • Developed a GIS capability at OEM Emergency Operation Center • Implemented Emergency Resource Management Software • Developed Agency by agency business /continuity recovery plans • Centralize - City data centers and network fared very well, agency servers and networks failed

14. Crash of Flight 587 - November 12th, 2001 • Developed Mobile GIS capability • Rapid deployment of Family Assistance Center

15. Northeastern U. S. Blackout – August 14th & 15th, 2003 • Needed to have better control over the Maintenance of UPS’s and Motor Generators at some of our sites • Needed to strengthen our protocols with the Telco’s, needed insight into their Motor Generator maintenance and testing • 311 – Handled 172,000 calls in a single day. Harden 311. • 311 provided feedback from public • 311 helped to relieve 911, 911 experienced normal volume • Needed additional battery capacity for Public Safety Radio repeaters – spares, portable generators • Needed satellite phones and radios for executives • Needed to harden city hall • Data centers & network available, no users

16. Staten Island Ferry crash – October 15th , 2003 • Needed a system on hand at 311 to track missing, & casualties / next of kin.

17. One and a half hour partial 911 outage – March 26th, 2004 • 311 – received and routed 911 emergencies to precincts via non-emergency CRM workstations Lessons Learned – strengthen procedure to route 911 calls thru 311, ring down phones between NYPD/FDNY/EMS • NYPD had work around procedures for 911 outage • Painstaking change control procedures with Verizon • The highest levels of redundancy need to be implemented for 911

18. Major Transit Strike - December 20th – 23rd 2005 • Needed to develop telecommuting policies and capabilities • Needed a plan and the capacity to transport 911 and 311 call takers to work and back. • May have needed uniformed NYPD trained to handle 911 calls if 911 call taker attendance was too low. Needed to have NYPD officers trained. • The benefit of Call center capacity outside of the region began to be apparent • NYC 311 handled 243,000 calls in one day, normal 911 call volume

19. Blizzard, largest recorded snowfall in NYC- February 12th 2006 • The need for call center capacity outside of the region was reaffirmed.

20. All calls are answered by a live operator, 24 hours a day/7 days a week/365 days a year Provides immediate access to language translation services in over 170 languages Allows callers to quickly and conveniently: Be directed to a specific City, State or Federal agency or program Request detailed information about services and programs File a request for City services Professional, courteous, knowledgeable, accessible New York City 3-1-1 - Invaluable Everyday – Essential during Emergencies New Yorkers only need to remember two numbers to contact City government: 911 for emergencies and 3-1-1 for everything else.

21. 311 Providing Access to Govt.– Increasing call volume • 3-1-1 has increased the public’s access to non-emergency government services. • Since March 9, 2003, 3-1-1 serviced over 30 million calls, 15 Million/year • 3-1-1 averages approximately 47,000 calls per day. • Since its inception, the average number of calls per day has steadily increased • 3-1-1 has set of a service level of answering over 98% of calls in less than 30 seconds, no calls on hold beyond 3 minutes.

22. New York City --- Why 3-1-1 is necessary