New Techniques for Restoring Telecom after Disasters & Outages

1. New Techniques for Restoring Telecom after Disasters & Outages Mark Reynolds, University of New Mexico

2. Agenda UNM Telecom Network UNM “Old” Recovery Plan The 6 Vulnerabilities Old Plan Vulnerabilities UNM “New” Recovery Plan New Plan Vulnerabilities Summary

4. The Problem Need to protect… Incoming local DIDs Incoming toll-free numbers Outbound dial tone …from major disasters and minor outages

5. 1970 1975 1980 1985 1990 1995 2000 2004 Droughts Floods Earthquakes Epidemics Slides Volcanic eruptions Wild fires Wind Storms Insect infestation Source: EM-DAT : The OFDA/CRED International Disaster Databasehttp://www.em-dat.net, UCL - Brussels, Belgium

6. Uh-oh…

7. Common Causes of Phone Outages Equipment failure Power outage New Installation Moving Water leak Animals Cable cut Wiring short “Upgrades” Technician mistake Winter storm Accidents

8. October 17, 1989 4:00 PM

9. October 17, 1989 6:00 PM

10. Silicon Valley, April 9, 2009 1:30 AM – Vandals sever major fiber backbone with hacksaws AT&T, Verizon, Sprint down Landlines, cellular, internet down No 9-1-1 service, no credit cards Lasted 14 hours

11. Pandemic Quarantine United States H1N1 stats of 2009: 98,000 hospitalizations 3,900 deaths What if the next strain is more severe? What if some/most/all employees must work from home?

12. UNM’s 6 Vulnerabilities Qwest Local Qwest LD

13. Vulnerability 1: Equipment Failure Qwest Local Qwest LD

14. Vulnerability 2: Power Qwest Local Qwest LD

15. Vulnerability 3: Last Mile (cable cut) Qwest Local Qwest LD

16. Vulnerability 4: Qwest Local failure/congestion Qwest Local Qwest LD

17. Vulnerability 5: Qwest LD failure/congestion Qwest Local Qwest LD

18. Vulnerability 6: Evacuation / Quarantine Qwest Local Qwest LD

19. UNM’s 6 Vulnerabilities Equipment failure Power outage Last mile (cable cut) Qwest Local failure/congestion (affects local #s) Qwest LD failure/congestion (affects toll-frees) Evacuation/pandemic quarantine

20. UNM “Old” Recovery Plan Qwest Dept. Cell Phones Qwest BCR Hospital (505) 272-2411

21. UNM “Old” Recovery Plan Good: Can divert calls after any failures except carrier failures and congestion Not so good: Calls go direct to cell phones, no transfer, no greeting, no centralized voicemail, no attendant, etc. Zero control, flexibility; “one and done” No plan for recovery if Qwest can’t forward Can’t recover outbound dial tone

22. UNM New Recovery Plan Qwest Hospital (505) 272-2411

23. UNM New Recovery Plan Qwest Hospital (505) 272-2411

24. UNM New Recovery Plan Telecom Recovery Qwest Hospital (505) 272-2411

25. UNM New Recovery Plan Telecom Recovery Internet Qwest Hospital (505) 272-2411

26. UNM New Recovery Plan Telecom Recovery Internet Qwest Hospital (505) 272-2411

27. UNM New Recovery Plan Telecom Recovery Internet Qwest Hospital (505) 272-2411

28. UNM New Recovery Plan Telecom Recovery Internet Qwest Hospital (505) 272-2411

29. UNM New Recovery Plan Telecom Recovery Qwest Qwest BCR Hospital (505) 272-2411

30. UNM New Recovery Plan Good: Incoming calls get back into Hospital PBX If PBX fails, calls get to same employees on cell, sat, home, other locations, recordings, etc. Real-time control, unlimited flexibility Backup dial tone for outbound calls Not so good: Local numbers down if Qwest is down (toll-free numbers would still be recoverable)

31. The 13 Most Critical Mistakes Underestimating the pain of outages “Scream & Wait” Confusing outbound with inbound Confusing disasters with outages Neglecting vulnerabilities Inadequate mass communication Inadequate control

32. The 13 Most Critical Mistakes No satellite phone Insufficient capacity Reliance on hot site Putting backup equipment on-premise Lack of testing Spending too much

33. 10 Low-Cost Improvements Twitter/Facebook Skype/MagicJacks Obtain hand-held satellite phone(s) Create voluntary reimbursement agreements with employees Gather alternate phone #s, email

34. 10 Low-Cost Improvements Create Emergency section on website Publish a toll-free hotline Retain some POTS (copper) lines, wired outside the phone system Establish “Remote Access to Call Forwarding” with carrier Obtain GETS/WPS cards (www.ncs.gov/services.html)

35. FAQs What if we already have a backup PBX? What if we already have a SONET ring, SHARPS, SHINS, etc.? What if we already have a mass notification system? What if we already have cell phones, radios, satellite phones, etc.?

36. FAQs What if the carrier is also down, and can’t forward calls to you? Neustar/Port DR We need your help in pressuring carriers to adopt Port DR! Advantages of toll-free over local Still need numbers loaded in a backup carrier Still need a place to send the calls Backup satellite T1 (for delivery & outbound)

37. Backup Satellite T1 Off-site Control system Public Phone Network

38. Backup Satellite T1 Off-site Control system Public Phone Network

39. Backup Satellite T1 Off-site Control system Public Phone Network

40. Backup Satellite T1 Off-site Control system Public Phone Network

41. Backup Satellite T1 Off-site Control system Public Phone Network

42. Backup Satellite T1 Off-site Control system Public Phone Network

43. FAQs What if the carrier is also down, and can’t forward calls to you? What’s Telecom Recovery’s disaster recovery plan? How much does this cost?

44. What Next? Determine what next steps to take. Identify people involved in making and approving a plan. Establish a project to implement the plan. Communicate and test the plan.

45. What are the questions? What are the important numbers at the institution that should function during a disaster? How will calls to the numbers identified above be handled? Who would authorize and activate the emergency routing of numbers? Who would administer and maintain the information?