Managing Failures in the PSAP: A View From Inside the Events A review of the events, lessons learned, and preventative m

1. Managing Failures in the PSAP: A View From Inside the EventsA review of the events, lessons learned, and preventative measures Keith Godwin Alachua County E911 Coordinator

2. The Sources Construction Maintenance UPS failures Power interruptions and outages CPE Service providers equipment failures Downed power lines

3. Critical Systems in a PSAP Radio CAD Telephone; GIS; ANI/ALI Recording UPS Generator Power

4. Critical Systems in a PSAPIf you were concerned about the systems that support your PSAP, what questions would you ask? When does the warranty expire? What is the cost for extended warranty? Is there a maintenance agreement? When does it expire? What is the backup plan for a failure? Is the plan in writing? Is the plan exercised periodically?

5. The Events November 2005 2006 Wednesday, February 11th 2009 Friday, March 6th 2009 Sunday, March 29th 2009 Tuesday, March 31st 2009 Friday, May 15th 2009 Tuesday, May 26th 2009

6. November 2005 UPS maintenance agreement expired Power outage resulted in the loss of all systems CAD, radio, recording, phone Backup generator came on but power to all these systems was interrupted thus shutting them all down at once

7. 2006 TRS went into failsoft No plan No procedures No practice No protocols Pure panic

8. February 11, 2009 Backhoe cuts Windstream fiber 7,000+ Wireless, wireline, public safety communications (20%) 9 hours Call takers X4 & 1 MCV DRP not current

9. March 6, 2009 Momentary hiccup All phones momentarily blinked off UTA what else blinked? (very important information!) AT&T and CenturyLink UTL problem

10. March 29, 2009 Power outage in area due to thunderstorm 911 and administrative lines impacted CAD & radio up Backup PSAP occupied for 911 call taking Dispatch remained at primary PSAP Administrative lines up at primary PSAP

11. March 31, 2009 Power outage in area due to thunderstorm 911 and administrative lines impacted CAD and radio up Backup PSAP occupied for 911 call taking Dispatch remained at primary PSAP Administrative lines up at primary PSAP

12. May 15, 2009 Power outage in area due to thunderstorm 911 and administrative lines impacted CAD and radio up Backup PSAP occupied for admin and 911 call taking Dispatch remained at primary PSAP No administrative lines in PSAP

13. May 26, 2009 Power outage in area 911 and administrative lines impacted CAD and radio up Backup PSAP occupied for admin and 911 call taking Dispatch remained at primary PSAP No administrative lines in primary PSAP

14. And the culprit is… UPS Batteries

15. The Bottom Line You do not have enough funding to offset every possible outage…but you should conduct a vulnerability assessment and determine the most affordable means available to reduce your exposure to failures and be in constant pursuit of hardening your facility

16. Lessons Learned • Annual Letters • DRP’s/PSAP Profile • Redundant Telephone Routes • Redundant Electric Power Routes • Critical & Important Tickler • Important/Emergency Contacts & Escalation Lists • PSAP Evacuation Kit • Backup Power • Problem Reporting • CPE Maintenance & Maintenance Agreements • Administrative Phone Lines • TSP • PSAP Information • Backup PSAP • Alternate Means of Communication

17. End Office Isolation Letter

18. Disaster Relief Plan / PSAP Profile From AT&T From Windstream

19. Redundant Telephone Routes

20. Redundant Telephone Routes

21. Creating a Redundant Telephone Route: $7,300Free Labor: Priceless

22. Alachua County PSAP

23. New AT&T Equipment Bay in PSAP Telco Room

24. Redundant Electrical Power Routes

25. Critical & Important(SharePoint)

26. Critical & Important

27. Important/Emergency Contacts & Escalation Lists • Update annually (minimum) • Intrado, CenturyLink, AT&T, Windstream, WSP’s • Where do you maintain this information? Who is responsible for updating it? • Blackberry / Cell Phone • Flash Drive • Updated quarterly by E9-1-1 Office & distributed to PSAP supervisory staff • Wallet • PSAP Supervisors • PSAP Operations Book

28. E9-1-1 Coordinator Emergency Contact Card

29. PSAP Evacuation Kit (PEK)

30. PSAP Evacuation Kit

31. PSAP Evacuation Kit

32. PSAP Backup Power*Transfer Switch Maintenance

33. PSAP Backup Power

34. PSAP Backup Power

35. “Tell me exactly what happened” • PSAP supervisors • The key • Ticket numbers: (nothing happens without them) • Notify surrounding counties before and after • Where do you record everything associated with the event? • CAD • Office document • Demand detailed notes (cannot provide too much information) • Times • What happened?; what is not working?; what is working? • Whom did you speak to? • What company are they with? • What did they say / going to do? TELL THEM TO CALL YOU BACK WHEN ANYTHING CHANGES • CB numbers

36. Equipment Problem/Failure Report

37. CPE Maintenance & Maintenance Agreements Backup power UPS Telephone Recording CAD Radio

38. Administrative Telephone Lines • Watch Cmdr; DC’s • Well-being Checks • Other PSAP’s • PSAP Management • Staffing • ER’s • Medical Examiner • PIO • PSAP Maintenance • Radio, HVAC, Plumbing, IT • JUST AS IMPORTANT AS 9-1-1 LINES

39. Telecommunications Service Priority (TSP) Eligible circuits include 9-1-1 voice, admin lines, radio circuits, end office to central office circuits Is not free TSP designated facilities have services restored first

40. PSAP Information PSAP ID used by LEC PSAP ID # used by service provider

41. Backup PSAP Exercise it Train in it It should not be in the same bursting radius of your primary PSAP Don’t have one> be creative (Jail, Ed Ctr, municipal LE agency)

42. Alternate Means of CommunicationsCommunicating with the Deployed • WebEOC messaging • Text messaging • Cellular px • Internet • Satellite px

43. Actions

44. Prevention Cycle

45. Thank Youkgodwin@alachuacounty.us352-338-3285