Let’s Get Real : Disaster Recovery and Business Continuity in Public Safety

1. Let’s Get Real: Disaster Recovery and Business Continuity in Public Safety Is Yours Just a Paper Plan or a Real Way to Prepare and Respond to Incidents and Disasters?

2. Presentation Overview • Key DR/BC Concepts and Issues • Report card and dashboard • Scenarios • Requirements: What has to operational by when for work to be done by how many at what locations serving what customers who are where? • Facilities • People • Systems • Integration • Coordination • Daily readiness and simulated escalations • Testing and independent verification and validation • Implementation and triage • Recovery, discovery, and improvements • Player Scorecard: Who Is In the Game and Why? • DR/ BC Framework • Action Steps to a Real Plan • First steps • Critical functions • Funding and leveraging scarce resources • Think out of the box • Integration with the big picture DR/BC plan and activities of your jurisdiction • Conclusions

3. Key DR/BC Concepts and Issues

4. The Report Card and Dashboard • All aspects of the plan, test, and implementation should be scored simply (Red, Yellow, and Green) • Key indicators of planning and readiness need a dashboard to enable assessment and action • Score or status • Trend • Key issue

5. Public Safety Scenarios • Public safety entities have a more difficult challenge • Your IT DR/BC plan is intertwined with risk scenarios • You may be affected by the risks of a given scenario and your IT plan must address those risks appropriately to maintain operations • You also have a role in response to the scenario so the events will affect your operational requirements

6. Scenarios Overview • Threat driven geographic circles of impact • Kinds of threats and events • Responsibility • What will you do, what is shared, what do others have to do for themselves • Tolerance for risk and uncertainty • Lesson learned: if you have a well known and documented local risk: • Have a real plan or get ready for a career change…

7. Source: IBM

8. Scenarios • Identify Possible and Likely Natural Disasters and Environmental Conditions By Kind and Duration of Effects • Tornado • Hurricane • Tsunami • Flood • Snowstorm • Drought • Earthquake

9. Scenarios • Identify Possible and Likely Natural Disasters and Environmental Conditions By Kind and Duration of Effects • Electrical storms • Fire • Subsidence and landslides • Freezing Conditions

10. Scenarios • Identify Possible and Likely Natural Disasters and Environmental Conditions By Kind and Duration of Effects • Contamination, Toxic releases and environmental hazards • Epidemic • Pandemic • Animal or crop disease outbreak

11. Scenarios • Organized and/or Deliberate Disruption • Act of terrorism • WMD • Acute and short lived (bomb) • Acute and long lived (dirty bomb) • Chronic • Long term (contaminants and biohazards) • Permanent (radioactivity, etc.) • WLD (suicide bombers, car bombs, utility sabotage) • Bioterrorism or genetically modified or inorganic organisms • Direct contact • Infectious • Contact • Airborne

12. Scenarios • Organized and/or Deliberate Disruption • Act of Sabotage • Product or food tampering • Act of war • Theft • Arson • Labor Disputes / Industrial Action

13. Scenarios • Loss of Utilities and Services • Electrical power failure • Loss of gas supply • Loss of water supply • Petroleum and oil shortage • Raw materials • Refined materials • Communications services breakdown • Loss of drainage / waste removal and trash pickup

14. Scenarios • Equipment or System Failure • Internal power failure • HVAC failure • Equipment failure (excluding IT hardware)

15. Scenarios • Serious Information Security Incidents • Cyber crime • Malware • Zombie attacks • Denial of service • Loss or alteration of records or data • Disclosure of sensitive information

16. Scenarios • IT system failure (local or hosted) • Hardware • Software • Commercial application • Locally developed application • Data • Communications

17. Scenarios • Other Emergency Situations • Workplace violence • Public transportation disruption • Neighborhood hazard • Health and safety issues

18. Scenarios • Multiple and compound hazards and events • Purposeful • Coincidental • Causally connected • Interrelated

19. IT Requirements • What systems need to function • How fast • Maximum and optimum time frame for each system or function to be restored • How well • Sometimes minimal functionality is sufficient

20. IT Requirements • Where will it be used and by whom and will the communications infrastructure support it? • Employees • Users or beneficiaries • By what priority will systems be restored • The priority will be modified by what contingencies • E.g. a long term total evacuation changes the operational needs for criminal justice systems and personnel

21. Facilities • Hot, warm, cold • Mirrored, recoverable, reload-able • Properly located • EOC • Non-EOC • Operational • IT facilities • For user interaction with IT systems

22. Facilities • New kinds of mutual aid and sister city/county/state arrangements • Work with friends, colleagues, associations, and vendors • To match you with a comparable entities that are located outside the various geographic threat circles • Who can mirror your IT operations (hardware, software, operating systems, and culture)

23. People • The right numbers, skills, location, redundancy, etc. • Skills and abilities inventory • Employees • Contractors • Vendors • Mutual aid and “the cavalry”

24. People • Force in depth—who is the backup to the backup to the backup? • Consider the actual health and physical abilities and disabilities of a person when assigning tasks for a disaster scenario • The disaster is not the time to find out the electrician in the hazmat suit has a heart condition • What family and personal duties may interfere with performing official duties (e.g. save your own kids or save a stranger)?

25. Systems • Daily operational • Interdependent systems • Emergency only • Identity security and access management for physical and logical security • Follow FIPS 201 for federal/state/local interoperability

26. Integration • With whom should you work closely? • Identify integration issues between: • Internal systems and public safety entities • Other governmental systems • Related actors • Non-governmental systems and processes • Example: 911 and 311or its equivalent • Normally separate but related • Emergencies blur the line • Co-location, cross training, and system integration

27. Coordination • Within organization • Within unit of government • Across units of government • Across levels of government • Across public and private boundaries

28. Daily Readiness and Simulated Escalations • A disaster a day (“What, that’s not normal?”) • Realistic scenarios • Captured lessons • Learning and actually responding to lessons learned within risk framework • A quality and security framework for daily operations has substantial overlap with DR/BC

29. Security Capabilities Models Security Leadership Security Sponsorship Security Strategy Security Program Security Program Structure Security Program Resources and Skillsets Security Policies Security Policies, Standard and Guidelines Security Management Security Administration Security Monitoring User Management User Management User Awareness Information Asset Security Application Security Database / Information Security Host Security Internal Network Security Network Perimeter Security Technology Protection and Continuity Physical and Environment Controls Contingency Planning Controls Causes Strategy Management Knowledge Technologies Support Effects Like similar capability models from the Carnegie Mellon SEI, SCMM models brings benefits: • Helps close security holes • Serves as a foundation for growth • Guides security leadership • Is evolutionary, not chaotic • Supports point solutions KPMG SCMM Model

30. Capability Maturity Continuously improving process Optimizing (5) Managed (4) Predictable process Standard, consistent process Defined (3) Repeatable (2) Disciplined process Initial (1) Informal process Like the SCI CMM models, the KPMG Security Capability Model has five levels of maturity:

31. Testing and Independent Verification and Validation • Does the planned response or action step actually work? • Who verifies that it does? • What do you do if it fails the test?

32. Implementation and Triage • Someone better be in charge • Dispute resolution processes • Who will be your Sensibility and Sanity Checker (off site, not affected by the disaster, and actually getting enough sleep to make sound decisions)? • Baton Rouge example with Mayor Holden

33. Recovery, Discovery, and Improvements • What will the new normal be and when will it happen • Learn from history, both recent and long past • Document while the event occurs if at all possible (make it someone’s job) or soon after before memories fade

34. Player Scorecard Who Is In the Game and Why

35. Overlapping and Inter-Related Responsibilities Disaster Preparedness and Recovery and Business Continuity Physical Security Quality Assurance Methodologies Public Safety Cyber Security

36. The Usual Suspects in Public Safety • Police • Fire • Other sworn officers (transit, game, building or branch based, etc.) • National Guard • Public Health • Public Works • Transportation • Environmental Protection

37. The Usual Suspects in Emergency Management • Federal, state and local emergency management entities • National Guard • NOAA, NWS, NSSL, other National Laboratories, • Corps of Engineers

38. IT Entities • CIO, CTO, and Enterprise IT Shops • Distributed IT Departments and leadership • Government IT contractors • DR/BC specific entities • Applications developers and software • Hardware • Service providers (ASP, MSP, call centers, etc. • Communications providers

39. Policy Makers • Executive, legislative, and judicial • Those who hold the seat and those who actually make the decisions… • Go below the top level to ensure clarity, alignment, and redundancy • EOC designees • Emergency authorizers

40. Non-Governmental Organizations • Media • Broadcast and satellite • Emergency Broadcast System Members • Print • New media • The Web • Government site mangers • Commercial site managers • Citizens and bloggers • Self-organizing communities (e.g. Craig’s List)

41. Non-Governmental Organizations • Charities • Businesses and business associations • Community organizations • Vital private services (hospitals, nursing homes, etc. )

42. A DR/BC Framework

43. Business Operations and Technology • Create a matrix, not a linear or organizational view • Strategy • Organization • Processes • Applications and data • Technology • Facilities

44. Source: IBM

45. Action Steps to a Real Plan

46. First Steps

47. First Steps • Leadership: clarity, alignment, and commitment • Authority or consensus? • Stakeholders roles and responsibilities • Be clear about risk tolerance • Applications and IT assets inventory • If needed, dust off and update your Y2K work • Good data on plan status, readiness, test results, response, and compliance

48. First Steps • Make a friend in accounting—actuarially accurate threat scenarios are more likely to be funded as risk and cost can be properly balanced • Review existing plan or make a plan • Borrow or buy a template • Review peer plans and conduct site visits • Communicate until it hurts

49. Critical Functions

50. Nail Down Your Critical Functions • Law and order essentials (people, mobility, tools, survival basics, etc.) • Communications • Personnel management (policies, scheduling, notification trees and systems, counseling, etc.) • Data and the connections to data and people • Transactional systems