Operational Recovery Planning

1. Operational Recovery Planning Presented by the California State Information Security Office

2. Agenda • Introductions – name and agency • CA State Information Security Office • Definitions • Four Types of Continuity Plans • Review of BL 07-03 – ORP Changes • ORP-COOP/COG Alignment • Discuss Test Scenarios

4. State Information Security Office • Vision • Leading the way to secure the State's information assets. • Mission • To manage security and operational recovery risk for the State's information assets by providing statewide direction and leadership.

5. Definitions • Emergency Response • Business Continuity Planning (BCP) • Operational Recovery Planning (ORP) • Continuity of Operations (COOP) • Continuity of Government (COG)

6. Emergency Response • The immediate reaction and response to an emergency situation commonly focusing on ensuring life safety and reducing the severity of the incident. • Definition from Disaster Recovery Journal (DRI) website at: http://www.drj.com/glossary/

7. Business Continuity Planning (BCP) • Process of developing and documenting arrangements and procedures that enable an organization to respond to an event that lasts for an unacceptable period of time and return to performing its critical functions after an interruption. Similar terms:  business resumption plan, continuity plan, contingency plan, disaster recovery plan, recovery plan. • Definition from Disaster Recovery Journal (DRI) website at: http://www.drj.com/glossary/

8. Operational Recovery Planning (ORP) • The management approved document that defines the resources, actions, tasks and data required to manage the technology recovery effort.  Usually refers to the technology recovery effort.  This is a component of the Business Continuity Management Program.  DISASTER RECOVERY PLAN (also known as – Operational Recovery Plan) • Definition from Disaster Recovery Journal (DRI) website at: http://www.drj.com/glossary/

9. Continuity of Operations (COOP) • The activities of individual departments and agencies and their sub-components to ensure that their essential functions are continued under all circumstances. This includes plans and procedures that delineate essential functions; specify succession to office and the emergency delegation of authority; provide for the safekeeping of vital records and databases; identify alternate operating facilities; provide for interoperable communications; and validate the capability through tests, training, and exercises. • Office of Emergency Services (OES)

10. Continuity of Government (COG) • The preservation, maintenance, or reconstitution of the institution of government. It is the ability to carry out an organization’s constitutional responsibilities. This is accomplished through succession of leadership, the pre-delegation of emergency authority and active command and control. • Office of Emergency Services (OES)

11. Relationship of Plans

12. Inter-Dependencies

13. Three Phases of Continuity Departments Emergency Response - Life Safety First 72 Hours Restoration Business back to normal IT Operational Recovery up to 30 days Planning, Documenting, Testing, and Training Business Recovery up to 30 days Damage Assessment First 72 hours Phase I Phase II Phase III

14. IMPLEMENTATION OF PLANS • Disruption of business occurs and you are informed, next steps 1. Emergency Response – safety and security of staff. 2. Securing the site. 3. Activate COOP/COG Plan to ensure the continuation of essential functions. 4. Implementation of the communication plan. 5. After assessing incident, determine if implementation of BCP & ORP is required. 6. Contact SISO to report incident. 7. Implement BCP and ORP

15. Budget Letter 07-03 • SAM Section 4843 – Operational Recovery Planning • Use results from risk analysis and business impact analysis to identify critical business functions. • Include the operational recovery considerations and costs in FSRs. • Develop ORP as part of a complete continuity program.

16. Budget Letter 07-03 – Continued • SAM Section 4843.1 – Agency Operational Recovery Plan • Rewritten to clarify and enhance operational recovery requirements. • Removal of minimum components from policy. • SIMM 65A – ORP Documentation for Agencies Preparation Instructions • Requires ten minimum components in ORP. • Additional three components for agencies without a BCP or COOP/COG.

17. ORP Documentation Revised • Components to be included in the ORP – updated in January 2007. • The April and July quarterly filers must provide a cover sheet indicating where the information for each topic area in SIMM 65A is located in the agency’s Operational Recovery Plan. • All components listed in SIMM 65A must be addressed and included in agencies’ ORPs beginning in October 2007.

18. Changes for ORP Development • Overall • Requires more details • New Components • Backup and offsite storage • Data Center Services • Contact information • Removed from SAM and Policy • Damage Recognition • Preparation of cost-benefit analysis • Selection of alternative • SIMM Section 140A

19. New Requirements • ORPs must describe: • Agency Administrative Information • Critical Business Functions/Applications • Recovery Strategy • Backup and Offsite Storage Procedures • Operational Recovery Procedures • Data Center Services • Resource Requirements • Assignment of Responsibility • Contact Information • Testing

20. Supplemental Requirements • Agencies that have not developed and implemented a full business continuity plan or COOP/COG must also address and include the following in their ORP: • Damage Recognition and Assessment • Mobilization of Personnel • Primary Site Restoration and Relocation

21. Agency Administrative Information • A communication plan should include strategy on: • How information will flow (escalation) • Decision making processes • Interrelationship among agency resources for response, recovery and resumption

22. Example - Escalation Process • Single site, minor impact. User calls into Help Desk with possible virus infection. Communication Plan strategy includes: • Process to dispatch field support to check PC • If infected, take steps to identify and quarantine • notify ISO and IT Management • Eradicate virus • Verify virus has not spread

23. What would you do? • Multiple site, major impact. The virus outbreak has spread from your headquarters to your remote offices and is running rampant. The anti-virus software will not eradicate it and all the systems in your agency are being impacted. What would your communication plan need to include?

24. Communication Plan • Document • Who to contact and under what circumstances • Lists name, phone #, cell #, home #, email address • Includes Chain of Command Management, other pertinent staff (ISO, ORP Coordinator, etc), and contractors • Distribute to applicable staff • Providing training to staff • Collect when duties change or staff leaves

25. Sample Call Lists • Wallet size cards: • Name, work #, cell #, home #, email • Call Tree: • Manager calls supervisor • Supervisor calls his/her staff

26. Critical Business Functions/Applications • This section includes a description of: • Critical business functions and their supporting applications • Maximum Allowable Outage (MAO) for each application • Recovery priorities

27. Example - Critical Business Function • Single site, minor impact. Help Desk identifies that the services on the email server are not working. As a critical business function, recovery strategy includes: • Process for IT staff to check services • If denial of service, follow internal procedures to identify and mitigate. • Notify ISO and IT Management

28. What would you do? • Multiple site, major impact. The email server has crashed, there are both hardware and software failures. Rebuilding the server will require replacement hardware, which will take several days to acquire and configure. What would your Critical Business Functions / Applications need to include?

29. Procedures for Critical Functions • Document • Critical Business Functions • Recovery Procedures • Responsible individuals or team for recovery • Distribute procedures to applicable staff • Provide training

30. Sample Procedure • Repair/replace hardware • Restore database structure • Restore post office • Restore gateway connectivity • Rebuild database • Keep users/management informed

31. Recovery Strategy • Recovery strategy should include alternate recovery site/sites that include: • Location of all sites • Requirements of facilities/equipment • Contact numbers

32. What would you do? • Single site, minor impact: Your department is located in several locations. A building adjacent to one location has a fire, the fire did not spread to your site. The Fire Dept and Law Enforcement block the street, so there is no access into your building. What would your recovery strategy need to include?

33. Recovery Strategy • Communication plan for employees, management, and contractors. • List all office locations. • Identify the alternate location. If multiple locations are available, prioritize them. • Address what functions could be restored at each site. • Determine who would need to be called, include as the contact list.

34. Sample Recovery Strategy • Department has three locations: • 1234 Headquarters St., Sacto, 95814 • 5678 Anywhere St., Sacto, 95825 • 9876 SomePlace St., LA 90210 • Critical operations would be restored at an unaffected site (identify priority and equipment needed). • Contact: • J Resto at (916) 555-1212 for Headquarters • R Quick at (916) 444-1212 for Anywhere • M Pia at (213) 555-1212 for SomePlace

35. Backup and Offsite Storage • The backup and offsite storage procedures should include: • Retention schedule • Procedures • List of authorized staff • Account information • Contacts of offsite storage

36. What would you do? • The data on one of your critical applications was corrupted and its MAO is 4 hours. It is 5:30 pm on Friday and Monday is a holiday. The business area have staff scheduled to work Saturday on this system. Technical staff has gone home, and several are out of town for the weekend. What would your backup and offsite storage procedures need to include?

37. Details – Backup and Offsite Storage • Document: • Retention schedule • Detailed procedures • Hardware and software (include version) • Offsite storage details (location, acct #) • Retrieval of backups (contacts (24x7) and personnel authorized to retrieve) • Process to identify data to be restored

38. Operational Recovery Procedures • These procedures systematically detail the operational procedures for recovery in a timely and orderly way, they should include: • Detailed procedures that the backup or other IT professional could follow • High-level network diagram that includes all critical applications

39. Data Center Services • This section should include a: • Description of service to be provided. • Interagency agreements, memorandums of understanding, or contracts. • Specific coordination efforts with the data center critical to the recovery efforts.

40. Example – Minor Impact • Single site, minor impact. Your Web server providing access to one of your critical applications located at DTS has been compromised. You have contacted DTS and DTS is working to get the server back online within the hour. What would your need to include?

41. What would you do? • Multiple site, major impact. There was a fire in a facility adjoining DTS facility where the servers are housed. The sprinkler system was activated and the servers had to be powered down. There is significant water damage. There is an estimate that it will take 14 to 21 days to reestablish services. What would your plan need to include?

42. Details - Data Center Services • Expectations • Meet with Data Center to identify • Hardware/Software requirements • Services required • Timeframe for services • Document Agreement – Before it’s needed • Create a Service Level Agreement (SLA) or Memorandum of Understanding (MOU) • Develop Recovery Procedures

43. Resource Requirements • This is a comprehensive list of: • Equipment • Software • Telecommunication needs • Data • Hard copy manuals • Personnel essential for recovery

44. Assignment of Responsibility • Designation of responsibilities and assignments should be listed. Procedures should include job title, and not individual names, for the recovery process. • Individuals names can be placed in a single location for ease of maintenance.

45. Contact Information • There are two types of contact information to be collected: • Employees, including management. • Resource List including contractors, Major Service providers, vendors, other government entities, and outside resources critical to the recovery process.

46. Contact List • Employee contact information should be designated as sensitive, and provided to authorized individuals. • Resource lists typically have business contact information. This information can be provided more widely.

47. Testing • Annual testing of the ORP is essential to: • Ensure for training the management and recovery teams. • Validate that the procedures have the appropriate level of detail. • Verify Call Back lists are current. • Confirm that Recovery strategies are appropriate for your environment.

48. Governor’s Office of Emergency Services • Introduction • Mission and Goals of OES • SEMS/NIMS • Disaster Service Worker

49. Planning • Be Smart, Be Responsible. Be Prepared. Get Ready Campaign • Your Intranets and Emergency Preparedness • Executive Order S-04-06 • State Emergency Plan / COOP-COG/ORP

50. Training and Testing • Emergency Management Training Requirements for Public Employees • The California Specialized Training Institute (CSTI)/OES Training Branch • How to develop a Table Top Exercise (TTex) • Definition of a TTex • The 8 Step Process Used to Design a TTex • After Action/ Corrective Action Process • California Master Exercise Calendar (CMEX)