330 likes | 480 Views
Lessons Learned. 2,000 Successful Recoveries . Get the right information. To the right people. At the right time!. Jim MacMicking. Director of Professional Services. Austin, Texas. Katrina. May We Learn From The Mistakes of Others!. Katrina’s Aftermath.
E N D
Lessons Learned 2,000 Successful Recoveries Get the right information To the right people At the right time!
Jim MacMicking Director of Professional Services Austin, Texas
May We Learn From The Mistakes of Others!
Katrina’s Aftermath • No landline phone service • No cellular service • No power • No drinking water • No sanitation • No primary or secondary roads • No fuel – gasoline or diesel • No lodging • No security Within an area of 75,000 square miles!
Katrina’s Aftermath • 500,000 families without homes or jobs • 40,000 military personnel forced to respond • Hazmat contamination throughout the area • 2.8 million gallons of oil spilled • 100 broadcast agencies knocked off the air • 25 hospitals closed • 80,000 businesses wiped out • 4,000,000 people directly impacted • No customers or taxpayers • No revenue or income • No business or government!
The PAST!
Business Continuity Disaster Recovery The best approach existing at the time! Worst Case
- Building Fire - Statistically Speaking … The probability of your building suffering a “smoke and rubble” disaster are minute
- Terrorist Bombing - Statistically Speaking … - Building Fire - 73% of all terrorist acts are bombings 75% of the targets are business related buildings
Hurricane with Landfall in Texas? Statistically Speaking … - Building Fire - - Terrorist Bombing -
100 % Statistically Speaking … - Building Fire - - Terrorist Bombing - Hurricane with Landfall in Texas?
As a point of reference … Of the 10,000+ clients of SunGard An average of 1 in 80 declared a disaster in 2003, 2004 and 2005
People Never take them out of the equation! Staff will not participate if their family is in harms way Temporary housing will be in extremely short supply Pets are family and most hotels will not take them Parents, grandparents and elderly relatives must be safe Alternate staff may be a requirement, but may refuse
1980 - 2,750,000 Greater Houston Population has doubled since Alicia in 1983 2005 - 5,300,000
Hurricane Watch You can’t wait for the Warning! Resources will be in short supply Tape storage vendors will stop making shipments Interstates turn into miles upon miles of parking lot Airlines will be filled and airports converted after landfall If you delay, you may not have an option Do you ask your staff to stay?
After Landfall It’s not the plan, it’s the planning! Emergency services will come first Common resources may not be available Unusual resources will be hard to find, premium items Communication, communication, communication Cash and/or corporate credit cards must be available Do you ask your staff for credit?
Recovery Most organizations did not achieve the RTO! IMT must have a presence plus active command & control System and network configuration requirements must be accurate and complete before the storm Commercial providers will cautiously accept and assign Business staff must have a location, but will not travel far Training and exercises are essential parts of company survival and should be considered a necessity of business What happens if you don't train?
Results of Inadequate Training Groups that had not trained together did not work as a team Individual roles and responsibilities remained unclear and confusing Inability to designate an appropriate leader A few individuals called the shots, often to the detriment of the group No process for sharing information; vital information was containerized Opportunities were lost when team members did not listen to each other or follow the plan No problem solving process for analyzing information: Breakdown in the ability to assess the situation Priorities remained unclear and no priorities were established Under pressure and stress, no unified strategy was ever developed Simple tasks took 4-5 times longer to perform then expected Promises to stakeholders could not be met
Maturity Level • Level 1 - Fully Exposed Ad hoc process – making it up as you go • Level 2 - Significantly Exposed Successes are repeatable, but not documented and/or knowledge is limited to a few select staff members • Level 3 - ProgramImplemented Processes are tailored from a standard framework and all potential staff have participated in an exercise! • Level 4 -Managed Command and Control Capable of tracking an incident and measuring success • Level 5 -Optimized Program Capability improvement – Change management
8% Level 1 Fully Exposed 37% Level 3,4,or 5 Implemented, Managed or Optimized 55% Level 2 Significantly Exposed Perceptions of Chief Financial Officers Source: Association of Finance Professionals
Incident Command and Control Human and Physical Attributes Physical access to facilities People/employees Protecting intellectual capital Public infrastructure (public services) People communications Crisis and event management Organizational buy-in/culture Internal and Business Processes Workflow interdependencies Change management Supply chain interdependencies Budget Integrating new processes and services Non-automated processes Continuity planning Insured versus self insured Technology and Information Architectures Applications and software Hardware Network Storage Architecture security Information security Technical resources SLA commitments Database Environmental Market and External Drivers Business climate and insurance influences Business needs Regulatory issues Practical technology solutions Outside threats/risks Patient demands Pillars of Availability
Incident Management Tactical Operations-level management personnel who provide organization-wide support to the response effort and interface with corporate management, government agencies, media, and the public.
Operations Section Chief Individual with best knowledge of affected operations/area and leadership/management skills Damage Assessment Team/ Facilities Team Activated for incidents affecting the facility On-Scene Commander Incident Management Team INCIDENT COMMANDER Executive or senior manager with broad knowledge of company operations and strong leadership skills Incident Commander Executive or senior manager with broad knowledge of daily operations and strong leadership skills. COMMAND STAFF PR/Information Officer Investor Relations Legal Advisor Environmental Affairs Safety/Security Advisor Human Resources Advisor IS Liaison Planning Section Chief Logistics Section Chief Solid knowledge of SLAs; management, delegation skills Finance Section Chief Strong project management skills, very detail oriented; able to coordinate well with other functions Responsibility for overall finance and accounting operations Purchasing Unit Leader Procurement & financial background/skills Insurance Unit Leader Insurance/claims processing and audit experience Situation Unit Resource and information tracking, posting, publishing – good communications Business Resumption Branch Director Responsible for liaising with business units for incidents requiring activation of business continuity plans Supply Unit Leader Vendor relationships/multi-tasking and order processing skills Accounting Unit Leader Financial Analyst/ A/R and A/P experience Documentation Unit Records management skills; very organized Transportation Unit Ldr Travel and accommodations. Liaison with travel agency
Operations Section Chief Individual with best knowledge of affected operations/area and leadership/management skills Damage Assessment Team/ Facilities Team Activated for incidents affecting the facility On-Scene Commander Incident Management Team Command Staff Individuals with specialized knowledge that act in an advisory capacity to the Incident Commander to make decisions on approach to mitigate impacts, respond to and resolve the incident. INCIDENT COMMANDER Executive or senior manager with broad knowledge of company operations and strong leadership skills COMMAND STAFF PR/Information Officer Investor Relations Legal Advisor Environmental Affairs Safety/Security Advisor Human Resources Advisor IS Liaison Planning Section Chief Logistics Section Chief Solid knowledge of SLAs; management, delegation skills Finance Section Chief Strong project management skills, very detail oriented; able to coordinate well with other functions Responsibility for overall finance and accounting operations Purchasing Unit Leader Procurement & financial background/skills Insurance Unit Leader Insurance/claims processing and audit experience Situation Unit Resource and information tracking, posting, publishing – good communications Business Resumption Branch Director Responsible for liaising with business units for incidents requiring activation of business continuity plans Supply Unit Leader Vendor relationships/multi-tasking and order processing skills Accounting Unit Leader Financial Analyst/ A/R and A/P experience Documentation Unit Records management skills; very organized Transportation Unit Ldr Travel and accommodations. Liaison with travel agency
Operations Section Chief Individual with best knowledge of affected operations/area and leadership/management skills Damage Assessment Team/ Facilities Team Activated for incidents affecting the facility On-Scene Commander Incident Management Team Section Leaders Specialized units that focus on coordinated, organization-wide tasks. Both strategic and tactical. INCIDENT COMMANDER Executive or senior manager with broad knowledge of company operations and strong leadership skills COMMAND STAFF PR/Information Officer Investor Relations Legal Advisor Environmental Affairs Safety/Security Advisor Human Resources Advisor IS Liaison Planning Section Chief Logistics Section Chief Solid knowledge of SLAs; management, delegation skills Finance Section Chief Strong project management skills, very detail oriented; able to coordinate well with other functions Responsibility for overall finance and accounting operations Purchasing Unit Leader Procurement & financial background/skills Insurance Unit Leader Insurance/claims processing and audit experience Situation Unit Resource and information tracking, posting, publishing – good communications Business Resumption Branch Director Responsible for liaising with business units for incidents requiring activation of business continuity plans Supply Unit Leader Vendor relationships/multi-tasking and order processing skills Accounting Unit Leader Financial Analyst/ A/R and A/P experience Documentation Unit Records management skills; very organized Transportation Unit Ldr Travel and accommodations. Liaison with travel agency
Operations Section Chief Individual with best knowledge of affected operations/area and leadership/management skills Damage Assessment Team/ Facilities Team Activated for incidents affecting the facility On-Scene Commander Incident Management Team Operations Section Chief Pre-assigned individual with the best knowledge of the affected operations and functions. Must have strong leadership skills. “Yes Men” may not apply! INCIDENT COMMANDER Executive or senior manager with broad knowledge of company operations and strong leadership skills COMMAND STAFF PR/Information Officer Investor Relations Legal Advisor Environmental Affairs Safety/Security Advisor Human Resources Advisor IS Liaison Planning Section Chief Logistics Section Chief Solid knowledge of SLAs; management, delegation skills Finance Section Chief Strong project management skills, very detail oriented; able to coordinate well with other functions Responsibility for overall finance and accounting operations Purchasing Unit Leader Procurement & financial background/skills Insurance Unit Leader Insurance/claims processing and audit experience Situation Unit Resource and information tracking, posting, publishing – good communications Business Resumption Branch Director Responsible for liaising with business units for incidents requiring activation of business continuity plans Supply Unit Leader Vendor relationships/multi-tasking and order processing skills Accounting Unit Leader Financial Analyst/ A/R and A/P experience Documentation Unit Records management skills; very organized Transportation Unit Ldr Travel and accommodations. Liaison with travel agency
Operations Section Chief Individual with best knowledge of affected operations/area and leadership/management skills Damage Assessment Team/ Facilities Team Activated for incidents affecting the facility On-Scene Commander Incident Management Team Business Resumption Director Responsible for direct liaison with the business units for incidents that require activation of the individual team plans. May be the alternate site Commander, especially if a regional facility is the impacted site. Must have strong leadership skills. INCIDENT COMMANDER Executive or senior manager with broad knowledge of company operations and strong leadership skills COMMAND STAFF PR/Information Officer Investor Relations Legal Advisor Environmental Affairs Safety/Security Advisor Human Resources Advisor IS Liaison Planning Section Chief Logistics Section Chief Solid knowledge of SLAs; management, delegation skills Finance Section Chief Strong project management skills, very detail oriented; able to coordinate well with other functions Responsibility for overall finance and accounting operations Purchasing Unit Leader Procurement & financial background/skills Insurance Unit Leader Insurance/claims processing and audit experience Situation Unit Resource and information tracking, posting, publishing – good communications Business Resumption Branch Director Responsible for liaising with business units for incidents requiring activation of business continuity plans Supply Unit Leader Vendor relationships/multi-tasking and order processing skills Accounting Unit Leader Financial Analyst/ A/R and A/P experience Documentation Unit Records management skills; very organized Transportation Unit Ldr Travel and accommodations. Liaison with travel agency
Operations Section Chief Individual with best knowledge of affected operations/area and leadership/management skills Damage Assessment Team/ Facilities Team Activated for incidents affecting the facility On-Scene Commander Incident Management Team INCIDENT COMMANDER Executive or senior manager with broad knowledge of company operations and strong leadership skills Facilities and Damage Assessment Activate only for incidents that physically affect the facility. Responsible for the damaged site’s restoration. COMMAND STAFF PR/Information Officer Investor Relations Legal Advisor Environmental Affairs Safety/Security Advisor Human Resources Advisor IS Liaison Planning Section Chief Logistics Section Chief Solid knowledge of SLAs; management, delegation skills Finance Section Chief Strong project management skills, very detail oriented; able to coordinate well with other functions Responsibility for overall finance and accounting operations Purchasing Unit Leader Procurement & financial background/skills Insurance Unit Leader Insurance/claims processing and audit experience Situation Unit Resource and information tracking, posting, publishing – good communications Business Resumption Branch Director Responsible for liaising with business units for incidents requiring activation of business continuity plans Supply Unit Leader Vendor relationships/multi-tasking and order processing skills Accounting Unit Leader Financial Analyst/ A/R and A/P experience Documentation Unit Records management skills; very organized Transportation Unit Ldr Travel and accommodations. Liaison with travel agency
A leader takes people where they want to go.
A great leader takes people where they don't necessarily want to go but ought to be!