110 likes | 250 Views
Case Studies in Business Continuity: What NOT to Do May 26, 2006. Kathleen A. Lucey kalucey@montaguetm.com tel: +1-516-676-9234 cell: 516-384-6437. Business Continuity Teams. Information Technology Recovery Teams. INTERRUPTION MANAGEMENT MODEL.
E N D
Case Studies in Business Continuity: What NOT to DoMay 26, 2006 Kathleen A. Lucey kalucey@montaguetm.com tel: +1-516-676-9234 cell: 516-384-6437
Business Continuity Teams Information Technology Recovery Teams INTERRUPTION MANAGEMENT MODEL Initial Interruption Management Interruption Management Team Executive Oversight Team Employee Support EMT Government Liaison Emergency Funding Media Relations Team Transportation, Communications Emergency Logistics Command Center Support Team Physical Security HAZMAT Admin. Services Damage Assessment Business Continuity Coordination Insurance Liaison Recovery Management Purchasing Business Recovery Coordination IT Recovery Coordination Site Repair or Relocate Site Relocation and Recreation Site Repair and Restoration 2006 Montague Technology Management, Inc. All rights reserved.
Where are MOST of the Continuity Challenges ?? CONTINUITY ISSUES SOLUTIONS Catastrophic Interruptions Continuity/Recovery Minor Interruptions Availability Core Business Value Chain Processes Everyday Blips Reliability Process Dysfunctions Engineering
Case #1 Loss of Street Power and Emergency Power • Project completed 3 months earlier to install diesel generator for supplemental power. • Installation certified by in-house engineering inspector. • Friday afternoon, 3 p.m.: loss of power to data center. • Power does NOT flow from diesel generator.
Case #1 Main Cable To Data Center Diesel Generator Transfer Switch To/From Street building wall
Case #1 Contributing factors: • Cost-cutting policy requires accepting low bid. • Insufficient cable ordered by contractor for job. • Inspection performed by unqualified staff member. • No follow-up maintenance or inspection. • No redundant main cable in design.
CASE #2 Premature Dismissal of Staff • Water pipe in building ruptures at 10 a.m. on Wednesday. • Water begins to flow through multiple floors. • No announcement from Building Management or Company. • Rumors circulate; staff members of financial institution evacuate to street. • Unauthorized executive dismisses staff for the day. • Business continuity plan not activated. • Nearly a full day’s productivity is lost: estimated at >$400,000. • Re-entry to building allowed 2 hours later.
Case #2 Contributing factors: • Business Continuity Plan not activated. • Crisis Management Team not engaged. • Excessive focus on worst-case scenario. • Little awareness of individual default response on evacuation. • Poor communication procedures.
Case #3 Loss of Integrated Management System and Data • Upgrade of system software fails at 11 p.m.; system unavailable. • Backup not performed just prior to upgrade attempt. • Yesterday’s backup is empty, as are all others looked at. • One system backup found from 6 months prior; system restored to this level. • Manual entry of missing data necessary for prior 6 months. • Construction project disruption and legal liability.
Case #3 Contributing factors: • Outsourced IT staff cannot be evaluated within firm. • Management insistence on minimizing costs for IT. • Dangerous system upgrade procedure or lack thereof. • No monitoring of backup completion status: backup was not performed unless system was quiesced with no users. • Luck saved the day: 6 months ago the system was down when the backup was run.
Questions ?? Kathleen Lucey Montague Technology Management Inc. kalucey@montaguetm.com +1-516-676-9234