490 likes | 1.62k Views
IT Major Incident Management Response and Communication. A Collaborative Approach to IT Major Incident Management. Nancy Proctor Chris Wright K evin Chenoweth October 2012. Collaborative Approach to IT Major Incident Management. Objective: Expeditious Return to Normal Operations Managed
E N D
IT Major Incident Management Response and Communication A Collaborative Approach to IT Major Incident Management Nancy Proctor Chris Wright Kevin Chenoweth October 2012
Collaborative Approach to IT Major Incident Management • Objective: Expeditious Return to Normal Operations • Managed • Measured • Deliberate • Safe • Collaborative effort: IT; Hospital Administration; VUMC Communications; (Inpatient) Systems Support Services; (Outpatient) Operations Systems Engineering, Operational Units; Others as needed
IT Major Incident Definition* • High impact, or potentially high operational impact • Requires a response that is above and beyond that given to normal incidents. Typically, requires: • Cross departmental coordination • Management escalation • Mobilization of additional resources • Increased communications *ITIL Incident Management
IT Major Incident Phased Response and Escalation Points • Phase 1 From IT Incident Awareness to activation of IT Technical Conference Call Bridge • Phase 2 From Activation of IT Conference Call Bridge to Activation of Administrative Conference Call Bridge • Phase 3 From Activation of Administrative Conference Call Bridge to Return to Normal Operations
Communication Protocols • Early “Heads Up” Advisory to Hospital Administrative Coordinators (via text page) • Technical IT Conference Call Bridge (CCB#1) • Reserved for IT troubleshooting and internal IT communications only • Administrative Conference Call Bridge (CCB#2) • Reserved for IT and Hospital Administration communication, and coordination of Operational response • Activated by IT on request from Hospital Administration • Medical Center Alerts and Communications • IT and Operations jointly approve content of alerts and medical center communications
Roles and ResponsibilitiesIncident Manager • Helpdesk Manager On Call assumes the role of Incident Manager* • Coordinates initial incident response and IT impact assessment • Liaise with Helpdesk, workgroups, and Informatics Admin On Call • Communicates IT impact assessment and status updates to Informatics Admin On Call, (via DR Admin on Administrative Conference Call Bridge if activated) • Maintains list of impacted systems and status Currently Helpdesk Manager but this may change
Roles and ResponsibilitiesInformatics Administrator On Call (IC AOC) • Provides IT leadership and direction • Communicates impact assessment and status updates to Hospital Administration (ACs and AOC, depending on “phase”) • Presence on either Conference Call Bridge dependent on Incident Management Phase • Provides input to, and approves content of, enterprise communications • IT Counterpart to Hospital Administrator On Call
Roles and ResponsibilitiesSystems Support Services/Operations Systems Engineering • Receive initial notification from Helpdesk • Receive notification when Technical CCB is opened; SSS Primary On-Call will join the bridge to receive status briefing • Escalate to Systems Support Management as necessary • Assist with impact assessment(s), workarounds, end user communications, resource requirements, issue verification, and recovery verification • Provide advice re need for House-wide downtime • Assess need for StarPanel banner message; provide input to IT AOC and Hospital ACs on content • Receive notification when Administrative CCB is activated • Interact with Informatics Center Admin On Call, Hospital ICs, IT Leadership Liaison, Hospital AOCs on CCB #2 (Admin Bridge Line)
Roles and ResponsibilitiesHospital AC (Administrative Coordinator) • Receives initial “heads up” text alert, notification of IT Conference Call Bridge activation (but does not join call), and IT impact assessment from IC AOC • Conducts operational impact assessment • Determines need to activate Administrative Conference Call Bridge (CCB#2) • Determines need for overhead announcements, alerts and enterprise communications • Collaborates with IC AOC on content of announcements, alerts, and enterprise communications • Engages other Operational (non-IT) resources as necessary • Determines need to escalate within Hospital Administration hierarchy
Roles and ResponsibilitiesHospital AOC (Administrator On Call) • Receives initial notification and/or updates from Hospital Administrative Coordinator • Joins Administrative Conference Call Bridge • Leads Operational Response Activities • Establishes communication with IC AOC via Administrative Conference Call Bridge • Determines need to activate Emergency Operations Center (EOC) • Determines need for overhead announcements, alerts and enterprise communications • Collaborates with IC AOC on content of announcements, alerts and enterprise communications • Engages other Operational (non-IT) resources as necessary