80 likes | 444 Views
DTCC’s Response to Super Storm Sandy Cecilia Humphrey Americas ’ Representative, DTCC ACSDA General Assembly, April 2013 Antigua, Guatemala. Special thanks to: David. LaFalce, Business Continuity Management; and
E N D
DTCC’s Response to Super Storm SandyCecilia Humphrey Americas’ Representative, DTCCACSDA General Assembly, April 2013Antigua, Guatemala Special thanks to: David. LaFalce, Business Continuity Management; and Michael Obiedzinkski, Enterprise Communications DTCC Green
DTCC’s Sandy Recovery: ‘Why It Worked’ DTCC Green
Business Continuity Management (BCM) Preparations – Pre-Sandy Business Continuity Management preparation includes: • Consistent execution of BCM tests --announced and unannounced • Cross-training and transfer of work functions (e.g. stand-down sustainability exercises, remote access) across sites • Well-documented procedures and job aides for critical products/processes • Rotation of operational responsibility for product and technology operations • Activation of emergency notification systems to ensure information sharing Prior to Sandy, DTCC’s facilities were operating as follows: • Staff located in DTCC offices in New York, Tampa, Dallas, London, Shanghai, Manila, Chennai, and Pune • DTCC Dallas operating as backup for technology • DTCC Tampa backing up business operations • DTCC Brooklyn operating as primary data center, with synchronous replication to DTCC Manhattan and asynchronous replication to DTCC Dallas • Only production data was being replicated between sites • Initial Work-from-Anywhere (WFA) capabilities enabled through Virtual Desktop Infrastructure (VDI) established in preparation for move to DTCC Jersey City DTCC Green
BCM Response to Sandy Friday- October 26:Corporate Command Team activated pre-Sandy. Saturday- October 27: State of Emergency declared in New York. Sunday- October 28: Production operations transferred to Tampa. Technology operations transferred to Dallas. Monday- October 29: • DTCC processing fully operational. • Disaster declared at 7:45pm by NYS. All core production systems recovered within 2 hours and transferred to DTCC Brooklyn facility. Tuesday- October 30:Markets closed. DTCC processing fully operational. Wednesday- October 31:Markets re-open. 300 DTCC employees redeployed to Brooklyn. Thursday- November 1: Securities processing for physical securities established (Brooklyn). Over 400 employees now operating out of DTCC Brooklyn. Friday- November 2:Physical document recovery plan/relocation plan takes shape. November 5-9: 650 employees in DTCC Brooklyn. Volumes return to normal levels. November 12-16:Decision to accelerate move to DTCC Jersey City. Ongoing recovery planning for DTCC Manhattan. November 19-23:Vault recovery begins. Certificates begin to be packed and prepared for restoration. November 26 to Present:Vault recovery completed. 1,500+ employees relocated to DTCC Jersey City. DTCC Manhattan normalized. DTCC Green
Post-Sandy Recap-- Areas of Success Successful execution of business continuity and disaster recovery plans • Transfer and dispersal of core services Global support from teams in Tampa, Dallas, London , Shanghai and Chennai Restoration of employee connectivity and functionality • Work-from-Anywhere program, and accelerated DTCC’s Jersey City move. Re-creation of compromised capabilities, e.g., IT developers’ desktop tools and test environment for GTR Participant Testing Leveraging vendor relationships minimized costs and loss of productivity • Temporary rental of equipment kept operations running, backed up • Creation of an innovative service-credit arrangement, i.e., switched some processing to optimize equipment efficiency Continuous processing of transactions Proactive and frequent communications with regulators, clients and employees After years of preparation, well practiced plans with defined roles and responsibilities were activated, with little to no need for improvisation DTCC Green
Next Steps- Governance • Address concentration risk and the potential problems that a similar or larger storm might cause: • Regional exposure in the Northeast due to having 3 sites in the region • Formally measure and address location risk and all viable options • Reassess footprint drivers; regulatory, business, technology, financial, and operational • Evaluation of business needs to ensure recovery prioritization is correct • Expand business continuity to include plans to remotely maintain all processing and operations for an extended period of time • Achieve dematerialization for physical securities – this will reduce risk for all parties involved 0 hours 4 hours Next day > 5 days 2 hours 3 days DTCC Green Keep in mind that different products may have different timing for risk tolerances
Next Steps- Planning • Develop a more comprehensive inventory of all assets across people, places, and things • Continue to ensure we have all the required skills, resources and capabilities to seamlessly handle operations and functions across all sites • Enhance general logistics and procedures; including up-to-date mailing and phone number lists and ensuring employees have all equipment necessary for remote access as well as have all sign-in information available • Integrate plans to include recovery of people, places and things DTCC Green
Next Steps- More Testing • Create the ability to test end-to- end • Enhance crisis management and testing • Leverage the emergency notification tool for employee status checks during tests • Transition from planning and testing for a “single type of impact” catastrophe to a more variable set of impacts DTCC Green