310 likes | 465 Views
WORLD TRADE CENTER And Related Recoveries Len Guddemi Vice President Northeast Operations. Comdisco Confidential. SunGard Worldwide Disaster Support Experience. Recoveries per Year. Environmental 8/2%. Software 10/2%. Telecomm 14/3%. Terrorist Attack 100/19%. Bomb 16/3%.
E N D
WORLD TRADE CENTER And Related Recoveries Len Guddemi Vice President Northeast Operations Comdisco Confidential
SunGard Worldwide Disaster Support Experience Recoveries per Year
Environmental 8/2% Software 10/2% Telecomm 14/3% Terrorist Attack 100/19% Bomb 16/3% Civil Unrest 2/0% Earthquake 19/4% Hardware 97/18% Miscellaneous 20/4% Fire/Explosion 27/5% Data Center 7/1% Flood 57/11% Hurricane 84/16% Power Outage 76/14% SunGard Worldwide Disaster Support Experience Disruption Cause:Number/Percentage
DEC 43/8% HP 22/4% Miscellaneous 26/5% Workarea Recovery 130/24% AS/400 56/10% Advanced Recovery 13/2% RS/6000 10/2% Unisys 2/0% Shell 7/1% Tandem 3/1% Systems On-site Mobile 39/7% Sun 16/3% Mainframe 138/26% Item Processing 5/1% Trading Floor 19/4% Stratus 1/0% Mobile Cluster 3/1% NT Server 4/1% SunGard Worldwide Disaster Support Experience Recovery Platforms Used: Quantity/Percentage
Civil Unrest 3/0% Telecomm 30/1% Terrorist Attack 1154/20% Earthquake 136/2% Data Center Move 269/5% Miscellaneous 858/15% Hurricane 318/6% Fire/Explosion 359/6% Power Outage 370/7% Flood 691/12% Bomb 146/3% Hardware Problem 666/12% Environmental 573/10% SunGard Worldwide Disaster Support Experience Days of Occupancy: Days/Percentage
Declaration Information • Number of Events • Overall - 93 Schedules, 47 Customers • Simultaneous High Point - 76 Declarations, 16 Facilities, 14 Platforms • Reason • Facilities Unavailable • Loss of Power, telecommunications, denied access to buildings, concerns over building structural safety Comdisco Confidential
SunGard Sites & Platforms • North Bergen: Voice/Work Area, Main-Frame, ARS & Networking • Phoenix: Work Area • Queens: Work Area, Trading Floor, Dedicated, Trading PIT • Rosemont: Work Area • San Ramon: Work Area, Trading Floor • Wood Dale: DEC/VAX, DEC/Alpha, VAX3100, VAX7620, Filenet, AS400, Tandem • Wood Cliff Lake: Intel Servers, Work Area • Jersey City, workgroup, AS/400 • Philadelphia: Work area, RS6000, AS/400,Mainframe, Voice • Herndon: Work Area • Warminster: Work Area,Unisys • Atlanta: Work Area Seats, Remote Sun, AS400, RS6000 • Carlstadt: Voice/Work Area, Business Recovery/Quick Ship, Trading Floor, Dedicated Seats, SUN, AS400, DEC/VAX, RS6000, HP9000, Intel Based Servers; Dedicated HP9000, RS6000, NT Servers • Boston: SUN, Work Area • Cypress: Work Area, Remote Tandem • Mobile Recovery: 13 Units/5,000sq.ft., 10 Units/3,000sq.ft. for Work Area • Minnetonka: Work Area Comdisco Confidential
Declaration Information - Continued Items of Note • Implement Your Plan & Associated Vendor Services with Urgency • 14 Declarations in 30 minutes; 24 within 1 hour; 19 in 2nd hour • You will be in competition with other business for services & manufacturers • Quickly establish damage assessment & need for services, and implement • Under-subscription or additional needs assessment remains an issue • Many businesses requested additional needs ATOD • Responded to over 500 special requests for service. • Built out with best efforts • This puts recovery & business continuity at risk due to extended recovery timeline. • Personnel & Intellectual Property Keys to Recovery • Methodologies, ideas and documented plans utilized. • Assigned backup roles and succession planning implemented. • Intellectual capital and personnel in alternate locations. • Use of professional organizations to augment staff. Comdisco Confidential
Declaration Information - Continued Items of Note - Continued • Reluctance to Travel • End Users generally won’t travel far from home. • Ensure adequate end user facilities available nearby (about 50 miles) • Air Travel Unavailable • Delayed delivery of offsite stored media to recovery location • Identify alternate means of transportation • Assess need for electronic journaling of data to alternate location • Expert staff unable to travel • Identify options for local access by staff to remote alternate systems (and possibly home systems) • Assess value of ‘turnkey’ type services with vendors • Offsite Storage Critical • Evaluate for frequency, completeness • Paper records and transactions may be totally lost • Evaluate if your offsite storage is far enough away • Evaluate need/value of electronic journaling Comdisco Confidential
Declaration Information - Continued Items of Note - Continued • High Availability Solutions Extremely Successful • Electronic Journaling, Data Base Mirroring, SRDF, etc. • 45 Terabytes ready very quickly • Data requiring physical transport delayed • Active Testers • More successful at meeting RTO/RPO • More realistic understanding of application performance • Fewer problems • Very High Demand for Voice Recovery • Realistically evaluate needs: line trunking, extensions, call recording, voice mail, call forwarding • Avoid Fast Busy Comdisco Confidential
SunGard Challenges & Tactics • Disaster Hotline & Notification • Increased Staff handling incoming Disaster calls • Security • Immediate increase in security staff on all shifts. • Contacted local authorities to assist in access control • Additional Staff • Request for additional support staff • Mobil Recovery Team • Additional Resource • Contacted third party vendors • Arranged additional parking with neighbors
SunGard Challenges & Tactics • SunGard Facilities and Staff not directly affected • Implemented Swat Support Teams • Established 24/7 Command Center at Corporate HQ with Open Bridge • SunGard Canadian employees, Illinois, Minneapolis, and Texas employees drove to Carlstadt & North Bergen to help • SunGard’s Professional Services Consultants, Sales, Sales Support, Administration and SunGard’s Operations staff worked side by side. • Dispatched relief support from other SunGard facilities once air travel restored • Activated facilities, equipment procurement, shipping and solutions teams to respond to special requests • Received offers of help from customers • Gratefully declined • Network Alternate Routing Critical • Significant damage due to the event • Fiber cut after event by construction/demolition crews • SunGard alternate routings worked without issue • Vendors terrific with T1s, DS3 installations in days Comdisco Confidential
Customer Challenges & Tactics • Inaccurate Needs Assessment • Special requests for PCs, & Servers • IP addressing • Storage • Tape Drives • Electricity • Processing power • LAN bridging • Televisions & Printers • Network capability, Voice services & Routers • Faxes • Software and more Comdisco Confidential
Customer Challenges & Tactics • Logistics • Travel Restriction an Impact - used ground travel for many shipments • Shipped over 4,200 PCs along with much other equipment • Customer parking, bussing, and catering • Getting Restoration Media to Alternate Site • Transportation challenge • Complete set available and labeled • Missing or unavailable documentation • Start-up scripts • Run instructions Comdisco Confidential
Learned / Re-Affirmed • Value in Pre-Planning Staff Placement within Rooms • End User priorities & usage • End User equipment requirements – avoids setting up more than once • Important to Have a Team Leader for Each Shift • Integration of recovery requirements into change control • Number of PCs, Positions • Network Capabilities • Processors, disk, tape capabilities • Printers, fax machines • PBX features & functionality • Platform Technology upgrades, changes, maintenance • Importance of Quick Assessment & Implementation of Plan when appropriate Comdisco Confidential
Learned / Re-Affirmed - Continued • Value of Solid Backup Records Program • Tape, electronic, & paper records foundation of recovery • Evaluate right balance for your company • Documented & labeled so vendor(s) can assist • Have scripts shipped with restore media or stored at alternate location • Frequency of backups • Offsite location & transportation dependencies • Examine benefit of electronic transmission of offsite storage data • Can significantly improve recovery time and reduce data loss • Evaluate Strategy of Shared versus Dedicated Resources • Guarantees resource access for some number of most critical requirements • Complement with shared resources • PC Image Restoration Tool Effective • Ghost, or similar Comdisco Confidential
Learned / Re-Affirmed - Continued • Basics are Fundamental • Establish personnel roles & assign backup • Document methodologies, ideas, business plans & recovery plans • Review, update & follow established procedures • Test the way you would recover, recover the way you test • Worthwhile to review how home procedures apply in alternate site production mode • Offsite Storage, Disk mirroring, Miscellaneous other • Revisit your Plan & Recovery Requirements Often – Technology & Business • In Times of Need, Help Comes From Many Sources Comdisco Confidential
Customer Comments “Although operating in our primary location is preferred, the equipment, professional and courteous staff, and outstanding overall accommodations provided for an effective and successful business recovery. I am glad that we have contracted with your organization!” “I have in the past worked with a number of outstanding individuals, but never before have I worked with such a highly functional, customer service oriented team. Individually each team member I have had the pleasure to deal with was exceptional; as a team there is none better.” Comdisco Confidential
Customer Comments “I would like you to know of the "more than wonderful" service we are receiving at the SunGard site. ... have gone out of their way to assist us with some special needs. ... have answered numerous questions and assisted --she is great. ... worked with ENS last night, assisted me with set up so I can stay on line (due to the situation in New York and the weather closing down our site) and has been a great help to the IT staff.” “Thanks for all the support and assistance. We’re glad you are here.” “I would like you to know it is people like this that make SunGard a vendor we choose to depend on in the event of an emergency. Please pass this message along to their managers, it would be greatly appreciated.” Comdisco Confidential
Questions and Answer ??? Comdisco Confidential