210 likes | 344 Views
VERMONT STATE COLLEGES WHO WE ARE. VERMONT STATE COLLEGES SHARED SERVICES. From the data center: (located in a state-owned building) Wide-area network & single Internet connection ERP Identity & Access Management Portal IT Help Desk Data warehouse Conference & events application
E N D
VERMONT STATE COLLEGESSHARED SERVICES • From the data center: (located in a state-owned building) • Wide-area network & single Internet connection • ERP • Identity & Access Management • Portal • IT Help Desk • Data warehouse • Conference & events application • Enterprise IT Staff: • 6 programmers; 3 system admins; 1 network; 1 security & data center
VERMONT STATE COLLEGESDISASTER RECOVERY • First plan in 2007 • Approved by Council of Presidents • Reviewed twice a year - updated February, 2011 • Key feature: assumed local and business process planning “to accommodate the absence of critical systems for a period of up to two weeks” • In-scope and out-of-scope clearly articulated
VERMONT STATE COLLEGESBUSINESS CONTINUITY • Plans for admissions, accounts payable, accounts receivable, finance, financial aid, payroll and registrars processes • Reviewed and updated annually • Some business areas could go for several weeks without extraordinary measures • Some could only function for few hours or days
VERMONT STATE COLLEGESWHAT HAPPENED • On Aug 28, 2011, at 8:13 PM, "Pollak, Dianne M" <dianne.pollak@vsc.edu> wrote: • > Just got a text from michael... First floor of the hospital is under water... They are evacuating randall st. We are on generator. • > • > Dianne Pollak • > Director of Administrative Information Systems Vermont State Colleges
VERMONT STATE COLLEGES-WHAT HAPPENED • By 6 am on 8/29 – water had filled basement and up to 5 feet of the first floor • Generator under water = power gone • Entire state office complex flooded • Roads to town closed
VERMONT STATE COLLEGESBUSINESS IMPACT • Two colleges had started classes • Two colleges were about to start classes on Tuesday • One college was still in “marketing mode”
VERMONT STATE COLLEGESSHORT-TERM SOLUTIONS • Monday - Emergency web site up by Monday • Monday - Alternate e-mail addresses collected • Tuesday - Wide-area network and Internet restored • Monday - Colleges using internal e-mail • Monday-Thursday – Some paper paychecks handwritten • Tuesday-Friday – Air-conditioning and server issues • Friday morning – ERP and other critical systems restored
VERMONT STATE COLLEGESEARLY LESSONS • Even if you can’t afford what you really need, have a plan! • Business continuity plans had some issues • Never underestimate the power of denial • Good relationships sustain you; bad ones are killer • Stay on the lookout for good ideas and be flexible • It’s true: never waste a good disaster • Keep track of who’s doing what • Communications are critical & difficult
High Water Mark DATA CENTER GENERATOR...
VERMONT STATE COLLEGESEARLY LESSONS • Take notes everywhere, in every conversation • Have a conference call system set up • Texting is critical (and set up those contacts!) • Have an alternate e-mail address for all critical contacts – IT and priority business areas • Have a plan for alternate web site hosting • Staff check-in: food and sleep • Staff in different places on the resilience scale • As everyone gets tired, mistakes will be made • Danger in false sense of stability/reliability
UP NEXT: SNOWTOBER From: The EDUCAUSE CIO Constituent Group Listserv [mailto:CIO@LISTSERV.EDUCAUSE.EDU] On Behalf Of Theresa RoweSent: Thursday, November 03, 2011 8:44 AMTo: CIO@LISTSERV.EDUCAUSE.EDUSubject: Re: [CIO] Snowtober Thanks so much for taking the time to write out the details of your experiences. We all learn from these posts. Our experiences with electrical power failures and generators have been very difficult to manage. Every situation has been different, so you really need thinking, problem-solving people on hand, rather than a scripted action. You've given us more to think about.Theresa
VERMONT STATE COLLEGES OPPORTUNITIES • Reinforce the importance of both disaster recovery and business continuity planning and underscore the differences • Make the case for disaster recovery resources-less difficult in the reality of business interruption • Change things that were “unchangeable” • Focus on what’s important: people and mission
VERMONT STATE COLLEGESTODAY • Access to building restricted to 2-3 staff • Off generator; data center is only room with power • Legislature to determine future use of the complex • IT Staff distributed across the state • Building two data centers – looking at co-location • Dealing with displaced employees • Re-projecting the annual project list • Getting back to “normal” - IT reporting cycle