150 likes | 430 Views
Data Center Services Update Lisa Petoskey Director, Information Technology May 12, 2010. 1. Agenda. Data Center Services (DCS) Overview Budget Issues List. 2.
E N D
Data Center Services UpdateLisa PetoskeyDirector, Information TechnologyMay 12, 2010 1
Agenda • Data Center Services (DCS) Overview • Budget • Issues List 2
“To solve these significant issues, the TWDB needs to be exempt from this mandate, and resources and operational authority returned to TWDB. This will allow TWDB to ensure that vital and critical water-related data essential to the future management of water in Texas is secure, backed-up appropriately, and recoverable. This will also ensure that TNRIS can continue to provide emergency response data services to federal entities during emergency situations.” 2009 TWDB Self Evaluation Report 3
DCS Overview • HB 1516 of the 79(R) Legislative Session • Mandated Interagency Contract with Department of Information Resources (DIR) • IBM Team for Texas (TfT) manages our current Data Center • Servers and Storage • Systems Administration • Disaster Recovery of Agency data • State Auditor’s Office (SAO) Audit • In August 2009, the SAO released an audit report on DIR and the State Data Center Consolidation • Four key points included in the report: • The Department’s oversight of the application remediation process has been insufficient • The Department’s monitoring of IBM’s reported service levels has been insufficient • The Department has not ensured that the procurement of equipment and software for agencies has occurred in a timely manner • In their responses to the SAO’s survey, agencies stated that resources are still being used to support state data center services. 4
DCS Overview • DIR – Independent Data Center Assessment • In September 2009, DIR contracted with Sierra Systems in partnership with EquaTerra to conduct a full review of data center consolidation efforts and challenges • Review was commissioned by DIR to reassess the business model in order to guarantee successful continuation of the project • Report released November 2009 • The report recommends: • Realigning DIR’s existing agreement with IBM and state agencies to reflect the varied business needs and organizational and operational structures of the 27 state agencies that are subject to data consolidation • Developing a means to more quickly review, prioritize and address disputes over billing and data issues • Redesigning the governance structure to give state agencies a greater role in developing and implementing data consolidation • Achieving a revised agreement with IBM by February 2010. The consultants noted that revising the agreement with IBM can be achieved without additional cost to the state. 5
DCS Overview • DIR and IBM - Contract Negotiations • Original completion February, 2010 • Revised completion May, 2010 • TWDB to Sunset Presentation • Presented DCS Issues to Sunset on April 16, 2010 • Indicated desire to be exempt • DIR currently under Sunset review 6
DCS Budget • FY 2010 • $1,563,732 Appropriated • Estimated shortfall $181,865 • FY 2011 • $1,560,390 Appropriated • Estimated shortfall $ 425,625 • FY 2007 to Date - DCS Contract Expenditures • $3,835,234 7
DCS Issues • IBM’s inability to respond during an Emergency Disaster • Lack of technical expertise and experience in Geographic Information System (GIS) • Insufficient number of IBM staff • High cost of storage; could increase further • Aging Hardware • 1/2 of TWDB servers are five years old or older • Of those, 17 are seven years old or older 8
DCS Issues • Unable to upgrade software in a timely fashion • IBM software versioning document (N/N-1) does not include latest versions of various software • “N” represents current software version allowed by Agencies • “N-1” represents current software version allowed minus one version back • Windows Server 2008, MS SQL Server 2008, and Groupwise 8.0 not currently included on “N/N-1” document • Two production server outages during the Fall of 2009 • Restore lasting over a week for each server • SAN Outage – firmware two versions behind 9
DCS Issues High level of agency oversight required to monitor incidents, change requests, backups, invoices and general monitoring Backlog on service desk tickets; 73 Open Tickets as of May 7, 2010 Inconsistency in successful backups 10
DCS Issues • Suspension Release Signature • On October 28, 2008, Governor Perry suspended consolidation of agency equipment and data within the two state date centers due to IBM’s failure to back up data for some state agencies • DIR required IBM to implement a plan and demonstrate that it could comply with the backup requirements in its contract • DIR required each agency to certify that IBM had identified and documented the agency’s critical data and prepared an appropriate schedule for routine backups • Agency certification would occur via signature on a Suspension Release form • TWDB required two full weeks of successful backups prior to signature; the TWDB finally signed the Suspension Release form in October, 2009 • Backups degraded again following signature of the Suspension Release form. 11
DCS Transformation Issues • Transformation • TWDB transformation to data centers not yet started • TNRIS development environment exempted from the contract; this poses new funding and maintenance considerations that have yet to be fully analyzed and resolved prior to transformation • Department of Assistive and Rehabilitation Services (DARS), the Office of the Attorney (OAG) and the Texas Department of Insurance (TDI) have all identified network degradation once server(s) are transformed • Department of State Health Services (DSHS) and TDI conveyed instability issues on VMs (Virtual Machines) once deployed to data center 12
DCS Transformation Issues • Transformation • IBM Scorecard (completed monthly by Agencies) for February – 85% of Agencies rated IBM “2” or below on a scale of 1-5. • New “application approach” vs “server approach” • April 7, 2010 meeting with DIR and IBM • Enterprise application approach vs Agency application approach • Scientific method • Priority approval via governance structure • Results in final completion delay for TWDB 13
DCS Transformation Concerns Serious concerns and fear Negative impact to operations Risk vs Reward – extremely risky TWDB approximately 1.33% of overall contract costs Risk mitigation – exempt from mandate 14
Questions? 15