1 / 11

CITI

CITI. Status Update for ECC Reporting Project August 21, 2009. EMC Commitments to Citi . June 4 th Meeting with Howard Elias, Tony DiSanto, Paul Stemmler and Gennevieve Schimpfle ECC Use Cases: scheduled Resource to Assist Reconciling Reports: in process

jett
Download Presentation

CITI

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CITI Status Update for ECC Reporting Project August 21, 2009

  2. EMC Commitments to Citi • June 4th Meeting with Howard Elias, Tony DiSanto, Paul Stemmler and Gennevieve Schimpfle • ECC Use Cases: scheduled • Resource to Assist Reconciling Reports: in process • Agentless Technology Review: reviewed in context of SRM7 • On Site Senior Technical Resource: on site • Future-Proofing: underway for 6.1 planning • ER Tracking: underway for quarterly review • Other Commitments • Customer Service Improvements: PREM commitment solidified • CLARiiON RAID6 & CLARiiON Meta Lun Data Inaccuracies: HF provided July 30 • Host Agent Discovery Issues: under review for Root Cause and permanent fix

  3. Baseline & Current Reporting Environment • Baseline Status as of July 31, 2009 • 7007 Hosts that should report on storage utilization • 4750 Hosts now reporting • 2257 Hosts that are not reporting • Breakdown of Issues identified • Crosses RAID, Tier or array/internal boundaries; FS or DB on unknown storage • Last Discovery > 8 days ("LDT_NC" = Last Discovery Time / Not Current) • No Host Agent (or No ESX Agent or No VMware Agent) • Inactive Agent - Master/Host Agents that are down. • HDS/SP4 or Solaris/SP4 - Host Agents need to be upgraded. • Unsupported file-system type • Current Status as of August 21, 2009 • 7007 Hosts that should report on storage utilization • 5454 Hosts now reporting • 1553 Hosts that are not reporting • 704 Hosts that have been remediated

  4. ECC Reporting Approach #1: Short Term Goal: FMO for ICG Report for Tony DiSanto Execute Plan – Modify Reports, Remediate Agents 7/30 through 8/5 Report 8/6 Tweaking Reports - 8/11 #2: Establish a Repeatable Process to maintain LDT current Continue remediation Document process for maintaining LDT current Already in Process Deliver By 8/31 #3: RCA and Recommendations for Long-Term Stability Document Findings and Recommendations Already in Process Deliver By 9/30 Note: “Deliver by” provides a final delivery date; could be delivered earlier.

  5. #1 Short Term Goal: ICG Business Unit FMO Baseline as of July 31, 2009 2433 Hosts that should report on storage utilization 1715 Hosts now reporting 718 Hosts that are not reporting Current Status as of August 21, 2009 2433 Hosts that should report on storage utilization 2283 Hosts now reporting 150 Hosts that are not reporting Additional Information: 150 ICG Hosts could not be corrected without direct access to the physical hosts. This would require engagement of Citi SA’s. 73% of ICG Allocated Storage is now being report correctly for Storage Utilization Total capacity being reported on is 1.1 PB. This an increase of 80TB of storage on 8/6/2009

  6. #2 Establish a Repeatable Process:Maintain Last Discovery Time (LDT) current • Repeatable Process • A How-to document was written to provide step-by-step instructions on how to Remediate Hosts to achieve host storage utilization reporting • This process include these utilities written by EMC for Citi: • Master Agent Restart Script Handles the restart and rediscovery hosts • Databases on remediated hosts need to be rediscovered • EMC Engineering provided DB batch load utility for DB Rediscovery • Provides the ability to feed the database discovery information from a file • Identification of physical servers which need Citi SA intervention • Citi-wide Remediation Project for Hosts • 2,257 Hosts Identified with Agent Discovery Problems Citi-wide • 1150 Hosts have been processed using the Master Agent Restart script • Results: • 704 (62%) Hosts are reporting current discoveries after action by EMC • 446 (38%) Hosts will require individual remediation by CITI SA’s with EMC • 1107 Have not been attempted, due to Citi’s own restrictions • Hosts that contain database agents and hosts that are determined to be critical

  7. Long-Term Stability: RCA & Recommendations • EMC to Document Findings and Recommendations • This will cover: • Use Case Analysis • Gap Analysis • RCA (Root Cause Analysis) • Findings will explain why CITI is experiencing problems • This will be based on the Use Case Analysis, Gap Analysis, and first-hand experience while working issues and interviewing the Cit Dev and OPS staff. • Recommendations will cover • Bugs in ControlCenter • Enhancements to ControlCenter • Changes to CITI operations procedures • Changes to CITI Development

  8. Moving Forward • Continue to work the Agent Remediation process. • As this is not solution to the cause Citi will continue to experience Agent fall-out to the permanent solutions have been implemented. • Continue to pursue Agent Push • Addressed the Host manifest request by Citi SA group • Citi is currently conducting a pilot to further evaluate the Pushing Agents patches and upgrades from the ControlCenter Console • Benefit to Citi - Will stream line the agent management process • Finalize the Use Case and Gap Analysis • Meeting planned for week of Sept 7 • Document Findings and Recommendations • To be completed and delivered to Citi by Sep 30 • Will include identification of EMC Actions to be taken as well addressing the Citi Enhancement Requests • Will include specific recommendation on how to use ControlCenter more efficiently taking into consideration the operational guidelines of Citi

  9. Some examples… • Windows Host Agents • Most of the installs of ControlCenter Agents on Windows system are corrupt • Cause has been identified as a Citi process and incorrect usage of CC 6.0 Native Install packages • EMC has addressed the Native Install packages to meet Citi requirements with the latest ControlCenter 6.1 UB5 Native Install Packages • Agents become identified with different Server FQDNs (Split-domain issue) • Assumption is – different agents are finding different FQDNs for the host it is installed on because Citi has three different sources for resolving Host FQDN. • These different sources are not all in sync to resolve with the same Host FQDN

  10. Getting to ControlCenter 6.1 • Two-day planning meeting was held August 19 & 20 which addressed: • In-Depth planning on Citi Agent Packaging • Discussion and requirements for Solutions Enabler • Citi Certification Process • Readiness requirements for First ControlCenter 6.1 Upgrade • Citi Agent Wrapper, how and what is needed • Citi requirement to get all pre ControlCenter 6.0 agents upgraded to 6.0 agents • Meeting planned for August 27 for John Maloney with EMC Engineering • Ability to meet the Vmax deployment into Citi • Discussed need for ControlCenter during Citi development period • Establish first ControlCenter 6.1 deployment date (10/30) which was inline with the Vmax schedule into Citi production • Advantages of load balancing and redundancy of Agents • ControlCenter High Availability features and options • Customer Reporting requirements

More Related