80 likes | 295 Views
CITI. Status Update for ECC Reporting Project August 20, 2009. ECC Reporting Relief for CITI. #1: Short Term Goal: FMO for ICG Report for Tony DiSanto. Execute Plan – Modify Reports, Remediate Agents 7/30 through 8/5. Report 8/6. Tweaking Reports - 8/11.
E N D
CITI Status Update for ECC Reporting Project August 20, 2009
ECC Reporting Relief for CITI #1: Short Term Goal: FMO for ICG Report for Tony DiSanto Execute Plan – Modify Reports, Remediate Agents 7/30 through 8/5 Report 8/6 Tweaking Reports - 8/11 #2: Establish a Repeatable Process to maintain LDT current Continue remediation and document process for maintaining LDT current Already in Process Deliver By 8/31 #3: RCA and Recommendations for Long-Term Stability Document Findings and Recommendations Already in Process Deliver By 9/30 Note: “Deliver by” provides a final delivery date; could be delivered earlier.
Short Term Goal: ICG Business Unit FMO • As of August 3rd, 622 ICG Hosts were identified with Agent discovery problems • These hosts were processed using a batch deletion script provided by Engineering (“Master Agent Restart Script”) • Upgrade 5.2 Agents • Upgrade to 6.0 agents was not possible, CITI policy for uninstall/install is the obstacle • Ran remediation against 5.2 agents • Modify FMO Report developed by Jeff Tang’s group • Additional resources provided by Dave Wallace’s group
Short Term Goal: ICG Business Unit FMO, cont. • Results: • 500 (80% ) ICG Hosts now are reporting current discovery times • 122 (20%) ICG Hosts will require individual remediation by CITI SA’s with EMC • This resulted in an increase of 80TB of storage (tot. capacity is 1.1 PB) • EMC was instructed to focus on ICG only and not the complete 1800 hosts • EMC will continue to work through the 1800 hosts over the following 2-3 weeks • Agents that need to be re-installed will take an extended period of time; reinstallation is CITI’s responsibility and EMC will assist as much as possible while onsite.
Establish a Repeatable Process:Maintain LDT current • Now a site-wide remediation project • 2,257 Hosts Identified with Agent Discovery Problems site wide • Of this 2,257: • 1150 Hosts have been processed using the batch deletion script • Results: • 704 (62%) Hosts are reporting current discoveries after action by EMC • 446 (38%) Hosts will require individual remediation by CITI SA’s with EMC • 1107 Have not been attempted, due to CITI’s own restrictions • Hosts contain database agents and hosts that are determined to be critical
Establish a Repeatable Process:Maintain LDT current, cont. • Process Includes: • Using the Master Agent Restart Script provided to remediate the host • Master Agent Restart Script Handles • MO deletion from CC Repository • MO with multiple domains • AAD DB batch load script • Provides the ability to feed the database discovery information from a file • Allows for bulk processing to discovery DB’s • Document the steps for repeating this process periodically by CITI • Purpose is to extend the short term remedial action • EMC will have up to 2 resources at CITI to continue agent remediation through the remainder of the Hosts as directed by CITI • This team will also work as much as possible with the CITI SA’s to get the Agents reinstalled on the host which can not be restarted.
Long-Term Stability: RCA & Recommendations • EMC to Document Findings and Recommendations • This will cover: • Use Case Analysis • Gap Analysis • RCA (Root Cause Analysis) • Findings will explain why CITI is experiencing Problems • This will be based on the Use Case Analysis, Gap Analysis, and first-hand experience while working issues and interviewing the Cit Dev and OPS staff. • Recommendations will cover • Bugs in ControlCenter • Enhancements to ControlCenter • Changes to CITI operations procedures • Changes to CITI Development