430 likes | 673 Views
EMC ControlCenter: Planning for a Successful ControlCenter 6.1 Deployment. Jim Stringer, EMC Principal Corporate Systems Engineer Allen Ward, Senior Storage Technical Specialist. Agenda. A Customer’s Upgrade Experience Summary of the Upgrade Key to a Successful Deployment Bonus Material.
E N D
EMC ControlCenter: Planning for a Successful ControlCenter 6.1 Deployment Jim Stringer, EMC Principal Corporate Systems Engineer Allen Ward, Senior Storage Technical Specialist
Agenda • A Customer’s Upgrade Experience • Summary of the Upgrade • Key to a Successful Deployment • Bonus Material
A Customer’s Upgrade Experience • Allen Ward’s experience with • Upgrade from ControlCenter 5.2
Data Center Environment EMC ControlCenter Managed Datacenter EMC ControlCenter Managed Datacenter EMC ControlCenter Managed Datacenter FC Fabrics : 14 (Ports: ~1500) FICON Fabrics: 2 (Ports: 96) CLARiiON: 9 (~300 TB) Symmetrix: 2 (~250 TB) Centera: 1 (~20 TB) SONET between Datacenter MirrowView/A, SRDF/A EMC ControlCenter Managed DR Datacenter FC Fabrics : 3 (Ports: 240) FICON Fabrics: 2 (Ports: 96) CLARiiON: 1 (30 TB) Symmetrix: 1 (153 TB) Centera: 1 (~20 TB) SONET between Datacenter MirrowView/A, SRDF/A EMC ControlCenter Unmanaged Datacenter FC Fabrics : 3 (Ports: 56) CLARiiON: 2 (10 TB) EMC ControlCenter Unmanaged Datacenter FC Fabrics : 3 (Ports: 56) CLARiiON: 2 (10 TB) EMC ControlCenter Unmanaged Datacenter FC Fabrics : 3 (Ports: 56) CLARiiON: 2 (10 TB)
How hard could it be… The Starting Point • Q4 2007 • Running ControlCenter 5.2sp5 • Started looking at CC 6.0 but there was no immediate urgency • Planned to upgrade to 6.0 by end of 2008 (before EOSL on 5.2) • Drivers for this upgrade included: • Support for Enginuity 5772/5773 • Significant updates to StorageScope • Remain on a supported version Datacenter #1 Datacenter #2 ControlCenter 5.2 Infrastructure ControlCenter 5.2 Agent Server DS-M Connect FibreZone Bridge DS-M Connect FibreZone Bridge
ControlCenter 5.2 Environment Datacenter #1 Datacenter #2 CC_Server_01 Server/Repository StorageScope SAN Manager Integration gateway Centera Agent CC_Server_05 FCC Agent Symmetrix & SDM Agent CLARiiON Agent DS-M Connect FibreZone Bridge CC_Server_02 Store PM WLA Archiver DS-M Connect FibreZone Bridge CC_Server_03 Performance Manager View FCC Agent Symmetrix & SDM Agent CLARiiON Agent IBM ESS Agent CC_Server_04 JAVA Console FCC Agent CLARiiON Agent
Planning Starts… Now what are my options? • Q1 2008 • Began serious work on planning migration to CC 6.0 • Balanced options of upgrade and replacement • Leaning toward replacement to simplify re-architecting the infrastructure • Seriously considered virtualizing the entire environment on VMware ESX • Planned on engaging EMC Professional Services for architecture and design • Implementation would be performed by company resources
Rethinking… The “Crunch” • End of Q1 2008 • There was an opportunity to rebuild the environment from scratch • At this point it made more sense to install CC 6.0 fresh than worry about upgrading from CC 5.2. • Planning proceeded without engaging EMC PS
Let’s get started… Ready, Set, Go • Early Q2 2008 • CC 6.0 planning completed and reviewed by our TC • Installation begins • Ran into a few hitches in the process • Repository and STS Repository Oracle instances conflicted (resolved quickly through an SR) • Issues with DS-M Connect and FibreZone Bridge running together with CC Master Agents on VMs • Issues getting SMC running with our Symm Agent hosts (workaround)
ControlCenter 6.0 Environment Datacenter #1 Datacenter #2 CC_Server_01 Server/Repository Store StorageScope SAN Manager Integration gateway CC_Support_01 DS-M Connect FibreZone Bridge CC_Server_05 FCC Agent Symmetrix & SDM Agent CLARiiON Agent SMC CC_Support_04 DS-M Connect FibreZone Bridge CC_Support_02 DS-M Connect FibreZone Bridge Performance Mgr View CC_Support_05 DS-M Connect FibreZone Bridge VMware Agent CC_Server_02 Store PM WLA Archiver Centera Agent CC_Support_03 DS-M Connect FibreZone Bridge CC_Server_03 FCC Agent Symmetrix & SDM Agent CLARiiON Agent SMC JAVA Console CC_Server_04 JAVA Console FCC Agent CLARiiON Agent VMware Agent PM WLA Archiver
ControlCenter 6.0 is Installed… Hitting Our Stride • Mid Q2 2008 • We have a stable CC 6.0 environment up and running and are working on getting all the host agents re-registered with the infrastructure. Things are looking good! • Late Q2 2008 • We receive our first ever “B-series” Connectrix switches • These integrate beautifully with our McData Native fabrics in Interop Mode 2 • Interop Mode 2 requires FOS 6.0.0c • FOS 6.0.0c requires CC 6.1 (still in development) • Early Q3 2008 • Started hounding every contact I have at EMC for every ounce of information on CC 6.1 available (and some that wasn’t supposed to be) • Planning begins in preparation for an upgrade as soon as CC 6.1 is released • Continue to hound my EMC contacts for release date confirmation and additional details
Deva Vu… “Deva Vu…?” • Mid Q3 2008 • Begin upgrading from CC 6.0 to CC 6.1 • Immediately learned the importance of preparing to roll back before upgrading • Spent the first day (and most of the first night) beginning the upgrade then rolling back to CC 6.0 • A problem in the upgrade process was failing the upgrade of the STS repository, but when it backed out the STS repository upgrade it took out the CC repository with it • Eventually we manually uninstalled STS while rolled back to CC 6.0 and then upgraded • STS was installed fresh on a different host after the rest of the upgrade completed • We ran into a few ongoing issues where host agents were losing connectivity to the infrastructure and consoles would be unable to connect to the server • Eventually resolved with Patch Update Bundle 4414 Recovery Plan
Got this one covered… The Easy Part • Q4 2008 • There was a separate instance still running CC5.2 sp4 within the environment. • Decision was made to integrate this environment into our Enterprise implementation instead of upgrading it separately • Fairly simple and painless process: • Shut down old CC 5.2 infrastructure • Deployed new agents (where required) to monitor infrastructure at this site • Reinstalled agents on hosts to point to central CC 6.1 implementation • Lost historic information for this site, but the value gained in integrating implementations far outweighed the loss
ControlCenter 6.1 – Current Environment Datacenter #1 Datacenter #2 CC_Server_01 Server/Repository Store Key Management SAN Manager Integration Gateway CC_Support_01 McData EOS Provider CC_Server_05 FCC Agent Symmetrix & SDM Agent CLARiiON Agent SMC CC_Support_04 McData EOS Provider CC_Support_02 Brocade FOS Provider CC_Support_05 Brocade FOS Provider VMware Agent CC_Server_02 Store StorageScope Server/Rep PM WLA Archiver Centera Agent CC_Support_03 Performance Mgr View Datacenter #3 CC_Server_03 FCC Agent Symmetrix & SDM Agent CLARiiON Agent SMC JAVA Console CC_Server_06 Symmetrix & SDM Agent CLARiiON Agent SMC CC_Support_06 McData EOS Provider CC_Server_04 JAVA Console FCC Agent CLARiiON Agent VMware Agent PM WLA Archiver
Finally! Where we are today • EMC ControlCenter 6.1 is currently monitoring: • Multiple Data Centers and the DR recovery center. • ~750TB of Online Disk (Symmetrix & CLARiiON) • 17 Fibre Channel Fabrics with ~1700 ports • ~500 physical hosts including • Multiple Z/OS Mainframe LPARs • AS/400 hosts • AIX (& VIO), Solaris, HP-UX, LINUX • Windows Hosts • VMWare ESX with hundreds of Guest VMs
Are we done yet! Current Challenges • Full support of mixed FC fabrics with M-series and B-series switches in a McData Native Fabric mode • Scheduling of custom queries in StorageScope • From a customer perspective the process for getting FC switch fabrics discovered and reporting properly is even more “convoluted” now than it was under CC 5.2 and CC 6.0 • Understanding that this is a multi-vendor issue doesn’t make it easier for the customer
What went well • Lined up EMC people to assist • Knew what ControlCenter documents to use • Created an implementation plan • Reviewed all possible deployment options • Planned to engage EMC for a design review
What were the Challenges • Working through a deployment over time plan • Changes in the datacenter to newer Hardware and Software • Caused designing on the fly • Installing and configuring the enabling software • DS-M Connect and FibreZone Bridge • SMC • Brocade and McData SMI Provider Agents • StorageScope 6.x • Discovering the Managed Objects • Recovering from failed upgrade attempts
For a Successful Deployment Plan, Plan, Plan • Plan for the next 24 months • Hardware/Software changes in the datacenter • Number of Managed Objects (MO) • Physical Server vs. Virtual Servers • Business Continuity (HA, DR, HA/DR) • Have a roll back plan • Engage EMC Global Services • Consulting during the design phase • Implementation Services • Post Implementation HealthChecks
Virtualizing ControlCenter • Installing ControlCenter on VMware • P2V (Convertor) ControlCenter to VMware • Engage your VMware Administrator during the planning phase • Review EMC Technical Note for ControlCenter on VMware • Tip for handling VM Snaphots • Shutdown the host before taking a Snapshot in Virtual Center
Before Spinning CDs Make sure you are prepared • Resolve all ControlCenter issues • Clean up alerts • Review Primus for Upgrade or Install issues • Download all required patches from Powerlink • Install the Document library from CD#4 • Download updated library from Powerlink • Document your step-by-step implementation plan • Copy the ControlCenter CD’s to a Network Share
Check and Check again The Starting Point • Is the environment ready? • DNS forward and reverse resolution correct • Servers are full domain members • Proper storage requirements meet • Correct Server hardware and OS • Windows 2003 Enterprise Server ( 32-bit or 64-bit) • Windows 2008 (will be supported 2H09) • Multi-Processor hardware • Fast Front Side Bus
Check Again Are the servers ready? Server Network Interfaces • No NIC Teaming • load balancing methods are not supported • Set to Full Duplex • No DHCP • Register Only the Primary NIC with DNS • Primary NIC is the 1st in the NIC Binding Order Supported Servers • CINT2006rate SPEC – 60.5 or higher • http://spec.org OS configuration settings • Pagefile – 1.5x physical memory • Boot.ini settings - /PAE, no /3GB • DEP – Essential Windows program only
ControlCenter things to consider Am I forgetting something? • Is StorageScope in the right place? • Move to a dedicated server • Best Practice is to relocate StorageScope 5.2 before upgrading • Can be moved later but is more complex • Remote Access Server? • Place for Storage Admin tools to be run from • Console • Performance Manager GUI Client • How many WLA Archivers are needed? • More is not better • I have multiple Datacenters, should I have a WLA Archivers in each? • MO’s will not automatically be assigned to the closest WLA Archiver
Other things to consider SAN and Storage Related? • What do I need to manage my SAN? • SMI Providers for McData and Brocade? • Where to find the SMI Providers? • http://www.brocade.com/services-support/drivers-downloads/smi-agent/application_matrix.page • What Solutions Enabler Version • Do I need SMC for ControlCenter 6.1? • Yes • Where can it be install? • ControlCenter Link-n-Launch of SMC
Post Installation of GA Software Are there patches that are needed? • Periodic Update Bundles • Provides all pervious patches and enhancements • UB1 Nov 2008 (obsolete) • Patch 4414 • UB2 Feb 2009 (obsolete) • Patch 4459
Post Installation of GA Software Are there patches that are needed? • Periodic Update Bundles • Provides all pervious patches and enhancements • UB3 April 2009 • Include all previous patches and relevant SQL Scripts • Enhanced Invista Support • Support for VMware VMotion • Brocade/McData Interop mode 3 • Tolerance for Symmetrix V-MAX • UB5 Target release is early June 2009 • Include all previous patches and relevant SQL Scripts • Support for Symmetrix V-MAX • New Install/Upgrade CD Media • Will be released at the same time as UB5 • Will include all patches UB3 and earlier • UB5 will need to applied
The AQP process Ongoing Qualification Process for ControlCenter • New Feature Summary for April 2009 • ControlCenter 6.1 • The EMC Invista Agent supports Invista 2.1 and 2.2. Update Bundle 3 or higher is required. • Add support for IE8 • Add support for Support Brocade interop mode 3 • Added the host level performance metrics on AIX platforms for Performance Manager • Add support for Brocade SMI Provider 120.8.0 on ECC & on VMware • Add support for v2.6.1 of the McData EOS - Fusion SMI-S Provider on ECC and on VMware
Resources Available Support Resources for Customers • Powerlink Forums • Excellent resource for all EMC technologies • You can learn lot’s just by “lurking” • You can learn even more by participating… even if you can’t answer any of the questions today, you can usually get your questions answered • If you aren’t a member yet, join today! • Powerlink https://powerlink.emc.com • Read ALL the documentation available before you even touch the servers • Planning and Installation Guide • Support Matrix • Performance and Scalability Guide • Even if you don’t think you are going to use certain features, read up on them so you don’t shoot yourself in the foot
Resources Available Support Resources for Customers (cont.) • EMC Professional Services • Account Team • Make sure you engage your Account Team early on in the planning. Your TC should be able to provide you additional support/advice as needed even if you aren’t engaging PS • ControlCenter Online User Community • This community is more focused on the future of ControlCenter • Interact with Product Managers, Developers, etc to help influence the future of ControlCenter • If you aren’t a member yet, join today! • http://www.controlcentercommunity.com
Find your Serve’rs SPEC Rating • Navigate to the SPEC website at http:\\www.spec.org, • From the top navigation bar choose "Result", "CPU2006" then "Search CPU2006 Results.“ • From the drop down list for Available Configurations, select "SPECint2006 Rates.“ • Select "Advanced" radio button for Search Form Request and click on "Go!" • Fill out the form with appropriate information (e.g. Hardware Vendor, System, # Cores, # Chips, # Cores Per Chip). Note: In SPEC nomenclature, the term for "Processor" is "Chip". • Submit the search by clicking "Fetch Results" at the bottom of the page. • The value listed under Baseline column for your host is value that s compared to the minimum specified in this document. Note: If the CPU2006 benchmark is not available, try the CPU2000 benchmark. The steps to find the result for CPU2000 benchmark are the same as CPU2006.
Primus Solutions • CC 6.1 upgrades • emc197536 Handling CC 6.1 Upgrade Issues • emc199181 Upgrade from CC 5.2 • CC 6.0 Install • Emc194752 CC 6.x Troubleshooting Guide • Host Agents • emc198305 & emc201524 Rediscovering installed Agents • VMware • emc203588 VMotion Support • emc166271 General VMware Support Information • Cisco Switches • emc147089 Interop Modes
Before Spinning CDs • Execute a Roll Back plan • Download and install the correct Backup and Restore Scripts • HF3322 (CC 5.2) • HF3928 (CC 6.0) • HF4365 (latest) (CC 6.1) or HF 4345 with the Migration Documentation • Run the Backup.bat • Copy ECC and ECCbackup directories to a Network Share • Set Oracle services to Automatic and start them • Set ControlCenter services to Automatic and start them
Final Preparations • For Installs and Upgrades, add this at the beginning of the path variable to eliminate a DLL conflict • <drive>:\ecc\repository\admin\ramb_scripts; (Remove when install/upgrade is complete, but leave the original entry found later in the path) • Libeay32.dll • Saleay32.dll • Update Bundle 3 is the last time this will be needed • Test for accessing the Repository and a shortcut to ..\ramb_scripts • ecc_getversion (ControlCenter), • emcsts_getversion (StorageScope)
Final Preparation for upgrades • Upgrading CC 5.2_SPx to CC 6.0, apply Patch 4013 to CC 5.2_SPx • Upgrading CC 6.0 to CC 6.1, apply Patch 4399 to CC 6.0 • ControlCenter 6.0 Free Space View Patch • Causes the upgrade to CC 6.1 to fail • Check for SQL_PATCH_HF4028.bat in the ..\ramb_scripts directory • Apply Patch 4599 which will reverse the patch • These problems will be handled automatically with the ControlCenter 6.1 install/upgrade media obtained from EMC after May 22nd, 2009
Upgrading StorageScope 6.0 • If multiple users are defined for StorageScope, apply Patch 4378 • Sets the preferred StorageScope Repository User account • If unsure, install Patch 4378 and run it • Review if multiple users have been defined • If only one, exit script
SMC & Solutions Enabler • ControlCenter 6.1 GA through to Update Bundle 3 • Solutions Enabler 6.5.0, 6.5.1 & 6.5.2 (Recommended) • SMC 6.0.2.5 & 6.1.1.2 (Recommended) • Known compatibility problem with SE 6.5.3 • Schema change in SE • Data type was changed to 32/64 bit, and not compatible with CC 6.x up through UB3 • ControlCenter 6.1 Update Bundle 5 • Solutions Enabler 7.0.0.0-914 & 7.0.0.0-915 and higher • SMC 7.0.0.3 and higher