FirstWatch to VCS Migration

1. FirstWatch to VCS Migration Grahame I Curtis Morgan Stanley Dean Witter gcurtis@ms.com

2. Morgan Stanley Dean Witter • The preeminent global financial services company • Offices in 24 countries • Over 50 000 employees worldwide • Involved in the Securities, Asset Management and Credit Services Businesses

3. The Morgan Stanley UNIX Environment • 7 500+ UNIX Machines • 1 500+ UNIX Servers • Primarily Sun/Solaris with some SGI/IRIX Large Compute Servers • Standard Server Build with Redundant R/O Operating System/Application Availability via AFS • Software Distribution via AFS • 24x7x365 Availability Requirement

4. Why HA? • Reduce Downtime by • Reducing the risks from Hardware Failure • Reducing the risks from Application Failure • Reducing the recovery times after failures

5. VERITAS FirstWatch • Simple 2-node clusters • Asymmetric or Symmetric Failover • Host Level Failover * • No dependency mapping • Simple

6. FirstWatch at Morgan Stanley • Infrastructure created to easily build FirstWatch clusters • Hooks built into Enterprise Monitoring System • Services deployed include • File Services (NFS/Samba) • Sybase • Market Data Services • Autosys • MQ Series infrastructure • Web Infrastructure • Print Servers

7. Why Clustering? • Application Server Consolidation • Reduce complexity of the environment • Reduce overheads • Guarantee availability • Reduce hardware costs • Facilitate non-disruptive Hardware Maintenance • Campus Computing • Server Farms • Storage Farms • BCP • Failovers between buildings

8. VERITAS VCS • Up to 32 nodes • Asymmetric, Symmetric and N-to-1 Failover • Service/Resource level Failover • Dependency mapping • Support for SANs • Complex!

9. VCS at Morgan Stanley • Deployed in-house infrastructure to more easily configure clusters and services • Separated in-house resource definitions into an in-house MS_types.cf • MS_* agents to differentiate between in-house and VERITAS agents • Service configuration specifications • Define resource and inter-group dependencies • Build and administration scripts and Web-based GUIs • Consistency checking mechanisms

10. VCS at Morgan Stanley (2) • Trained VCS “specialists” from the Service groups • Facilitated knowledge transfer to others • Provided points of contact for escalation • Provided useful input in creation of infrastructure • Created FW Migration Utilities • FW services are most obvious target for VCS • Easy to map services and resources between FW and MS Service definitions • Implemented and deployed SAN solution • Implemented and deployed VCS/SRDF for BCP

11. Issues • Asymmetric, Symmetric or N-to-1 Failover? • Cluster Sizing • SSCI vs. SAN • Batch Scheduling in a cluster • Understanding Client Failover • Understanding VCS agent monitoring vs. FirstWatch • Resource Management (Share II, SRM)

12. What VCS Does NOT do • Make application recovery faster than the application itself can recover • Deal with clients that do not automatically handle failover

13. Benefits • Reduced Planned and Unplanned Downtime • Reduced Costs • Reduced complexity of the Environment • Facilitated BCP Mechanisms • Facilitated further consolidation

14. Lessons Learned • Ensuring Correct Service Definitions is Vital for VCS clusters • A better understanding of applications within the Environment is needed • Application dependencies • Inter-application dependencies • Applications that do not need HA • Critical vs. “Less” critical applications • Clustering facilitates other technology directions (VLANs, SANs, Large servers)

15. Plans for the Future • Integrate Service Brokers • Deploy SAN Infrastructure Globally • Deploy VCS 1.1 • “Loki” • VCS on NT • Deploy more applications that handle application failover better

16. Summary • Environment moving from distributed to consolidated paradigm • Requires higher availability and easier application management • Requires understanding of dependencies between applications and resources • Requires an infrastructure that supports SANs • VCS facilitates all the above!