Automation to the Rescue

1. Automation to the Rescue Jim Malnati and Robert Jamieson UTF 2008

2. Agenda • Client Experiences • What else can Operations Sentinel do? • Futures • Summary UTF 2008

3. TravelSky Technologies Ltd • Major provider of airline applications in China • Major applications for reservation, inventory and departure • Significant year-on-year growth plus the Olympics • 8 production OS2200 partitions on 3 Dorado servers • Plus many dev/test/DR partitions/servers • 2 DMX subsystems for each appl – SRDF to DR • Over 100 ES7000 partitions for OMSE • CRS txn rates forecast at 4,000/sec and 50,000 IO/sec • Scalability, availability and response critical UTF 2008

4. Challenges at TravelSky • Increased complexity without additional staff • 3 to 8 partitions • Increased txn workload • Increased IOs • Increased batch jobs • Shared resources e.g. tape drives, locking • Olympics 2008 has meant significant political pressure • SPO and Opcon/xps are relied upon UTF 2008

5. SPO at TravelSky • OS2200 console • Automated operator key-ins • Monitor key system variables • Alert and automated actions if exception detected • Pass KPIs to OMSE (ES7000) for network traffic balancing • SOMA used to automate tape drive sharing • DMX automation and monitoring • SRDF Async/Sync/Semi-Sync, RDF ports etc • Consolidated console and logging tool UTF 2008

6. DMX Monitoring and Alerting • Probe function for monitoring • SRDF state, RDF directors, RDF link • Configurable with default of every 5 mins • Alerts raised for errors and command success • Severity used to differentiate • Informational to Critical • Use Alert Details to get help text (saved IP from experts) • STATUS application for graphical single view • Icon colours indicate state or problem UTF 2008

7. SRDF Synched Port offline SRDF Split Probe ON Status Diagram – Colours indicate States UTF 2008

8. SRDF Function Triggers • SPO Console Application Menu UTF 2008

9. SPO SRDF Alerts and Help Text UTF 2008

10. Logging • Numerous log files maintained including symm_host UTF 2008

11. Opcon/xps at Travelsky • Task based license – 600 tasks per day for OS2200 • Job scheduling for most OS2200 batch work • Database dumps • System file backups • Application jobs • Simplified upgrade from 3 to 8 production hosts • Planning upgrade to new Opcon/xps release • Planning for improved SPO-Opcon/xps integration UTF 2008

12. SPO at Asian bank (Libra client) • Monitor MCP console traffic • Alert if no payment gateway transactions received in 10 secs • Alert if more than 10 txn timeouts in 10 secs • Automated operator key-in if deadlock detected • SRDF monitoring and automation • Metering usage monitoring and logging • Proved they didn’t need to worry about runaway tasks • Showed what their processing profile was like UTF 2008

13. Deadlock Automation • Problem • Deadlock with IBM port caused all txns to queue • Credit cards could not be used • Analysis • Used to rely on end-user advising operator advising support • Take 5-30 minutes to resolve while many txn blocked • Development of SPO script in 30 mins • Resolution occurs in under 30ms with informational alert raised • Benefit • No loss of revenue, no user/operator/support cost, satisfied card user UTF 2008

14. ClusterDisplay CurrentODTs SPO Workstation Desktop AlertIcon STATUSfor DMX UTF 2008

15. SPO Status Diagram for DMX • Initialised state indicated by blue colour • Red indicates alert – probe not active, R1-R2 not in sync • Yellow indicates RBCV is split UTF 2008

16. SPO Menu Driven Functions • Submit check of the RDF status UTF 2008

17. Other Automation users • 3 Korean banks use DMX automation • Aust DoD, Cathay, EDS, LTNZ and others use Opcon/xps • Angkasa (MCP), LTNZ, EDS use SPO • SPO Basic Edition comes with IOE – MCP and OS2200 • Monitor the ClearPath system • Needs Window server • Some training and consulting UTF 2008

18. What Else Can SPO Do? • Monitor ClearPath, Windows, Unix, AS/400, environmental devices etc • Including tape library controllers (ACSLS, VTLs) • Integration with other tools • AppManager, problem ticket systems, SNMP feeds • Opcon/xps • Provides a unified console and logging facility • Can provide basic BPM UTF 2008

19. Fidelity Information Services (FIS) • Large MCP user: provides banking application services • FIS understands the value of automation • Cost reduction program (SPO to replace other mgmt apps) • Focus on improving quality to customers • Needed to upgrade SPO 6 (unsupported and on UnixWare) • SPO 9 now in production; Operations Sentinel 11 trial • Late 2007: • SPO 9 Getting Started Service • SPO 9 Formal Training • Operations Sentinel 11 trial implementation UTF 2008

20. Operations Sentinel 11 Dev. Goals • Complete rewrite of SPO graphical interface • Leverage Microsoft investments in GUI technology • Take advantage of .NET, C#, XML, etc • Remove dependence on 3rd-party products (SNAP/Geneva) • Provide a Microsoft-compliant interface • Implement global configuration • Standardise tools and utilities (common look and feel) • Implement customer-requested features UTF 2008

21. SPO 9 Graphical Interface UTF 2008

22. Operations Sentinel 11 Demonstration UTF 2008

23. SPO Audible Alerting (SPAA) • Visual alerts are essential: some require audible emphasis • Need to synchronise ACK/clear of alerts • SPAA C# program installed on each SPO Workstation • SPO Alert Policy specifies: • The sound file to be played • List of workstations on which to play it • Option to kill the SPAA GUI (stop sound) after time period • Test program to drive SPAA with event reports UTF 2008

24. SPO Audible Alerting (SPAA) UTF 2008

25. UTF 2008

26. Shared Object Manager Application (SOMA) • Tape drives shared between Unisys, IBM, Tandem, and Sun hosts • SOMA means fewer tape drives • Error-free movement of driveson an as-needed basis • Multiple “tape pools”configured • Individual drives and/or hosts excluded • SPO 11 SOMA soon SOMA SPO andOpCon/xps Windows 2000or 2003 Secure Wide Area Network UTF 2008

27. Less Than Optimal Drive Sharing UTF 2008

28. A More Cost-effective Approach UTF 2008

29. Operations Is Affected • Greater equipment efficiency requires manual intervention by Operations staff • Operators are diverted from system monitoring • Many systems must be examined for surplus tape drives • Repetitive, manual keyins required on multiple systems • Entire process is time consuming and error prone, especially with heterogeneous hosts • Batch processing is delayed while operatorslook for available drives UTF 2008

30. Overview of SOMA • Automatic, error-free movement of shared tape drives between heterogeneous systems (Unisys, IBM, Tandem, etc) • Configured package of SOMA and SPO automation • All actions logged, for audit trail and metrics • SPO Status provides a “head-up display” so Operations can easily track tape drive movement • Built from standard Unisys SPO product and the SPO C# API UTF 2008

31. HOSTA HOSTA HOSTB Tape Drive Sharing Without Automation Operator searches other hosts for free tape drives while HOSTA waits. HOSTA: “I need 4 tape drives. Now!” 1 2 Host with a free drive is found and the operator “downs” the drive through system console keyins. 3 HOSTC HOSTD 4 If the tape drive is downed correctly the operator can bring it up on HOSTA. HOSTA: “Well, that’s 1 of 4….” 5 The cycle repeats, as other important console events are ignored and the system waits. 6 UTF 2008

32. HOSTB HOSTA HOSTA HOSTA Automated Process Flow 1 “I need (4) drives of type (high-density-tape-pool)” SOMA 2 Automated Shared Object Manager: “Get (drive) from HOSTB” 3 4 “Down (drive) for me” “OK, (drive) is down” 5 “UP (drive)” 6 The cycle repeats three more times, without operator intervention. UTF 2008

33. UTF 2008