Data Center Centralization for Performance Improvement

1. TITLE Data center centralization and operation via virtual hosted workstations leads to performance improvement and hardware location independence J. Bocarro (SNC-Lavalin) / Client: HQ Transénergie

2. Introduction Hydro Québec Transénergie owns and operates the transmission network for the entire province of Québec Business streamlining initiatives let to restructuring options for the operations of the network While restructuring was mandated, improvement in performance, reduction of maintenance costs as well as security concerns were equally emphasized along with the restructure process

3. Introduction Pre-Centralization Seven Control Centres + backup sites • Geographically distributed according to region Seven Maintenance Centres • At the Control Centre Pre-production test sites Training site

4. Introduction • The core concept of centralizing hardware and software maintenance was seen as key to realizing the business goals • The what, when, how, why were subject of a feasibility study. • The study was performed validating various architecture and technologies as well as investigation of implementation strategies. • Result : Citrix thin client architecture would best meet performance optimization, reduction of hardware , maintenance time reductions and increased security goals

5. Feasibility Study Identify Key elements or categories • Virtual workstation – thin client • Architecture based on master and backup sites with a single database ( Fusion of databases) • How to communicate with Field RTU’s? • Integration with External systems (distribution, production, weather, etc…)

6. Feasibility Study Categories Virtual Workstation Category1 Category 2 External Systems Category 4 External Systems Category 4 Control Centre 1 Control Centre 2 RTU’s Category 3

7. Feasibility Study Strategy For each category: • Current state versus business objective • Prototype the proposed solutions • Reuse technology or processes from prototyping • Identify any Software development required

8. FeasibilityStudy: Category 1:Remote Display Workstations Hardware / architecture considerations • “Hollow PC” with appropriate graphic capability • Servers , powerful enough to support multiple clients • Thin client technology • Best technology available to satisfy the requirements • Infrastructure requirements and costs

9. Feasibility Study: Remote Display Workstations – Solutions considered

10. Study – UI Prototype • Based on Client criteria, and the preliminary analysis , the Windows / Citrix server with “hollow” PC was selected. • Prototype was constructed and performance tests were performed with a large representative database and number of users.

11. FeasibilityStudy: Category 3RTU Communications • RTU connect to one single centre. Prototype testing led to IP/HSTC as preferred communications. But > 350 out of the ~550 RTU’s can communicate via IP • Basis of study: • Status Quo – Serial lines ( actual) • Conversion of ADCCP messages from IP to ?? • Send IP messages from Business Centre (BC) via the CIT

12. Study: criteria Communications • CSICO card to “translate” ADCCP frames then send over IP • Needed more infrastructure • Protocol converter • Too costly • Convert X25 to IP • Labour intensive - modifications at each RTU SMP

13. Study: criteria Communications: • Keep the Front End processors at the BC’s, capitalizing on the sync-async., conversion capability of the DAC . • This was the selected solution

14. Selected Communications architecture

15. FeasibilityStudy: Communications withExternalsystems - Category 4 2 major groups: • Communications via ICCP • Communications with Distribution • The major issues was ensuring adequate processing for the “fused” databases and sufficient bandwidth. • For communication with Distribution, an enhancement to the protocol scanner was deemed necessary to permit processing multiple distribution posts

16. FeasibilityStudy: Category 2Software Developments User interface/graphics • Accommodation for Citrix – memo buttons, selection , pointers, printing • Graphical object and navigation enhancements for performance • Alarm Browsers • Outage Scheduling(OSS) - filters for combined employee databases • Pre-switching modifications for combined network

17. Fused site/hardware Configuration Final configuration

18. Deployment Strategy • Database fusion • Upgrade the Computer hardware • Additional CPU, memory, etc.. • Remove server /workstation hardware from existing control centre ( to become a BC) • Install software in pre-production site • Prepare control Centre (CIT) • Communications testing • Citrix Configuration • Proceed with cut-over from old to new CIT

19. Problems encountered Major categories: Graphical Event Identification Performance Operational

20. Graphical Issues • Number of users per Citrix server • Client limit the number/server in addition to “Client -assigning software” • Citrix-related issues to printing, pointing indicators, screen management • Resolved with configuration/trial error and some software modifications • Performance due to Citrix for remote centres • Resolved at the graphical software level

21. Event Identification • With Citrix server – multiple user to the server – there was no obvious method to correctly associate user to control action • Software modification was necessary to associate user to action

22. Performance 3 Areas: • Communications • Graphical • Application

23. Performance - Communications • Majority of problems related to “traffic” volume. Telecommunications infrastructure modified as a result: • External Communications Links from the Broadcast LAN moved • Deployed NIC Teaming • Ie: FEP, Distribution , ICCP other external links

24. Performance Graphical Causes • Increased numbers of graphical objects /per schematic • Increased traffic due to fused database • Resolved: • Software modifications at the graphical object layer

25. Performance: Applications • Due to usage of certain Operators multiple windows of same application were opened causing large CPU usage of servers while not affecting the Citrix server. Eg ( 75 windows of same application for a single user) • Client would not accept a limit! • Previous local workstation probably had the same problem but the user was sensitized to the issue, and in the worst case, only the PC would “freeze” and not the application • Resolved: Further optimization of selected dynamic objects and server modification

26. Problems - Operational • Fusing the database ”inadvertently” fused the network. • Problems with Operating practices • Problems with PSV applications • Resolved: • Modifications to the applications • Operator practices enforced

27. Operational and Commercial Benefits Operational: • Re-localization of servers to maintenance centres reduces down-time in case of computer maintenance • Creation of “natural” backup in case of hazardous conditions for Operators

28. Commercial Benefits • Reduction of numbers of personnel needed to operate system during vacation, holiday periods • Localization of computing hardware leads to reduction of hardware inventory, spares, and overall maintenance costs. • Installation of software updates is easier as software needs only to be installed on servers and not on numerous PC’s

29. Conclusion The implementation of Citrix permitted the following: • Centralization of computer Hardware • Control-centre location independence from Hardware • Cost savings in terms of maintenance and updates • Cost savings in terms of personnel assignments • Operational advantage in terms of co-work-assistance, and natural backup