Austin Energy Power Generation IA 8.5 Secure Edition Upgrade Austin Texas

2. Austin Energy Power GenerationIA 8.5 Secure Edition UpgradeAustin Texas Session Name: Austin Energy Session ID #:PBFX113 Presented by Wayne Ray 11/08/11 Invensys proprietary & confidential

3. Introduction • Wayne Ray – Invensys Field Service – Austin Energy Project team • Marc Richards – Invensys Field Service – Austin Energy Project team • Austin Energy Project Team Members Include: • Tung Nguyen – Invensys Lead Engineer • Len Vining – Invensys Project Manager • James Basset – Invensys Cyber Security • Michael Velarde – Austin Energy Project Manager/Site Engineer • John Edwards – Austin Energy Principal Design Lead With the combined efforts of the above and multiple other personal that assisted, the Austin Energy project was a success.

4. Sand Hill Energy Center • Constructed: 2001-2004 • Location: Del Valle, Texas • 6 Simple Cycle Units • 1 Combined Cycle Unit • Fuel: Natural Gas • Output: 580 MW

5. Decker Creek Power Station Constructed: 1967-1978 Location: North east Austin, Texas 2 Gas-Fired Boilers – Also use Fuel Oil 4 Gas Turbine Units Fuel: Natural Gas Output: 926 MW

6. Justification for Upgrade to I/A 8.5 Secure • Resolve issues with failing hardware. Both AW’s at Decker Plant had failed several times within the past year. • Wanted to resolve obsolescence early. Already aware that legacy Gateways were not supported. Gateways at Sand Hill Plant had failed within the last year. Invensys no longer sold replacements. • Needed a secure system with domain controllers to handle NERC Critical Infrastructure Protection Policies associated with cyber security. Austin Energy does not have any Critical Cyber assets at this time, but upgrade provided preparation should policies change. • New System Administration features on the upgraded system promised easier and less time-consuming maintenance.

7. System Architecture • The NERC Advantage Upgrade Project consisted of a switched fast Ethernet network called a MESH control network. This network architecture integrated all control processors and workstations in a 100 MB / 1 GB Ethernet network using fibre optic cabling. The MESH control network is designed to provide multiple communication paths between any two devices or stations connected to the network.

8. Pre-work performed by Austin Energy Fiber pulls from server room to control room as RGU’s were used. Coordination with IT during setup of firewall. Also had to reserve IP addresses through them. Coordination with other software systems OSI-PI, Satellite Clock, etc. Created secure Server room only accessed by approved personnel. Ran Critical Service Panel to Server room for new racks that would house WP’s, AW’s, Domain Controllers, Mesh Switches, etc.)

9. Invensys Cyber Security Group Prior to upgrade, Austin Energy had used Invensys Cyber Security group to harden existing Plant Control LAN at Decker Plant to comply with NERC cyber security good practices. Cyber Security group had installed and configured Juniper Firewall and Intrusion Detection Device. Zones were created to separate plant systems. DMZ was created to isolate Plant Control LAN from IT Corporate LAN. All firewall and Plant Control LAN maintenance is now handled by Controls Engineering group.

10. Cyber security Zones

11. Benefits of I/A 8.5 Secure Upgrade Solved hardware failures and obsolescence issues. Domain Controllers have helped with cyber security (logins, profiles, etc.) Austin Energy is now better prepared should NERC policies consider these units Critical Cyber Assets. System Administration of I/A is less difficult with 8.5. Symantec automatic backups have reduced maintenance time. System Monitor Interface is much more user friendly and displays a lot more information.

12. Concerns of I/A 8.5 Secure Upgrade Windows OS is “buggy”. Have had several “Windows”-type failures during the first few months of operation (blue screen of death, frozen screen/keyboard/mouse, services failing to start, etc.) We experienced an issue with a USB extender connected to a KVM switch. The KVM caused a delay with the USB 2.0 devices causing the machine to blue screen. Resolution was a different style KVM switch. Resolution to services failing to start was a Quick Fix issued by Invensys. The QF allowed services to start prior to logging in to the secured account of the machine and run as a local system account instead of the local user account.

13. Concerns of I/A 8.5 Secure Upgrade • Hardware issues – gateway communication problems, PIO bus errors due to older rev FBM’s, off – platform historian issues, trending issues. Invensys does have a plan to address these issues but hasn’t had the opportunity due to lack of unit downtime. • Gateway Communication issues – Timing issue between device and IA system. To be resolved when plant conditions allow Invensys to troubleshoot issue. • PIO Bus errors – Resolved with replacing older FBM’s that had capacitors that were causing to much load on field bus. • Trending issues – QF to be installed when plant conditions allow Invensys to do this

14. Concerns of I/A 8.5 Secure Upgrade Training on new system. May take technicians awhile to get used to new system, especially secure side of upgrade. Training class has helped but will need to attend other classes.

15. Question and Answer Session