1 / 46

Building Reliable, Secure and Manageable Substation Communications

Building Reliable, Secure and Manageable Substation Communications. Dragan Dokic | CCIE, CISSP, MCSE. Introduction - Experience. Dragan Dokic | President, Summit Energy Tech Focus on utility sector Infrastructure systems management Custom business systems software development

mada
Download Presentation

Building Reliable, Secure and Manageable Substation Communications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE

  2. Introduction - Experience • Dragan Dokic | President, Summit Energy Tech • Focus on utility sector • Infrastructure systems management • Custom business systems software development • 16 years of experience in IT industry • 10 years in utility sector • Managed network operations for PNGC Power [Portland, OR] from September 2002 to October 2011 • Presentation focuses on lessons learned in field network reliability, security and manageability from this experience

  3. Introduction • PNGC’s 2001 – 2011 field network • 92 office, substation and repeater sites at 11 distribution utilities inOregon, Idaho • System mission • Gather real-time load data 24/7 for power scheduling operation in Portland • Support local utility SCADA/AMI/Site Security operations

  4. PNGC Power WAN – July 2011

  5. Toledo, OR

  6. Boardman, Oregon

  7. Junction City, Oregon

  8. Lewiston, ID

  9. Malta, ID

  10. The Moon

  11. Areas of Focus Reliability Security Manageability Presentation available for download at summitenergytech.com in the Events section

  12. Reliability – Network Design • Keys to success • Diversity in media • Combine land lines, fixed wireless [private/public], mobile wireless and satellite • Diversity in providers • Local and national • Dynamic Routing [OSPF] • Routers exchange knowledge of local network with neighboring routers • Enterprise grade routers / switches a requirement • Perfect world configuration • Private wired/wireless ‘island’ with two Internet gateways using distinct media and distinct providers

  13. Connectivity overview Backup router Primary router

  14. Link cost overview Primary Backup

  15. Link cost calculation Sub A -> Main Office via Satellite tunnel: 3 + 1 = 4

  16. Link cost calculation Sub A -> Main Office via 900Mhz+DSL tunnel: 1 + 1 + 1 = 3

  17. Open Shortest Path Link cost via Satellite tunnel [4] higher than via DSL tunnel[3]; therefore, packets will traverse 900Mhz/DSL tunnel in normal operation

  18. Normal Operation Open Shortest Path From substation A to Main Office

  19. Normal Operation Open Shortest Path From substation B to Main Office

  20. Link down operation If DSL tunnel is down, packets will traverse satellite tunnel; Sub A  Main Office X

  21. Link down operation If DSL tunnel is down, packets will traverse satellite tunnel; Sub B  Main Office X

  22. Questions?

  23. Security – Overview • Wireless link encryption • Function specific VLANs • No default routes!

  24. Wireless Link Encryption • Media device level [e.g. Radio, Modem] • WEP, WPA, WPA2 • Routing device level [e.g. Cisco 891 router] • IPSEC • End device level [e.g. DIGI TS4 port server] • SSL

  25. At what level to secure data?

  26. Security - Wireless Link Encryption[continued] • Most secure option? • Use all three if management overhead is not an issue • Most efficient but secure enough option? • Use routing device site-to-site VPN capabilities • Advantages: • Support for best commercially available security technologies [e.g., AES-256] • Comprehensive change logging capabilities • Standardized configuration throughout the system [less management overhead]

  27. Security – Function Specific VLANs • Define VLAN’s per business function • SCADA, AMI, Security System, Wireless, VOIP, Network Mgmt. • Firewall traffic between VLANs on need-to-access basis • E.g., Prevent personnel attached to substation wireless VLAN to access documentation stored on a server at the main office from accessing recloser controls in the SCADA VLAN • Reliability advantages • Non-critical VLANs [e.g. AMI, security] can be shut down automatically/remotely if link quality is too poor to carry all traffic, but good enough to carry SCADA

  28. One VLAN per business function

  29. High-speed link outage scenario

  30. Security – No Default Route! • Do not use default routes through service provider-supplied gateways • Define a singlehost route back to the main office, then establish default route through VPN tunnel • This is the most effective method to prevent attacks sourced from the Internet • Always use in conjunction to regular firewall configuration lists [not a substitute!]

  31. Less secure Provider gateway

  32. More secure Provider gateway

  33. Questions?

  34. Manageability - Overview • Tools – network management systems • Addressing – developing a scheme • Watchdog system – preventing lockout

  35. Manageability– Tools • Network Management Systems [NMS] • Protocols used • SNMP, Syslog, ICMP, HTTP • Applications • PRTG • Solarwinds Syslog

  36. Manageability– Tools [continued] • How to collect data? Push vs. Pull • Pull: Poll devices using SNMP/HTTP/ICMP at regular intervals [e.g., every • Push: Devices send data per defined event triggers • SNMP traps • Syslog messages • What data to collect? • Availability [ping] • Network utilization • Input voltages • RSSI [radio link quality]

  37. Manageability– Tools [continued] • Pull example: • 5 minute SNMP poll of UPS for input voltage • If voltage drops below threshold of 108VAC for a duration of time longer than 5 minutes, an alert will be triggered by NMS [e-mail, text message, event log] • But what if voltage drops for 2 minutes only in between polls? You may not know it even happened. • Push comes to rescue: • UPS sends SNMP trap to NMS as soon as voltage drops below 108VAC • Alert is triggered by NMS when trap is received

  38. Paessler PRTG – Screen shot

  39. Solarwinds Kiwi Syslog – Screen shot

  40. Manageability– Addressing • Develop consistent scheme to use system wide • Recommended private range: 10.0.0.0/8 • First octet: same for entire system • Second octet: site ID [e.g. 8=Springfield Sub] • Third octet: business function ID [e.g., 4=AMI] • Fourth octet: device itself [e.g., Collector #1] Subnet Mask [255.255.255.0] 1st octet ‘fixed’ 4th octet = device 3rd octet = vlan/business function 2nd octet = site ID

  41. Manageability– Addressing [continued] • Large network? • Group sites by region using second octet • Allows for address summarization if needed. • Example: • Eastern division region: • 10.64-127.0.0 • Summary address: 10.64.0.0/10 • Western division region: • 10.128-191.0.0 • Summary address: 10.128.0.0/10

  42. Manageability– Watchdog System • General concept • Reboot key remote communications devices if connectivity to central site is interrupted • Benefit • Prevent unnecessary site visits due to • Operator error • Device lock-up [e.g., buggy firmware, heat issues]

  43. Manageability– Watchdog System [continued] • Hardware requirements: • SNMP-capable switched PDU with task scheduling and delayed power cycling command capabilities • Example:APC AP7900 8-port 15A PDU • Software capability requirements: • Centralized command override mechanism using NMS • Send SNMP ‘Set’ to cancel pending power cycling command

  44. Manageability – Watchdog System Example • ‘Delayed’ power cycle schedule is defined on PDU: • Outlets to power cycle: 1,2 [e.g., radio, router] • Frequency: 60 minutes • Command execute delay: 30 minutes • Network management system running at main office sends an SNMP delayed power-cycle command cancel message • Frequency: every 5 minutes • Process • If delayed power cycle cancel command cannot reach the PDU at least one time during the 30 minute reboot delay period, outlets 1 and 2 will be power cycled and communication will (hopefully!) be restored

  45. Questions?

  46. Thank you!

More Related