RATC/ARPA-E Review meeting RATC Communication System

1. RATC/ARPA-E Review meeting RATC Communication System Dr. Sami Ayyorgun – ACS Dr. Alex Sprintson – TAMU Dr. John Sucec – ACS

2. RATC communication system • Addresses RATC communication needs • Support data flows among RATC components • Support RATC performance requirements, e.g. Quality of Service (QoS) constraints • Support cybersecurity of RATC operations • Leverages existing substation and WAN communications infrastructure • SCADA/EMS • WAN

3. RATC communication needs • Intra-substation • substation – control center

4. RATC Data Flows

5. RATC Data Flows

6. RATC Data Flows IEEE 1379 ModBus+ UCA/MMS (ICCP or IEC 60870-6/TASE.2

7. Progress since June 29 review • For each data flow • The source of data • The destination of data • Periodic/On Demand • Estimated size of the data • Delay requirements/ QoS constraints

8. Current status and actions • Establish connections to existing standards

9. Current status and actions • Finalize the system and communication architecture of SCADA • Centralized approach • Most of the data processing is done at the central substation • Decentralized approach • Substations can perform certain operations • Measurements and Model Data Integration • Circuit breaker Operation Evaluation

10. Actions until the next meeting • Finalize the data flows • Mapping to physical architecture • Implementation in typical utilities • Analysis of QoS requirements

11. Responsibilities TAMU (Sprintson) ACS (Ayyorgun) • Identification of data flows • Identification of Quality of Service (QoS) constraints • Participate in protocol section and developing new protocols if needed • Analysis and evaluation of proposed solutions • Identification of RATC data and control flows. • Selection of protocols. • Identification of communication architectures. • Identification of communication performance requirements. • Identification of communication resource-adaptation needs. • Identification of performance and cybersecurity vulnerabilities, and potential mitigation approaches.

12. Progress Since June 29 Review and Ongoing Work • Completed identification of candidate power systems operations (PSO) network architectures pertinent to RATC, including both physical and network topologies • Intra-substation and inter-substation communications • PSO WAN design • Intra-control center (CC) communications • Completed mapping of RATC network data and control flows to content sources/receivers and the network communication paths connecting them • Began specification of RATC substation communications requirements • RATC communications within the substation; communications between the SA RATC elements and CC RATC over the PSO WAN • Began mapping of network flow event sequences • Began evaluation of potential risks/issues/challenges including assessment of the following as they relate to the communications architecture, e.g.: • QoS-related performance goals, network congestion, network availability, cyber-security vulnerabilities

13. Global View of RATC Data Flows and Protocols A Intra-CC flows Flows between substation and CC Inter-substations flows Intra-substation flows B C D

14. Summary of RATC Network Data and Control Flows • RATC modules will leverage the IED and model data maintained by existing utility substation data concentrator (SDC) and control center data concentrator (CCDC) systems but in some cases may need to acquire certain PSO data on its own. • Distributed RATC software processes also are expected to communicate with one another • RATC control signals computed by a CC RATC element will be disseminated to substation automation (SA) RATC elements situated at remote substation and power generation sites • Furthermore, local SA status and measurement data collected SA RATC elements will also be potentially backhauled to the CC RATC element

15. Example RATC Data Flow: Data Collection for Circuit Breaker Operation Evaluation • SA-RATC SW App triggers local circuit breaker operation evaluation (CBOE) data collection • SA-RATC CBOE SW App calls RATC Comms module for latest circuit breaker monitor (CBM) data • SA-RATCComms module: • Identifies SA LAN path and communication device adapter • Invokes SA-RATC Resource Adaptation to verify LAN conditions support request • Invokes RATC Cyber-Security to verify security policies permit request • SA-RATC Comms module completes CBM data transfer from CBM IED • SA-RATC CBOE SW App compresses / filters CBM data and calls RATC Comms module for transfer to CC-RATC • SA-RATC Comms module: • Identifies WAN path, CC-RATC comms port and DSCP marking • Invokes SA-RATC Resource Adaptation to verify WAN conditions support request • Invokes SA-RATC Cyber-Security to verify security policies permit request • SA-RATC Comms module completes compressed CBM data transfer to CC-RATC Comms module • CC-RATC CBOE updates CBM event records repository

16. Example RATC Data Flow Event Sequence Diagram

17. Internetworking RATC Communications Between Control Center and Remote Sites • The key communication content pertinent to RATC between substations and the Control Center (CC) is the SCADA information employed by a utility’s energy management system (EMS) • RATC elements situated at the CC likely will share network resources with EMS elements • RATC communications between CC-RATC and SA-RATC elements will share the utility WAN resources with SCADA communications • Data flow: SA-RATC agents send periodic status and event-driven alerts to CC-RATC; CC-RATC sends topology control commands to SA-RATC agents • Risk: If a WAN path between CC and substation be congested, critical RATC event or command may be lost or unacceptably delayed. • Mitigation(s): DSCP values of RATC data packets should be marked according to priority/urgency and utility WAN should support DiffServ. RATC-specific communication control algorithms need to be deployed.

18. RATC Communications Security Considerations • Risks: Compromised communication assets (links and devices) could lead to wrong input data (e.g. load-flow data) and wrong output data (e.g. switching instructions) to be sent and received. This could lead to wide-area system instabilities. • Mitigation: RATC cyber-security considerations should be part of the RATC communication architecture design. So far, we have discussed RATC communication architecture from performance perspective. • Protocol AND application (RATC) specific, stateful cyber-security mechanisms need to be developed for RATC. • RATC security agents need to be installed at each substation and the CC. • Enterprise security solutions are not sufficient to protect various vulnerabilities (in particular for communications both intra and inter substations). This is due to performance restrictions (e.g. involving encryption). • One novel approach (that we will pursue) is to develop finite-state-machine models of proper behaviors of protocol (e.g. 61850) actions specific to RATC (e.g. control of RATC object models). These FSMs would be used as “sensors” to detect potential attacks and to coordinate mitigation strategies.

19. Example to RATC Security Agent FSM Sensor • For communications and coordination among RATC computers in CCs, we would create ICCP associations. RATC head-end may need to execute certain SBO programs/devices in both the associated control centers and related substations. An adversary could inhibit these operations by staging various attacks. The following is a FSM sensor example to detect such attacks. • We will determine potential vulnerabilities specific to RATC and develop a representative set of FSM sensors. • Another novelty about these FSM sensors would be to include physical states (not just protocol states, over which RATC communications take place).

20. Projected Additional Progress for Jan 2013 Quarterly Review • Complete identification of communication flows and architectures • Complete flow sequence diagrams for each flow class • Continue work on communications requirements • Confirm QoS requirements (data throughput, delay) • Confirm cyber-security requirements • Identify intra-CC requirements • Identify inter-substation requirements • Draft architecture of RATC communication modules for: • Network resource awareness/adaptation • Cyber-security • Continue RATC communications risk assessment and resource adaptation • Argue preliminary mitigation measures

21. ARPA-E June 29 Assessment Issues(5. Communications)

22. ARPA-E June 29 Assessment Issues(6. Operational data layer)

23. Backup

24. Example RATC Data Flow: Data Collection for Circuit Breaker Operation Evaluation

25. Illustrative Substation Physical Topology: Station-wide Ethernet LAN with IEC61850-8 • The figure below shows the complete migration switched Ethernet technology • Topology affords fully decentralized peer-to-peer communications among substation elements • Furthermore, access via the substation gateway router potentially supports direct communications between the control center (CC) and individual substation bay sensor elements • Relevant RATC data: DFR, DPR, CBM and PMU event records • Relevant standards/protocols: IEC 61850 object model, IEEE C37.118, MMS, TCP/IP, Ethernet extensions (e.g. IEEE 802.1q) • Risks/Issues: • For sub-millisecond delivery of time-critical IED data, switched network should be Gigabit Ethernet with proper QoS mechanisms. • Ethernet link for RATC data concentrator could be a bottleneck due to collection of IED data and dissemination of such data with the RATC modules and the CC. • Mitigation(s): Network elements should support IEEE 802.1p QoS extensions for time-critical IEC 61850 objects. RATC specific communication control mechanisms need to be developed.

26. Illustrative Substation Logical Topology: Redundant double ring SCN architecture • Risk: Limited physical access to remote makes SCNs highly vulnerable failure events • Mitigation: Figure here shows a modified (i.e. reduced number of switches) redundant parallel double ring logical topology with the breakout of elements of substation bay 1 presented in detail. • This logical topology addresses the issue of network availability • Here, any two Ethernet switch elements can fail and still the core of Ethernet switches will still remain connected • All network hosts/IEDs are dual-homed to protect against link failure to edge devices • Also, of interest here is the dual-homed connectivity of individual bay elements to the redundant Ethernet bay switches • This protects against single link failure events within an individual substation bay. • Communication performance disturbance analysis needed for substation networking

27. Inter-Substation RATC Communications • The figure below shows an example of inter-substation connectivity based on Gigabit Ethernet virtual private LAN service (VLPS) • Relevant RATC data: Notification of fault events and batch IED (e.g. DPR, DFR, CBM, PMU) data • Relevant standards/protocols: IEC 61850 object model, C37.118, MMS, TCP/IP, IEEE 802.1q • Risks/Issue 1: Even with Gigabit Ethernet VPLS, sub-millisecond delivery of data between substations over the WAN is not likely feasible • Mitigation: SA decisions that demand responses on the order of milliseconds after IED event, likely need to be made based on data already available to the local RATC agent • Risks/Issue 2: When high-speed WAN connectivity is not available, e.g. due to failure of primary link) or WAN coverage limited to slow (56Kbps) links, RATC inter-substation throughput will be low • Mitigations: RATC agents should be “resource-aware” and filter (e.g. drop) or mark (e.g. DSCP value) according to the priority/urgent of the inter-substation data

28. IEC 61850 Object Model Interface with Protocol Stack • One option (for data transfer) is to create RATC data object model in 61850. We would leverage 61850 mappings for such an object model. This would provide unity and interoperability for RATC to work in 61850 compliant communication environments. • IEC 61850 defines mappings between SA data models and underlying communication protocols: • Mapping of time-critical raw measurement data samples and Generic Object Oriented Substation Event (GOOSE) data to the Ethernet link layer • Mapping of client-server communications to the Manufacturing Message Specification (MMS) protocol which is an industry standard protocol that operates at the Application Layer in the TCP/IP protocol stack

29. Substation Comm. Transition to Ethernet • The transition to Ethernet-based substation communication network (SCN) architectures, that RATC would benefit, provides: • Switched Ethernet offers attractive scalability and ease of management/maintenance advantages over conventional wiring • The advancement of high-speed Ethernet technology helps with low-latency delivery needs of time-sensitive raw measurement data and GOOSE events • The deployment of IEC 61850-compliant intelligent electronic devices (IEDs) provides an interface to the TCP/IP stack (and Ethernet) that was not available in legacy devices. • For legacy systems, we could add protocol convertors for 61850 over Ethernet for RATC devices. However, this only helps with interoperability and unit, not necessarily with performance. For performance, communication resource adaptation for RATC in legacy environments play a critical role.

30. RATC Control Center Communications • The figure here shows the key functional areas of the CC EMS plus its interactions with the control center business management system (BMS) and remote site operations • Data Flow: As discussed previously, CC-RATC will share CC EMS network resources to obtain SCADA/RTU information, accessing the Control Center Data Concentrator (CCDC); Furthermore, the CC-RATC processes will also access Model data (static, dynamic) including model data (line, generation, load) and current state updates • Risks: Unconstrained CC-RATC over-the-EMS-network-access to the voluminous quantity of data maintained at CC could potentially disturb CC network performance. • Mitigation(s): RATC intra-CC communications should incorporate network resource-awareness procedures (TBD) to intelligently access data based on the available CC network capacity.

31. RATC WAN Design Considerations: Remote site connectivity using utility-owned infrastructure • The figure here shows utility-owned non-backbone WAN connectivity to remote utility sites – e.g. TVA system. • Depending on site network reliability and performance requirements, a mix of link technologies may be deployed to support disparate site internetworking needs • Key advantages of the utility-owned infrastructure include the ability to reach remote sites not serviced by public networks and improved cyber-security • Of particular importance to RATC communications is the ability for the utility network operator to manage how traffic is serviced with DiffServ according to a packet’s priority/urgency as indicated by it DSCP field • Communication performance disturbance analysis is needed for WAN as well.

32. RATC WAN Design Considerations: WAN connectivity with non-utility infrastructure • Risk: Using non-utility-owned WAN resources means less control over network flow QoS • Mitigation: RATC WAN flows should be deployed in close coordination with other utility flows so as to use WAN resources in a manner that protects urgent/priority data

33. Relevant Power Systems Operations Data Items • The RATC algorithm modules use various types of intelligent electronic device (IED) measurement data and utility energy management system (EMS) model data • Estimates of data item polling rates and data item sizes are based (in part) on discussions with TVA • The data update rates required by the RATC algorithm are currently under study by the RATC team