Sandro Bologna - Claudio Balducelli – Giordano Vicoli ENEA – CAMO bologna@casaccia.enea.it

1. UNA PROPOSTA PER ESTENDERE LA CAPACITA’ DI SOPRAVVIVENZA DI RETI DI DISTRIBUZIONE DI ENERGIA INFORMATIZZATE – IL PROGETTO SAFEGUARD Sandro Bologna - Claudio Balducelli – Giordano Vicoli ENEA – CAMO bologna@casaccia.enea.it Alessandro De Carli – Giovanni Guida Università di Roma “La Sapienza” alessandro.decarli@uniroma1.it Convegno ENERSIS 2004 Milano 1-2 Aprile 2004

2. The World is a Network of Networks… Any Geographical Area, Any Network, Any Functional Area Is a Place of Vulnerability Oil and Gas Electricity Internet Core Telecommunications Water 2 The challenge Transportation

4. Organisational Infrastructure Inter-dependency Layered networks model Intra-dependency Cyber-Infrastructure PhysicalInfrastructure

5. Electrical Power Operators Independent System Operator for electricity planning and transmission Foreign Electrical Transmission Infrastructure Intra-dependency Inter-dependency Control and supervisory hardware/software components (Scada/EMS systems) Electrical Components generators, transformers, breakers, connecting cables etc Telecomunication Infrastructure National Electrical Power Transmission Infrastructure Oil/Gas Transport System Infrastructure Three Layers Model for the Electrical Infrastructure

6. CC Control and management layer (SCADA system) CNC WAN (Wide Area Network) CC Area 1 Area 3 SIA-R SIA-C SIA-R SIA-C SIA-R SIA-C Area 2 Data management network Remote Units Data Concentrator Control Centres Loads Generator Substations Physical Network General layout of typical control and supervisory infrastructure of the electrical grid Physical electrical layer (high-medium voltage)

7. Safeguard agents The Safeguard approach( a Middleware on the top of existing SCADA Systems or just a retrofitted add-on device to the existing SCADA)

8. RETROFITTED ADD-ON SOLUTION RTU Remote Terminal Unit SCADA System Safeguarding SCADA Systems Correlators Actuators Anomaly Detectors Safe Bus API Interface Safe Bus Safe Bus API Interface Safe Bus API Interface RTU Remote Terminal Unit RTU Remote Terminal Unit

9. RETROFITTED ADD-ON SOLUTION RTU Remote Terminal Unit SCADA System Safeguarding SCADA Systems Correlators Actuators Anomaly Detectors Safe Bus API Interface Utilities have significant investment in SCADA equipment. SCADA and similar control equipment are designed to have significant lifetimes. Protection mechanisms should not be developed that require major replacement of existing equipment in the near term. Safe Bus Safe Bus API Interface Safe Bus API Interface RTU Remote Terminal Unit RTU Remote Terminal Unit

10. RETROFITTED ADD-ON SOLUTION RTU Remote Terminal Unit SCADA System Safeguarding SCADA Systems Correlators Actuators Anomaly Detectors Safe Bus API Interface Safe Bus Because of the limited capabilities of the SCADA processors, protection mechanisms should be implemented as a retrofitted add-on device. Protection mechanisms management should be designed to operate in one or more control centers for disaster recovery and distributed management purposes Safe Bus API Interface Safe Bus API Interface RTU Remote Terminal Unit RTU Remote Terminal Unit

11. RETROFITTED ADD-ON SOLUTION SCADA System RTU Remote Terminal Unit Safeguarding SCADA Systems Correlators Actuators Anomaly Detectors Safe Bus API Interface Safe Bus SCADA systems are designed for frequent (near real-time) status updates. Protection mechanisms should not reduce the performance (reading frequency, transmission delay, computation) below an acceptable level. Safe Bus API Interface Safe Bus API Interface RTU Remote Terminal Unit RTU Remote Terminal Unit

12. RETROFITTED ADD-ON SOLUTION RTU Remote Terminal Unit SCADA System Safeguarding SCADA Systems Correlators Actuators Anomaly Detectors Safe Bus API Interface SCADA protection mechanisms should be designed to address all forms of SCADA protection, including: monitoring data transmission, cryptographic functions, state estimation functions, topology estimation, usage and actions taken by operators, etc. Safe Bus Safe Bus API Interface Safe Bus API Interface RTU Remote Terminal Unit RTU Remote Terminal Unit

13. Other LCCIs Foreign Electricity Networks Telecommunication Networks ------------------- Safeguard agent Architecture for Large Complex Critical Infrastructures (LCCIs) High-level agents Negotiation agent MMI agent Correlation agent Action agent Topology agent Low-level agents Diagnosiswrappers Intrusion Detection wrappers Hybrid Anomaly Detection agents Actuators Cyber Layer of Electricity Network Home LCCIs Commands and information Information only SAFEGUARD ARCHITECTURE Network global protection Local nodes protection

14. SAFEGUARD ARCHITECTURE At Level 1 – identify component failure or attack in progress Hybrid anomaly detection agents utilise algorithms specialised in detecting deviations from normality. Signature-based algorithms are used to classify failures based on accumulated functional behaviour. High-level agents Negotiation agent MMI agent Low-level agents Local nodes protection Diagnosiswrappers Intrusion Detection wrappers Hybrid Anomaly Detection agents Cyber Layer of Electricity Network Home LCCIs Commands and information Information only

15. SAFEGUARD ARCHITECTURE Other LCCIs Foreign Electricity Networks Telecommunication Networks ------------------- At level 2: Correlate different kind of information Correlation and Topology agents correlate diagnosis Action agent replaces functions of failed components T High-level agents Correlation agent Action agent Topology agent Low-level agents Local nodes protection Diagnosiswrappers Intrusion Detection wrappers Hybrid Anomaly Detection agents Actuators Cyber Layer of Electricity Network Home LCCIs Commands and information Information only

16. Other LCCIs Foreign Electricity Networks Telecommunication Networks ------------------- Safeguard agent Architecture for Large Complex Critical Infrastructures (LCCIs) High-level agents Negotiation agent MMI agent Correlation agent Action agent Topology agent Low-level agents Diagnosiswrappers Intrusion Detection wrappers Hybrid Anomaly Detection agents Actuators Cyber Layer of Electricity Network Home LCCIs Commands and information Information only SAFEGUARD ARCHITECTURE Network global protection At level 3: operator decision support MMI agent supports the operator in the reconfiguration strategy Negotiation agent supports to negotiate recovery policies with other interdependent LCCIs. Local nodes protection

17. HOW SAFEGUARD MIGHT SUPPORT MANAGING MAJOR SYSTEMS OUTAGE

18. (From UCTE Interim Report) ITALY BLACK-OUT NETWORK STATE OVERVIEW & ROOT CAUSES Pre-incident network in n-1 secure state Island operations fails due to unit tripping Event tree acquired from UTCE report

19. (From UCTE Interim Report) ITALY BLACK-OUT NETWORK STATE OVERVIEW & ROOT CAUSES In SAFEGUARD system Correlator agent intercepts anomalies and failures inside the sequence of events and Action agent try to re-execute the unsuccessful commands. Pre-incident network in n-1 secure state Island operations fails due to unit tripping

20. (From UCTE Interim Report) NETWORK STATE OVERVIEW & ROOT CAUSES Pre-incident network in n-1 secure state Island operations fails due to unit tripping SAFEGUARD might help to recognize the anomaly state and call for adequate countermeasures

21. COORDINATIONS PROBLEMS BETWEEN SYSTEM OPERATORS (From UCTE Interim Report) In this specific case ETRANS needs as corrective measures which are necessary to comply with the N-1 rule, also action to be undertaken in the Italian system. This was confirmed by the check list available to the ETRANS operators, which explicitly mentions that, in case of loss of Mettlen-Lavorgo, the operator should call GRTN, inform GRTN about the loss of the line, request for the pumping to be shut down, generation to be increased in Italy. This clause is mentioned in Italian on the ETRANS checklist for this incident.

22. (From UCTE Interim Report) SAFEGUARD makes available a Negotiation Agent in duty for coordination among different operators In this specific case ETRANS needs as corrective measures which are necessary to comply with the N-1 rule, also action to be undertaken in the Italian system. This was confirmed by the check list available to the ETRANS operators, which explicitly mentions that, in case of loss of Mettlen-Lavorgo, the operator should call GRTN, inform GRTN about the loss of the line, request for the pumping to be shut down, generation to be increased in Italy. This clause is mentioned in Italian on the ETRANS checklist for this incident.

23. US CANADA BLACK-OUT Power System Outage Task Force Interim Report

24. US CANADA BLACK-OUT The “State Estimation” tool, doesn’t work in the regular way because a critical information (a line connection status) is not correctly acquired by the SCADA system. The data utilized by the State Estimator could be corrupted by an attack or by a fault inside SCADA system On August 14 at about 12:15 EDT, MISO’s state estimator produced a solution with a high mismatch (outside the bounds of acceptable error). This was traced to an outage of Cinergy’s Bloomington-Denois Creek 230-kV line—although it was out of service, its status was not updated in MISO’s state estimator.

25. US CANADA BLACK-OUT Task Force Interim Report A SAFEGUARD anomaly detection agent has the duty to verify the correctness level of the data that must be used by the State Estimator. If the State Estimation tool knows what data can be considered “good” or “bad” it has the capability to furnish a more correct state of the network.

26. US CANADA BLACK-OUT 2A) 14:14 EDT: FE alarm and logging software failed. Neither FE’s control room operators nor FE’s IT EMS support personnel were aware of the alarm failure. The Alarm system of FirstEnergy electrical Company doesn’t work correctly and the operators are not aware of this situation

27. US CANADA BLACK-OUT Task Force Interim Report 2A) 14:14 EDT: FE alarm and logging software failed. Neither FE’s control room operators nor FE’s IT EMS support personnel were aware of the alarm failure. Safeguard Correlator agent could detect failures inside Alarm system correlating the sequences of signals flowing from RTUs towards Control Centres.

28. NEW SCADA SYSTEM CONFIGURATION FOR THE ITALIAN ELECTRICAL NETWORK (GRTN-ABB)

29. RTU RTU RTU RTU RTU RTU SIAR-1 SIAR-2 Real Electric Network Simulator e-Agora Simulation to Execute State Estimation SIAR-3 e-Agora Simulation (simulating real electric network) CC Operator Model Control Centres Interface Safeguard Testing Environment Distributed SCADA Emulator Safeguard Agents Remote Data Concentrator Devices Regional Control Center available SCADA events MMI Alarm Panel Low level agents High Level agents Load Charging Scripts SCADA Instrumentation Points available data/signals qualities demand evolution realistic data/signals qualities National Control Center Toward foreign electrical networks ‘pure’ data/signals Data filtering & corruption State estimation results Tele-commands Tele-commands Tele-commands

30. RTU 1 RTU 2 TEST PLATFORM Attacks/faults Console RTU 3 design running log/document RTU n Safeguard high level agents (correlator, action ect.) Hybrid detector for State Estimation (Checking Invariants) Low level agents Event course hybrid detector (Case Base reasoning) Communication hybrid detector (Data Mining technique) RTU state hybrid detector (Neural Network) Testing Environment and Test Platform AIA e-Agora Simulation Data source Message broker Regional Control Centre Network Data Base (On-line mode) Network Data Base (Update mode) National Control Centre SCADA data exchange bus

31. Goal GA0 AND GA1 GA2 GA3 Goal GO0 AND GO1 GO2 GO3 GO0 GA0 GA1 GA2 GO1 GA2 GO3 GO2 TEST PLATFORM: Modeling intrusion and failures by attack/fault trees • Define a “reference language” to model attacks and failures • Utilization of “attack trees” • The root of the tree represents an event that could significantly harm the infrastructure’s mission.The terminal leafs of the tree represent the actions to execute for reaching the high level goals • Every path in the attack tree represents a unique type of attack/fault • every node could be decomposed inside lower level nodes using <AND> and <OR> decomposition types • The attack trees could be visualized also in textual form

32. The tree generate the following two intrusion scenarios <G4, G2, G5, G6> GA0 GO1 G2 GA2 G3 G6 G4 G5 Generate intrusion scenarios • In an attack tree the “terminal leafs” represent the actions needed to execute the attack • An attack tree generates intrusion scenarios, composed bysequences of actions, in such way: <G3, G2, G5, G6>

33. Where 0 < %X < 100 0 < %Y < 100 0 < %Z < 100 • It is a special type of <OR> node where if a sub-goal, as GS2, is reached the Goal GS0 is reached with the %Y of certainty. %Z %X %Y GS0 GS1 GS3 GS2 Insert difficulty degrees • A possible extension of this reference model consists in another type of node (in addition to the OR and AND type). the SCORE type of node:

34. Textual form of the attack tree Goal GA0 Precondition Pstart AND GO1 SCORE (%60)G3 (%40)G4 G2 GA2 AND G5 G6 Post-condition: Presult The attack tree generates the following intrusion scenarios: <G3, G2, G5, G6> with 60% of Presult certainty <G4, G2, G5, G6> with 40% of Presult certainty

35. Attack or failure goal And/or trees Sub-goals or phases Design attacks or faults in form of trees Elementary faulting actions GENERATE ALL POSSIBLE SEQUENCES OF ACTIONS Run scenarios as sequences of malicious actions or faults A1 – A2 – A3 – A4 – A5 – A6 – …. On-line log are necessary becouse attacks or failure conditions could have a long duration (hours/days) TEST PLATFORM Attacks/faults Console design running log/document

36. Attacks/faults Console design attacks or faults in form of tree Generate from a tree all possible scenarios Run a scenario as a timed sequence of malicious actions or faults ATTACK TREES EDITOR AND SCENARIOS RUNNING CONSOLE

37. Attack/fault scenarios for testing Safeguard agents Events corruption story: a sequence of false commands generates the tripping of a critical line. The operators are not able to restore the line connection. Data corruption story: Some measured values and information statuses of the network are corrupted. The State Estimator tool is not able to make a good estimation of the network state. System corruption story: The normal functioning of SCADA system is no more guaranteed, due to malicious task consuming system resources.

38. CONCLUSIONS INCREASING NEED TO TRANSFORM TODAY’S CENTRALISED, DUMB POWER GRID INTO SOMETHING CLOSER TO A SMART, DISTRIBUTED NETWORK “THE ENERGY INTERNET” INCREASING NEED OF INTELLIGENT DATA INTERPRETATION TO CAPTURE NOVELTIES AND PROVIDE OPERATORS WITH EARLY WARNINGS. MULTI-AGENT SYSTEM TECHNOLOGY, COMBINED WITH INTELLIGENT SYSTEMS, CAN BE USED TO AUTOMATE THE FAULT DIAGNOSIS ACTIVITY AND TO SUPPORT OPERATORS IN THE RECOVERY POLICIES. SAFEGUARD MULTI-AGENT SYSTEM TECHNOLOGY CAN WORK IN AN AUTONOMOUS MANNER AS AN ADD-ON SYSTEM, INTERACTING BOTH WITH THEIR ENVIRONMENT AND WITH ONE-OTHER