Cyber-Enabled Smart Distribution Systems and Micro Grids

1. Cyber-Enabled Smart Distribution Systems and Micro Grids Bruce McMillin Department of Computer Science Missouri University of Science and Technology (Formerly the University of Missouri-Rolla) Rolla, MO 65409-0350

2. Introduction: CPS • Cyber Physical Systems (CPSs) are integrations of computation with physical processes. • Distributed Control • Advanced Power Electronics • Finer-grained control over physical entities • Schedulable entities • Design Issues • Complex and unpredictable interactions between the cyber and physical processes • Information flow across the cyber-physical boundaries

3. Schedulable Power Electronics Under Distributed Control

4. Communications Transmission Line Generation Energy Management Communications Satellite Distributed control and fault/attack detection Distributed Decisions FACTS Power Electronics Wind Power 33 FACTS device Sensing and monitoring Inputs Power Electronics Energy Storage Solar Power v v Power Electronics Energy Storage Transmission via Distributed Control Using Power Electronics

5. A Specific Problem • Prevent Cascading failures: • 2003 Blackout • Causes • Physical & Cyber contingencies • Deliberate disruption • Hackers • Terrorist Activity

6. Proposed Solution Flexible AC Transmission Systems (FACTS) • Power Electronic Controllers • Contain embedded computer and networking • Means to modify the power flow through a particular transmission corridor - UPFC • Operate under distributed control Work done with Mariesa Crow at S&T sponsored by NSF & DOE/Sandia: http://filpower.mst.edu

7. How to Start? We need a good formalism to work from. Maximum Flow in a Digraph

8. Power System

9. Power system as a graph Max Flow Predicts Best Power Flow to Set FACTS Devices

10. Max-Flow • Assign an initial flow to all arcs • Mark the source and sink • Search for a node that can be labeled. If none is found, flow is maximum, stop. • Backtrack the path computing the minimum ij used. Go to previous step.

11. 40 100 s t =40 s 15 20 60 s s s s s s s s s s s s s s s d s =15 t t t 50 s s 10 s 100 s s s s s 50 50 60 100 10 25 20 20 20 25 25 0 10 28 50 50 17 30 50 50 50 45 28 60 60 45 28 28 50 50 50 50 50 0 e 28 45 45 28 s c =17 50 28 22 e e 28 28 e e e e e 22 22 28 28 28 28 e 22 e 22 28 50 e e e e 22 22 22 28 0 28 28 28 28 0 50 e 22 22 22 22 22 s e =28 22 50 50 8 22 50 a a a a 15 50 50 50 0 8 8 22 50 50 50 50 22 22 50 8 0 5 5 5 b 15 15 a d 8 8 b 22 8 8 8 15 0 15 8 8 8 s 8 8 8 8 e t =22 b b a a b d d b b b b b b b b b b d d d d d d d 8 b d b a d b b b a a a d d d b b d d b a d b b b b a a a a d d d d 3 8 3 13 b 28 d 20 10 10 3 3 c s d t b 20 20 3 0 3 3 3 3 3 10 10 5 5 5 5 0 5 5 20 20 5 17 10 10 10 10 10 10 0 10 10 10 3 3 10 3 10 3 3 3 3 10 10 10 5 5 5 =3 5 5 5 5 c 10 10 3 10 c 17 17 5 5 10 10 10 10 25 5 10 17 17 17 c c s t =5 d b c c c c c c c c c c c 0 17 17 c c c c c c c c c c c 40 0 c 10 20 30 0 40 10 10 s t b =10 30 30 10 10 10 10 0 13 10 10 10 10 13 13 30 30 30 13 13 13 13 30 13 30 10 10 10 13 s t t =10 c b t t t t t t t t t t t t t t t t t t t t t t t t 10

12. In general, lines are not all maximally loaded. The power flow can then be re-directed to new transmission corridors. • Where re-direct? • How much to re-direct? • How account for KCL? • Control/communication between decision-making devices?

13. s s 50 100 50 100 e 28 22 e 28 22 50 50 15 8 15 X b a d b a d 3 20 10 3 20 18 17 17 c c 40 40 30 30 18 10/20 t Loss of Line B-D • Power will flow over b-c, overloading it • Recalculating Flow over b-t removes overload t

14. G G 1 Outage 37-39a Riversde Pokagon 3 S.Tiffin Howard 117 West End Corey 41 2 HickryCk 40 42 11 NwCarlsl 12 NwLibrty GoshenJt 4 14 39 SouthBnd Concord 15 13 TwinBrch EastLima Haviland FtWayne 33 Area 2 37 Kankakee Olive 5 N. E. 6 16 Lincoln 38 19 7 43 34 Olive McKinley 8 Rockhill 18 JacksnRd S.Kenton 35 17 Sorenson 36 Sterling WLima Sorenson From 65 MuskngumS 30 Bequine 9 Adams 20 113 Grant Deer Crk 31 29 DeerCrk 21 Jay Breed 10 Randolph 28 32 Mullin 22 Delaware

15. G G 1 Outage 37-39b Riversde Pokagon 3 S.Tiffin Howard 117 West End Corey 41 2 HickryCk 40 42 11 NwCarlsl 12 NwLibrty GoshenJt 4 14 39 SouthBnd Concord 15 13 TwinBrch EastLima Haviland FtWayne 33 Area 2 37 Kankakee Olive 5 N. E. 6 16 Lincoln 38 19 7 43 34 Olive McKinley 8 Rockhill 18 JacksnRd S.Kenton 35 17 Sorenson 36 Sterling WLima Sorenson From 65 MuskngumS 30 Bequine 9 Adams 20 113 Grant Deer Crk 31 29 DeerCrk 21 Jay Breed 10 Randolph 28 32 Mullin 22 Delaware

16. G G 1 Outage 37-39c Riversde Pokagon 3 S.Tiffin Howard 117 West End Corey 41 2 HickryCk 40 42 11 NwCarlsl 12 NwLibrty GoshenJt 4 14 39 SouthBnd Concord 15 13 TwinBrch EastLima Haviland FtWayne 33 Area 2 37 Kankakee Olive 5 N. E. 6 16 Lincoln 38 19 7 43 34 Olive McKinley 8 Rockhill 18 JacksnRd S.Kenton 35 17 Sorenson 36 Sterling WLima Sorenson From 65 MuskngumS 30 Bequine 9 Adams 20 113 Grant Deer Crk 31 29 DeerCrk 21 Jay Breed 10 Randolph 28 32 Mullin 22 Delaware

17. Add A FACTS Device • Under Proper Control • Avoids the overload that causes the outage that causes the cascade

18. G G 1 Outage 37-39a Riversde Pokagon 3 S.Tiffin Howard 117 West End Corey 41 2 HickryCk 40 42 11 NwCarlsl 12 NwLibrty GoshenJt 4 14 39 SouthBnd Concord 15 13 TwinBrch EastLima Haviland FtWayne 33 Area 2 37 Kankakee Olive 5 N. E. 6 16 Lincoln 38 19 7 43 34 Olive McKinley 8 Rockhill 18 JacksnRd S.Kenton 35 17 Sorenson 36 Sterling WLima Sorenson From 65 MuskngumS 30 Bequine 9 Adams 20 113 Grant Deer Crk 31 29 DeerCrk 21 Jay Breed 10 Randolph 28 32 Mullin 22 Delaware

19. QuestionWhat does this have to do with Distribution?

20. Future Renewable Electric Energy Delivery and Management(FREEDM) – NSF ERC An efficient and revolutionary power grid Integrating distributed and scalable alternative energy sources and storage with existing power systems

21. Pre-1980s Paradigm Shift Internet Centralized Mainframes Distributed Computing • Shipping 250M pcs/yr. • Ubiquitous ownership • Ubiquitous use • Ubiquitous sharing Innovation & Industry Transformation

22. Paradigm Shift Today FREEDM System Centralized Generation100+ year old technology Distributed Renewable Energy Resources (DRER) New technologies for distributed renewable energy • Ubiquitous sales • Ubiquitous ownership • Ubiquitous use • Ubiquitous sharing Innovation & Industry Transformation New energy companies based on IT and power electronics technologies

23. The FREEDM Concept • Distributed Intelligence • People share energy resources • Neighborhood or industrial level • Where is the centralized controller?

24. Substation Remote Wind Farm Renewable flywheel hydrogen PV array EV PV array Fuel cell car PV array Small turbine Fuel cell car PV array Substation Traditional power grid H 2 The FREEDM System Is Distributed • Distributed Intelligence • Spread over components of a FREEDM node • Components work together to provide a solution • Failure of a single component does not cause system failure • Components are not bound to any specific device or location • Multiple Points of Vulnerability

25. IEM and IFM nodes each run a portion of the DGI to manage their own resources • Coordinate to control the whole as a Distributed Algorithm IEM: Intelligent Energy Management IFM: Intelligent Fault Management DRER: Distributed Renewable Energy Resource DESD: Distributed Energy Storage Device

26. Schedulable Entity The Solid State Transformer

27. Inside an IEM Node • Solid State Transformer (SST) • Power Electronics • Schedulable Entity

28. How to use it?

29. IEM Nodes and Distributed Processes • Each IEM/IFM node contains a Computer running a Process that sends messages to its peers • No other sharing of information

30. Distributed Grid Intelligence • Distributed Long and Short Term Control • Distributed Systems Management • Distributed Leader • State Maintenance • Simulation Architectures • Power Economics Models and Control • Fault Tolerance of Cyber-Physical system • Security – Confidentiality, Integrity, and Availability of Cyber-Physical system • Resilience - Robust Distributed System • Formal Correctness • Usability as an automomous system

31. Distributed Algorithm –Load Balancing • Each IEM node has an aggregate (S)upply or (D)emand • Where to get power from or provide power to? • No centralized picture of the system

32. Distributed Load Balancing • Correctness: Keep all IEM nodes’ “balanced” in terms of Supply and Demand • Pass messages negotiating load changes until the system has stabilized • Global optimization decomposed into individual processes that cooperate to meet the global correctness.

33. I CAN SUPPLY REQUEST SUPPLY Satisfy IEM 1’s Demand IEM n IEM 0 IEM 1 IEM n then sends power and IEM 1 receives it

34. Distributed Leader Election • System management functions, configuration / reconfiguration on-line, automatic restoration, distributed state maintenance such that each IEM node contributes to DGI.  • In a hierarchical control, a Leader Election is necessary to dynamically reconfigure higher layers of control • Dynamic Leader/Coordinator

35. DGI Leader Election • A leader is a distinguished dynamically-elected node that may change during operation

36. DGI Leader Election • A newly elected leader due to failure of old leader

37. Invitation Algorithm Merge() Are you there() Input: Coordinator Node, Coordinating Group , Member Node, Yes or No  Calling node wishes to know if the coordinator it knows is still a coordinator and if so, does it still consider it to be its member • Input: Current Node (inviting coordinator), Coordinator Set • Sends invitation to merge to the coordinators it knows and to its members using invitation() with current node as leader • After a reasonable time, reorganization with the new members of the group is attempted • The new members are designated with a task using ready() • On time out, calls recovery() Invitation() Timeout() • Input: Invited node, Inviting node, group to join • Invited node in Normal State forwards invitation from inviting node to its members • Calls Accept() if interested to join • Input: Current Node • Every member that has not heard from its coordinator checks its status using Are you there() • If it yields a NO, recovery() is called Check() • Input: Current Node • Every coordinator checks for other coordinators by calling Are You Coordinator() • It invites the so-found coordinators for a possible merge of groups using Merge() Accept() Input: Invited node, Inviting node, group to join  Invited node acknowledges the invitation to join the group coordinated by the inviting node Ready() Recovery() Input: New Member Node, Coordinator Node , Group, Task to be assigned Coordinator of the group assigns a task to the new member node of the group to get it start with its membership • Input: Current Node • Put this node in a singleton group with itself as leader • Subsequently, this leader calls for election to merge Are you coordinator() Input: Every Node, Yes or No  Calling node wishes to know if node is a coordinator in normal state

38. 1 2 Election f Election 3 4 Election Election f f Election Coordinator node Member node Group Management and Election

39. Threats to DGI • Hardware Degradation • Maintenance required • Rollback and Recovery • Software Failure • Residual Design Flaws • Rollback and Recover with Alternate Algorithms • Hackers • Teenager in the basement hacking into an IFM • Denial of Service Attack • Information Warfare • Buffer Overflow and Quality of Service (Denial of Service) • Confidentiality of decision making • Integrity attacks • Confidentiality • Information flow • Multi-level security model • Less studied aspect in the cyber-physical world – key problems arise from observation ofphysical interactions

40. Confidentiality of CPS • Modern Infrastructures consist of Cyber and Physical Components • Distributed Energy Resources, Smart Houses, Air Transport, Vehicle Transport, Smart Structures, Oil and Gas Pipelines • All have an inherent commonality – Physical Actions • A Security Leak in a Physical System • Pizzas at the Pentagon

41. Motivation • Observable physical changes in cyber-physical systems divulge security related information • Security Policy defines what level of security

42. What security do you want?

43. Information Security (from NERC CIP Standards)

44. Information Flow Models • FREEDM contains Power Electronics Devices that perform physical actions that are observable • Cannot keep these secret – loss of confidentiality/privacy • Some other models • Non-Interference • High-level events do not interfere with the low level outputs • Non-Inference • Removing high-level events leaves a valid system trace • Non-Deducibility • Low-level observation is compatible with any of the high-level inputs.

45. Threats & Vulnerabilities? • Denial of (information) service • Localized power outages • Privacy • My neighbors can now infer what I’m doing • Gaming the system • Economic Gains • Hacker in the Basement • What fun!

46. Social Aspects • People Must Use This • Bridging the Cyber, Physical, and Social Worlds Workshop – May 27-28, Kansas City • Social Scientists, Engineers, Computer People • Linkages between the worlds • Many “a-ha moments” • Linkage between disciplinary theories • Sociology as a driving force • Enforce correctness, also, through social needs - ethics

47. Futures • Understanding what the CPS is truly an integrated system • Develop widely applicable analysis techniques finding commonality among infrastructures • Theories that can bridge the cyber and physical and social worlds such that information flow and power flow are uniformly understood. • Educational programs that cross train computer scientists with engineered domains and social domains

48. FREEDM DGI Team • The team, Bruce McMillin S&T, Frank Mueller, NCSU, Mariesa Crow, NCSU, Mo-Yuen Chow, NCSU, Chris Zimmer, NCSU, Derek Ditch, S&T, Ravi Akella S&T, Marfield Meng, S&T, Gerald Heydt, ASU, Alex Huang (Director), NCSU