Monitoring and Controlling the Smart Electric Power Grid with Isis2.

1. Monitoring and Controlling the Smart Electric Power Grid with Isis2. Ken Birman

2. The power grid is getting old… • Today’s electric power grid is a legacy of a long evolution that has many structural vestiges of the monopoly period • As a result much power is wasted • With new “green” power sources, some utilities literally purchase power and throw it away • A great deal of what we generate is lost in transmission • And much of what gets delivered ends up being used to light empty rooms, keep water hot when nobody is home to use the shower, or gets turned into heat by computers that has to be removed using air conditioning • Statistics • 66% of the energy produced for electricity is lost, 10% of that in transmission. • 71% of the energy produced for transportation is wasted. • 20% of the energy produced to run American industry is lost, and • 20% of the energy that we use in our commercial and residential buildings is wasted.

3. Smart-Grid Vision • Within the home, office or building: a smart-meter dispatches power-hungry tasks at times convenient to the grid • Home A/C, freezer • Hot water heater, dish or clothes washer • Recharging an electric vehicle • The meter knows a lot about you and privacy is a concern here. “Privacy preservation” highly desired

4. The smart grid would also control: • High-power inverters for DC / AC linkages • High-tension long-haul interconnects for sharing excess power needed to maintain load balance • Power storage units (pumped water, inertial storage systems, even batteries) • Microgenerators that phase in when demand is high and prices justify doing so, out when not in use • Grid use of excess power from home solar arrays

5. A little bit of history • Prior to the 1960’s, most power in the United States was controlled by regional monopolies • They had sole control over the power generation, delivery and billing structure • There was no competition, so prices were regulated • Owning a utility was a guarantee of hefty revenue • But the lack of competition also meant that there was little incentive for innovation • Similar stories in many countries worldwide

6. Restructuring: The US response • A process of restructuring has slowly changed this: • Big regions broken into a collection of power producing companies, “independent service operator” (ISO), consumer-delivery companies • Emergence of 24-hour ahead of time power exchange markets where larger “parcels” of power can be bought and sold, permitting 24-hour lead time planning to ensure that the grid will meet its needs • ISO helps ensure competitiveness and stability • Increased use of long-distance high-tension lines to share “reactive power” between regions

7. How a small power grid operates • Power flows “like water” • Path of least resistance • Governed by Kirchoff’s Law • Power enters at every generator,exitsat every load • Hierarchical structure: • Primary “power busses” • Secondary smaller local feeds 10-Generator, 39-bus New England System

8. Operating a restructured grid • Many entities, each with very limited visibility into the state of their neigbors • Two issues: limited sharing of topological data and limited sharing of real-time status, driven by competitive concerns, security worries, “evolution” of the system topology as lines break, are fixed, are changed • In practice, you “know your own grid” and know (a little) about tie-lines to your neighbors • Various ad-hoc sharing and collaboration solutions are in place.... long list of blackouts in past decade testify to the issues with this approach!

9. State estimation • This is the problem of collecting measurements of the power system • Then fitting the measurements to a model generated from the systems bus architecture • Like a least-squares optimization… complicated by: • Transients as devices come on/go offline • Imprecision of our models of the physical grid itself: The bus architecture is really a simplification of the actual system and fails to capture many aspects

10. Ways of doing state estimation • Historical: SCADA • Track the line voltage and frequency • If frequency drifts, adjust generators • “State estimation” by fitting power equation to observed data, a computationally costly task • New: “synchrophasor measurement unit” or PMU • Directly measures the phase angle of a power bus • Enables fast, hierarchical state estimation... but only if the current topology is accurately known

11. Adoption Limitations • Renewables very hard to “plan” for, hence often offer power at times when it isn’t convenient and can sag suddenly (visualize: cloud over solar field) • Poor past experiences with machine-control solutions in the physical loop • In a restructured grid, visibility into neighboring regions is of limited quality, topology data may be unavailable • Commercial market place dominated by six big players, and they sell all the products. Extremely proprietary

12. Power grid under attack? • Richard Clarke has argued (in his book CyberWar) that we are at grave risk of attack on the grid • He believes that the grid is already so connected to the Internet that it can be disrupted easily • Argues that some forms of attack would destroy huge amounts of hard to replace infrastructure: “logic bombs” • And he suggests that many countries may actually have prepositioned these logic bombs for use during times of political stress or war

13. GridCloud: Scalable, secure monitoring

14. As deployed: Mutually-distrusting domains • Distinct owners: Peersplus hierarchy (ISO) • Owner controls data flow:distinct security policies • ISO integrates data...but as we get furtherfrom sources, quality ofinformation is apotential concern

15. ... and we’re using Isis2 to build it! • Our challenge: • Ken and his students understand the Isis2 library • But the people who create the smart applications for the smart grid are not likely to be Isis2 programmers • Can we leverage Isis2 without it being too visible?

16. Initial insights • We want to run at cloud scale, but with an emphasis on accurate real-time data • Recall the Facebook challenge of computing with extensive non-coherent caching • We can only use that technique for data that changes very rarely, and in GridCloud there would not be a great deal of such data • Thus we will need to use coherent replication very heavily

17. Initial Insights • Most GridCloud users won’t be “programmers” • They won’t build new programs, but may run existing ones that evolved over decades in other settings • They will want to important those solutions and run them in a cloud but with minimal changes • So GridCloud needs to leverage Isis2 yet the GridCloud user probably won’t be a person who could learn to use Isis2 directly

18. Key design features • Redundancy to overcome network or cloud scheduling delays, faults • Open-flow network routing scheme integrated with a new security architecture we’ve developed • On cloud, can host various applications: some ignore sharded data format but others could leverage it • We use Isis2 to manage and control the system • Free open source group communication system • Includes a new scalable distributed “data analysis” tool

19. What will GridCloud applications do? • Consume monitoring data, output “advice” (later, perhaps do direct control for some technologies) • These applications fall into categories • Use of machine-learning to develop optimized plans for dispatch of power • Tools for helping operators manage and control their smart grid networks • Contingency reaction solutions for dealing with various kinds of environmental (or other) disruptions • “What-if?” technologies to assist the owner in evolving their system with new technology capabilities

20. Dangers of Inconsistency • Inconsistency causes bugs • Cloud control system speaks with “two voices” • In physical infrastructure settings, consequences can be very costly “Canadian 50KV bus going offline” Bang! “Switch on the 50KV Canadian bus”

21. GridCloud Consistency: Based on Isis2 A=A+1 A=3 B=7 B = B-A • Virtually synchronous runs are indistinguishable from synchronous runs • We are using Isis2 to create a control and management system for GridCloud Non-replicated reference execution Synchronous execution Virtually synchronous execution

22. Security • Mutually suspicious organizations that nonetheless must collaborate safely while repelling attackers • Human issue: design acceptable information sharing policies at a relatively “fine grained” level • API issue: Need suitable ways to express these policies so that operators will understand them • Technical issue: GridCloud managed network would need to correctly implement the desired behaviors • Given complexity of the setting, these goals represent significant challenges

23. Red Sky • First part of our solution, being developed by PhD students Zhiyuan Teo and Erluo Li • Special secure hardware endpoints for sensors • Redundant network routing over Open Flow switches • Security policy controls connections • Data arrives in Cloud on sharded data collectors • Special “TCP Reliability” (TCPR) layer allows us to shift endpoints from collector to collector in a seamless way • Isis2 used within Red Sky but invisble to the user!

24. DMake • Created by PhD student Theo Gkountouvas • Role is to monitor and manage cloud application • Each application is provided with configuration data (runtime command line arguments plus XML files with parameter information) • As it runs, each creates XML files of status, which DMake collects • DMake then runs an optimization module that updates the configuration data, which is pushed back to the applications

25. Why do we call it DMake? • On Linux, the Make program and Makefile format is familiar and very popular • DMake uses the exact same format for its control files! • New functionality is to collect information in a secure, consistent and fault-tolerant way • Optimization component is a “plug in” program provided by the GridCloud user • Thus DMake uses Isis2 and the GridCloud user benefits without seeing Isis2 directly!

26. Experiments with our prototype • 2 Area System Simulation – Standard IEEE test system • Two Induced Contingencies • @ 1min mark – line 7-8 disconnected • Reconnected after 5 seconds • @ 5min mark – lines 7-8 & 8-9 disconnected • Not reconnected

27. Early experience with GridCloud • Starting point: An existing state estimation system WSU Code Cornell Code WSU Code

28. Modified version leverages redundancy

29. Latency Distribution Graphs: Number of times a particular latency occurs

30. Jitter Graphs: Jitter of previous latency graphs

31. Replica Latency Reduction Graphs: Latency – Length of time until all necessary data is available

32. Improvement Summary • Vastly reduced latency and jitter • All replica latency reduced from typical [235 , 350] to [215 , 250] • All replica jitter reduced from typical [-10,10] to [-5 , 5] • Consistency from replica to replica has improved • Consistency between “all replicas” to a particular replica has improved • Consistency over time for each run has improved

33. Today: Latency Distribution Graphs: Number of times a particular latency occurs

34. Today: Jitter Graphs: Jitter of previous latency graphs

35. Case Study: Failure of 1/6th of Collector Nodes • We fail Replica1-Node2 of the collectors for 1 minute • This results in 62 out of 294 PMU streams being unavailable for 1 minute

36. Failure Case: Replica Latency Reduction Graphs: Latency – Length of time until all necessary data is available Data Loss

37. Failure Case: Latency Distribution Graphs: Number of times a particular latency occurs

38. Failure Case: Jitter Graphs: Jitter of previous latency graphs

39. A long road ahead... • Future monitoring need could involve tens or hundreds of thousands of PMUs operated by multiple mutually distrusting organizations • Security requirements will force us into a world with multiple side-by-side “micro-clouds” that share data only in controlled ways • Any serious system will host many “smart” applications and need to automate dispatch, scheduling, configuration management

40. Building Smart Grid Applications • These need to be machine learning algorithms • For example: in these weather conditions, demand for electric power tends to follow such-and-such a curve • With this knowledge, you can predict demand... and plan accordingly • If demand will exceed supply, you can learn what steps would lower demand, and by how much • So the typical application is a machine learning application that performs well (in real-time) over the Ida infrastructure where the data is living

41. “Killer apps” • Use grid state to • Predict load • Make decisions about what to repair first after a storm • Explore options if something unexpectedly fails and the grid must be operated in a damaged mode • Decide what additional power lines would have the best impact on power pricing • Use electric power from a wide region as “assurance” to permit better use of renewable energy sources • Identify malfunctioning components or those close to failure

42. Some Killer Apps slides Created by Professor Dave Bakken Expert on Electric Power Grids and Smart Grid

43. #1: Decision Support • Operators must constantly make decisions • Conditions change as lines come up or go down, power demand rises or falls, generators go online or offline… • Emerging Smart Grid is more and more complex: best decision is not evident and impact unclear • Smart tools can really help with planning • Acknowledgments: Professor Anjan Bose

44. #2: Integration of Renewables • More and more homes have micro-generators • But to use this green, renewable power the grid needs to be able to “plan” and balance loads • Even a cloud can have a big impact on use patterns • GridCloud commissions thousands of nodes analyzing candidate control steps in parallel • Best approach (actionable steps) is given to operators • Acknowledgements: Prof. Anjan Bose (WSU)

45. #3: Mitigation Control • Rare combination of events do happen • Have lead to many blackouts when not mitigated! • E.g., N-3 contingency (3 failures) never planned for • Infrequent but hugely expensive to analyze • GridCloud commissions thousands of nodes analyzing candidate mitigation steps in parallel • Best approach (actionable steps) is given to operators • Acknowledgements: Prof. Mani Venkatasubramanian(WSU)

46. #2: Oscillation Alarm Processing • Grids oscillate between regions • Negatively damping can lead to blackout • E.g., Oregon/California in July 1996: 0.3 Hz (!!) • GridCloud commissions massive parallel computations exploring huge permutation space • Looking for trends and correlations of alarm data • Also huge number of model-based simluations too • Finds root cause much faster than possible today in much broader set of conditions • Acknowledgements: Prof. Mani Venkatasubramanian(WSU)

47. #3: Post-Tripping Fault Diagnosis • Protection scheme trips a relay, but why? • Underlying cause must be ascertained post facto • GridCloud commissions massive computations to identify the fault(s) that provoked the trip(s) • Many different kinds of fault diagnosis algorithms, all could be run in parallel • Possible integration candidate: openFLE (fault location engine) from Grid Protection Alliance • Acknowledgements: Prof. AnuraugSrivastava (WSU)

48. #4: Multi-Resolution Frequency Disturbance Visualization • Grid operates in very narrow range unless stressed • Frequency excursions outside this give clues to problems • Frequency disturbance recorder (FDR): new device recording frequency disturbances at high rates • E.g., internal sampling of FNET device (in our lab): 1440 Hz • GridCloud commissions thousands of parallel frequency rendering computations • Provide operators a rich suite of visualizations with which to better understand nature and cause of present excursion • Acknowledgements: Prof. Yilu Liu (University of Tennessee, Knoxville)

49. #5: Multi-Dimensional Computations over Both Space and Time • Two existing GridSim apps can be combined in rich ways possible only with cloud computing • Hierarchical linear state estimation: rich coverage of (geographical) space • At one snapshot in time • Obvious extensions over more space with more PMUs • Oscillation monitoring • Uses moving window of time (a few seconds typically) • Over streaming data • Produces a single number: damping factor • Obvious parallel computations over different sets of data with different time windows and algorithms

50. #5: Multi-Dimensional Computations over Both Space and Time (cont.) • Combination: provide rich set of two-dimensional (space, time) data to any desired location • Enables extremely powerful new families of applications operating coherently over both space and time • At each location: different time windows, different algorithms, different sets of data • If available, people would inevitably think of many uses for this data • Acknowledgements: Prof. Anjan Bose (WSU)