August 14, 2003 Blackout

1. August 14, 2003 Blackout Summary Based on Interim Report of the United States – Canada Power Outage Task Force November 19, 2003

2. U.S.-Canada Interim Report • Released November 19, 2003 • Result of an exhaustive bi-national investigation • Working groups on electric system, nuclear plant performance and security • Hundreds of professionals on investigation teams performed extensive analysis • Interim report produced by the teams and accepted by the bi-national Task Force

3. Overview • Overview of power system and reliability • Pre-outage conditions on August 14 • Trigger events and start of cascade • Wide area cascade • Root causes • Next steps

4. Power System Overview

5. Reliability Overview • Balance generation and demand • Balance reactive power supply and demand • Monitor flows and observe thermal limits • Observe power and voltage stability limits • Operate for unplanned contingencies • Plan, design and maintain a reliable system • Prepare for emergencies Reliably operate the system you have!

6. 3 Interconnections / 10 NERC Regions

7. NERC Control Areas

8. NERC Reliability Coordinators

9. Footprints of Reliability Coordinators in Midwest

10. NERC Immediate Response to Blackout • First hours • Worked closely with NERC Reliability Coordinators • Identified what had tripped and extent of outage • Assessed restoration efforts • Maintained open line with DOE/FERC • Communicated with DHS, White House, and NRC • First days • Assigned project manager • Established Steering Group with industry executive experts • Began organizing investigation teams • 90+ volunteers + entire NERC staff

11. Investigation Organization Overview Steering Group U.S – Canada Task Force Investigation Team Lead Root Cause Analysis Cooper Systems Investigation Process Review Vegetation/ROW Management Project Planning and Support Transmission System Performance, Protection, Control Maintenance & Damage NERC & Regional Standards/Procedures & Compliance Sequence of Events MAAC/ECAR/NPCC Coordinating Group MAAC Operations - Tools, SCADA/EMS Communications Op Planning Generator Performance, Protection, Controls Maintenance & Damage Restoration ECAR Frequency/ACE Data Requests and Management NPCC System Modeling and Simulation Analysis System Planning, Design, & Studies MEN Study Group

12. Data Gathering and Analysis • Three fact-finding meetings • August 22 • September 8-9 • October 1-3 • Onsite interviews and inspections • Secure database of outage information • Extensive corroboration of data to determine facts • Analysis by teams of technical experts

13. Root Cause Analysis • Logical structure for investigating complex problems • Identifies changes, conditions, actions, or inactions at each causal step • Starts with final event and drills back through each branch of causal tree. • Asks “why?” at each step. • Accurate, reliable, defensible understanding of the root causes. Successfully used to investigate root causes of PJM voltage stability condition in July 1999 and established history in nuclear and defense industries.

14. 16:15 16:06 15:05 Root Cause Analysis Phases BLACKOUT Sammis – Star Star – South Canton Hanna – Juniper Chamberlin - Harding Initial Focus Pre-Existing Conditions E.g. voltages, wide- area transfers, line and generator outages, etc.

15. August 14 Conditions Prior to Blackout • Planned outages • Cook 2, Davis Besse nuclear plants • East Lake 4, and Monroe 1 • Transfers high to northeast U.S. + Ontario • Not unusually so and not above transfer limits • Critical voltage day • Voltages within limits • Operators taking action to boost voltages • Frequency • Typical for a summer day • System was within limits prior to 15:05, on both actual and contingency basis

16. Warm But Not Unusual for August

17. August 14 Imports to Northeast-Central Compared to 6/1 to 8/13/2003

18. Voltages Prior to 15:05 EDT August 14

19. Frequency Typical for Summer Day

20. Blackout was NOT Caused by • Heavy wide-area transfers • Low voltages, voltage collapse • Lack of IPP voltage/reactive support • Frequency anomalies • Cinergy outages starting at 12:08 • East Lake 5 trip at 13:31 • Contributing factor to later events, but not by itself causal to the blackout • DPL Stuart-Atlanta trip at 14:02 • Contributing factor to loss of MISO real-time monitoring, but not electrically significant

21. Outage Sequence of EventsTransmission Map Key

22. ONTARIO ONTARIO East Lake 5 Trip: 1:31:34 PM 2 1

23. East Lake 5 Exciter Failure Causes Trip

24. Stuart Atlanta Trip: 2:02 PM

25. MISO State Estimator and Reliability Analysis • MISO state estimator and contingency analysis ineffective from 12:37 to 16:04 • State estimator not solving due to missing information on lines out in Cinergy then DPL • Human error in not resetting SE automatic trigger • Using Flowgate Monitoring tool to monitor conditions on previously identified critical flowgates

26. FirstEnergy Computer Failures • 14:14 Alarm logger fails and operators are not aware • No further alarms to FE operators • 14:20 Several remote consoles fail • 14:41 EMS server hosting alarm processor and other functions fails to backup • 14:54 Backup server fails • EMS continues to function but with very degraded performance (59 second refresh) • FE system data passed normally to others: MISO and AEP • AGC function degraded and strip charts flat-lined • 15:08 IT warm reboot of EMS appears to work but alarm process not tested and still in failed condition • No contingency analysis of events during the day including loss of East Lake 5 and subsequent line trips

27. Phone Calls to FirstEnergy • FE received calls from MISO, AEP, and PJM indicating problems on the FE system but did not recognize evolving emergency • 14:32 AEP calls regarding trip and reclose of Star-S. Canton • 15:19 AEP calls again confirming Star-S. Canton trip and reclose • 15:35 Calls received about “spikes” seen on system • 15:36 MISO calls FE regarding contingency overload on Star-Juniper for loss of Hanna-Juniper • 15:45 FE tree trimming crew calls in regarding Hanna-Juniper flashover to a tree • PJM called MISO at 15:48 and FE at 15:56 regarding overloads on FE system

28. Chamberlin-Harding (3:05:41)

29. Chamberlin-Harding Indication of Ground Fault Due to Tree Contact as Measured by DFR at Juniper

30. Hanna-Juniper (3:32:03) (3:05:41)

31. Hanna Juniper Confirmed as Tree Contact atLess than Emergency Ratings of Line

32. Effects of Ambient Conditions on Ratings

33. (3:05:41) (3:32:03) Star- S. Canton (3:41:35)

34. ONTARIO Situation after Initial Trips 3:05:41 – 3:41:35

35. Canton Central – Tidd (3:45:41)

36. 138 kV Lines Overload and Cascade Near Akron

37. 138 kV Cascade Contributes Furtherto Overload of Sammis-Star 15:51:41 EDT 15:05:41 EDT 16:05:55 EDT 15:32:03 EDT 15:41:35 EDT

38. Sammis-Star (4:05:57.5)

39. Sammis-Star Zone 3 Relay Operateson Steady State Overload

40. Actual Loading on Critical Lines

41. Actual Voltages Leading to Sammis-Star

42. Remaining Paths Major Path to Cleveland Blocked after Loss of Sammis-Star 4:05:57.5 PM

43. ONTARIO 345 kV Lines Trip Across Ohio to West

44. ONTARIO Generation Trips 4:09:08 – 4:10:27 PM

45. 345 kV Transmission Cascade Moves North into Michigan 4:10:36 – 4:10:37 PM

46. Northern Ohio and Eastern Michigan Served Only from Ontario after 4:10:37.5 – 4:10:38.6 PM

47. Power Transfers Shift at 4:10:38.6 PM

48. Eastern Eastern Michigan (Detroit) UnstableVoltage and Frequency Collapse and Pole Slipping Ontario – Michigan Interface Flow and Voltages Beginning 16:10:38

49. Generator Trips to 16:10:38

50. Generator Trips – Next 7 Seconds