1 / 24

OMSAP Presentation April 2004 Power Outage Events

OMSAP Presentation April 2004 Power Outage Events. David F. Duest Massachusetts Water Resources Authority Deer Island Treatment Plant Sept 2004. Agenda. Introduction DITP Power outage events – What happened? - April 3, 2004 - April 11, 2004 Impacts Corrective Actions.

Michelle
Download Presentation

OMSAP Presentation April 2004 Power Outage Events

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. OMSAP PresentationApril 2004 Power Outage Events David F. Duest Massachusetts Water Resources AuthorityDeer Island Treatment Plant Sept 2004

  2. Agenda • Introduction • DITP Power outage events – What happened? - April 3, 2004 - April 11, 2004 • Impacts • Corrective Actions

  3. Deer Island Wastewater Treatment Plant • $3.8 Billion Construction Project • 2nd Largest in US • Treatment Capacity: • Maximum • 1.27 Billion Gal/Daycombined sewer system • 810 MGD by Secondary • Average Daily Flow: • 360 Million Gal/Day • Built on 200 Acres • includes 60 acre public access area • Removes 250 TPD of solids with 93 TPD for beneficial reuse • Consumes 18.5 MW daily with Peaks to 40 MW

  4. NStar Building On Deer Island NStar (South Boston) K-Street Substation 115,000 Volts Cross-Harbor Cable Main Breaker 13,800 Volts Main Breaker MSB A-Bus B-Bus CTG A CTG B STG Deer Island Power Distribution Thermal Power Plant Support Facilities North Pumping South Pumping Primary Treatment A & B Primary Treatment C & D Secondary A & B Secondary C Cryogenic Oxygen Facility Residuals Treatment

  5. Deer Island Thermal Power Plantincluding NStar Building, Main Switchgear Building, NMPS & WT Headworks

  6. What Happened – April 2004 • Event 1: April 3, 2004 07:54 (Saturday) • NStar fault causes loss of “B-Bus” • Most significant DITP effect: • loss of North Pumping (NMPS) for approximately 2 hours 20 minutes • Event 2: April 11, 2004 19:23 (Easter Sunday) • NStar line-to-line fault at K-Street Substation causes complete power loss • Most significant DITP effects: - loss of all systems on DITP for approximately 2 hours 50 minutes

  7. Event 1: April 3, 2004 07:54 • Event – NStar fault causes loss of “B-Bus” • Leak in roof at NStar building caused the B-Bus circuit breaker to trip open. • DITP Effect – • lost 50% of most systems on DITP • Partial impacts on Pumping, Primary & Secondary • No impacts to disinfection/dechlorination • Lost 100% of pumping operations at NMPS

  8. April 3 Power Outage NStar Building On Deer Island NStar (South Boston) K-Street Substation 115,000 Volts Main Breaker 13,800 Volts Main Breaker MSB A-Bus B-Bus CTG A CTG B STG Deer Island Power Distribution Thermal Power Plant Support Facilities North Pumping South Pumping Primary Treatment A & B Primary Treatment C & D Secondary A & B Secondary C Cryogenic Oxygen Facility Residuals Treatment

  9. April 3 Power Outage NStar Building On Deer Island NStar (South Boston) K-Street Substation 115,000 Volts Main Breaker 13,800 Volts Main Breaker MSB A-Bus B-Bus Correction Action - Cross-tie A-Bus into B-Bus - Power restored to B-Bus within minutes CTG A CTG B STG Deer Island Power Distribution Thermal Power Plant Support Facilities North Pumping South Pumping Primary Treatment A & B Primary Treatment C & D Secondary A & B Secondary C Cryogenic Oxygen Facility Residuals Treatment

  10. Event 1: April 3, 2004 07:54 • B-Bus breakers reset at all pump stations • Pumps re-started at South System Pump Station, NMPS & Winthrop Headworks • North Main Pump Station – Pumps still unavailable • Staff reset breakers in pump station • Started Pumps / received pump fail alarms • Staff investigated alarms on lube oil systems • ID’d double failure in UPS at “Force Main Control Panel” (FMCP) • FMCP bypassed manually to re-energize VFD’s • Pumps started (phased startup) at 10:15 • Station fully operational by 12:15 • NStar switch replaced. B-Bus restored onto NStar at 20:00

  11. North Main Pump Station VFD Typical Permissives VFD – bearing/winding temp, overload, vibration, FMCP Pumps – Flow path, seal water, bearing temp, vibration, lube oil Pump Shaft – bearing temp, vibration, lube oil B-Bus A-Bus VFD Motor Control Center (MCC) VFD VFD VFD Bypass VFD VFD Force Main Control Panel UPS VFD VFD Batteries VFD VFD = Variable Frequency Drive (controls speed of pumps through variable power input)

  12. Event 1: April 3, 2004 07:54 • Overall Effect: • Collection system decisions made to PREVENT: • Flooding • Loss of life / injury • Property damage • Infrastructure damage • Remote Headworks isolated • Staff deployed to monitor levels in collection system • Limited storage in collection system (due to rain event) • Cottage Farm CSO facility re-activated. ~22 Mgal to Charles River • Other storm sewer overflow sites activated • All flow sent through DITP received full treatment

  13. Event 2: April 11, 2004 19:23 • Event – NStar line-to-line fault at K-Street Substation • NStar power unavailable to DITP until after 22:15 pm • 1st event of this type in the history of DITP • DITP Effect – • loss of all systems on DITP for approximately 2 hours 50 minutes

  14. April 11 Power Outage NStar Building On Deer Island NStar (South Boston) K-Street Substation 115,000 Volts Main Breaker 13,800 Volts Main Breaker MSB A-Bus B-Bus CTG A CTG B STG Deer Island Power Distribution Thermal Power Plant Support Facilities North Pumping South Pumping Primary Treatment A & B Primary Treatment C & D Secondary A & B Secondary C Cryogenic Oxygen Facility Residuals Treatment

  15. April 11 Power Outage NStar Building On Deer Island NStar (South Boston) K-Street Substation 115,000 Volts Main Breaker 13,800 Volts Main Breaker MSB A-Bus B-Bus • Correction Action • - Start CTG • DC power supply failure kept CTG from powering up DITP grid • Swithgear breakers needed to be closed manually (very hazardous) • Power restored to A-Bus ~22:00 • 1st Pumps started 22:15 CTG A CTG B STG Deer Island Power Distribution Thermal Power Plant Support Facilities North Pumping South Pumping Primary Treatment A & B Primary Treatment C & D Secondary A & B Secondary C Cryogenic Oxygen Facility Residuals Treatment

  16. Event 2: April 11, 2004 19:23 • On CTG power from 22:15 until midnight • Midnight – while switching power back to NStar CTG tripped off-line. • Plant power / pumping restored (with phased startup) on NStar at 00:12 • All systems reset / plant operating normally as of 01:30

  17. Event 2: April 11, 2004 19:23 • Overall Effect: • All Headworks isolated • Staff deployed to monitor levels in collection system • Wastewater diverted to prevent flooding/backups (~3hrs) • North flows - Cottage Farm CSO (4.7 Mgal) to Charles River • South flows – NI Emergency outfalls opened – 17 Mgal raw sewage into Quincy Bay • All flow sent through DITP received full treatment

  18. Station number Location Description Enterococcus/100ml E. coli/100 ml 4/13/04 4/13/04 4/12/04 4/12/04 10 <5 117 Mid Hingham Bay 90 40 20 <5 080 Quincy Yacht Club 210 110 5 <5 079 Outfall 103 150 105 5 <5 082 Outfall 101 95 35 45 45 081 Outfall 102 125 55 Water Quality Monitoring in Quincy and Hingham Bays after NI Emergency Discharge on April 11, 2004 Enterococcus limits (for swimming, marine water): geometric mean 35, single sample maximum of 104E. coli limits (for swimming, freshwater): geometric mean of 126, single sample maximum of 235

  19. Date Time Station Location Enterococcus/ 100 ml E. coli/100ml 04/12/04 11:53 005 Magazine Beach, upstream of Cottage Farm 70 110 04/12/04 11:46 006 BU Bridge, downstream of Cottage Farm 50 60 04/12/04 11:31 007 MIT boathouse 100 120 04/12/04 11:25 008 Harvard Bridge 20 40 04/12/04 11:16 009 Between Harvard and Longfellow 20 20 04/12/04 11:08 010 Downstream of Longfellow 10 50 04/12/04 10:56 011 Downstream of Science Museum 10 50 04/12/04 11:37 145 Stony Brook outlet 3700 2800 Water Quality Monitoring in Charles River after Cottage Farm Emergency Discharge on April 11, 2004 Enterococcus limits (for swimming, freshwater): geometric mean of 33, single sample maximum of 62E. coli limits (for swimming, freshwater): geometric mean of 126, single sample maximum of 235

  20. Corrective Actions • MWRA retained two electrical engineering consulting firms to audit and review the incidents ($100K) • Retained an electrical testing firm to test all battery backup systems on DITP • Staff performed a thorough investigation of all systems involved in the events, independent of the consulting firms • Repairs made to major equipment that either failed during the events or were damaged as a result of the failure ($387K) • UPS at FMCP (secondary failure for April 3rd event) • DC Power system in Thermal Power Plant (secondary failure for April 11 event) • Facility cleanup ($57K) • NStar replaced roof (root cause of April 3rd event) on their building • Changed maintenance procedures for testing battery backup systems on DITP (UPS units) to identify issues before they become problems

  21. Corrective Actions (continued) • MWRA has performed a follow-up black-start test for the thermal power plant systems (on August 10, 2004.) This test did not include any of the treatment systems and the thermal power plant was electrically isolated from the treatment plant systems and NStar. The new DC control system operated as designed. • Staff performed a detailed analysis of all potential Pump Station single-point of failure mechanisms that may occur. • Continue to work on implementing recommendations by the consulting firms. 18 out of 61 are fully implemented (as of 8/13). • Continue to work with DEP/EPA on all follow up actions • …more…

More Related