280 likes | 406 Views
April 17, 2012|Westborough. Gary W. Pire. Manager, Corporate Planning. NEPOOL Reliability Committee Meeting. ISO New England Back-up Control Center Update. Contents . Background and Overview Detail on Reliability Issues Detail on BCC Options Project Schedule, Budget, and Status
E N D
April 17, 2012|Westborough Gary W. Pire Manager, Corporate Planning NEPOOL Reliability Committee Meeting ISO New England Back-up Control Center Update
Contents Background and Overview Detail on Reliability Issues Detail on BCC Options Project Schedule, Budget, and Status Appendix: Backup Facilities at Other ISOs/RTOs
BCC Facts and Figures • The current Backup Control Center (“BCC”) was upgraded during 2001 and 2004 • The BCC is 40 miles away from the Master Control Center (“MCC”) in Newington, Ct. (south of Hartford) • The current BCC site was selected because proximity to CONVEX reduced the risk of failover from the MCC to the BCC for System Operations • However, CONVEX moved to a new facility in 2011 and is no longer located in the same building as the BCC • The ISO discussed expanding the BCC with NU, but expansion at the current location is not possible • The original 2001 design parameters called for a maximum emergency occupancy of 50
Overview of Concerns Concerns, as confirmed by third-party review: • Ability to ensure business continuity • Market products have expanded and the required IT hardware support infrastructure has more than quadrupled over the last ten years • The current BCC was not designed to backup the complete operating needs of the current organization • The BCC data center and HVAC infrastructure is stressed and can’t be expanded • Ability to comply with new NERC reliability standard related to BCCs • Standard requires maximum two-hour timeframe to transition to BCC • 40-mile distance between MCC and BCC leaves little room for contingencies • Standard requires full BCCs that are able to operate “for a prolonged period” • Current BCC would be stretched to provide full services more than briefly • Out of synch with other ISOs/RTOs (see appendix) On the other hand, The underlying risk that the BCC was designed to mitigate is a LOW probability / HIGH consequence event, and the solution should be appropriately balanced
ISO-NE Business Continuity Plan • To ensure continuous reliable operations – including markets and settlements -- ISO-NE developed its Business Continuity Plan (“BCP”) • ISO-NE clears in excess of $9 billion each year in its role as market administrator • Risk that lack of market-established prices lead to imposition of administrative prices and disruptions that could affect up to $24M/day (based on today’s market values) • Additionally, delays in settlements would increase financial assurance requirements • Per the BCP, approximately 150 ISO-NE staff are initially required to restore critical functions (Operations, Market Operations, Settlements, Billing, and Accounting) • BCP now assumes that ISO-NE would find space for a prolonged absence from the MCC, or would rely on employees using virtual access • There is a risk that the markets could not be operated and settlements would be suspended • ISO-NE’s BCP drives BCC resource requirements
NERC Reliability Standards NERC Emergency Operating Procedure (“EOP”)-008-1, effective July 1, 2013, includes specific back-up control center requirements. It: (1) Delineates what must be included in a plan for back-up functionality; (2) Includes a provision for managing the risk to the Bulk-Power System during the transition from primary to back-up functionality; (3) Requires reliability coordinators to have dedicated full back-up control centers; (4) Provides that transmission operators and balancing authorities have either a dedicated facility or a contract for services to provide back-up functionality; (5) Requires formal review and approval of the plan for back-up functionality; (6) Mandates physical independence of the primary and back-up capabilities; (7) Requires testing of the plan; and (8) establishes a procedure for creating a plan to re-establish back-up capability following a catastrophic situation
NERC Reliability Standards - cont’d • In addition to the functional requirements, the revised standard also establishes a transition recovery period between the loss of the primary control center and the time to fully implement the back-up control center that is less than or equal to two hours • In its related Order (Docket No. RD11-4-000 at 14), FERC signaled its concern that two hours is too long and that “it is imperative that full backup functionality occur as soon as possible after the loss of primary control functionality” • The FERC also notes that “…. it may revisit this transition timeframe • Therefore, the two hour window should be viewed as the maximum time allowed, that may well be reduced in the future • NERC and FERC also require the BCC to be “capable of operating for a prolonged period and providing functionality sufficient to maintain compliance with all Reliability Standards that depend on primary control functionality”
ISO-NE BCC Study • In 2011, ISO-NE began a study to define the BCC configuration • System Operations, Market Operations, and IT requirements were gathered via detailed interviews with all levels of management • Results indicate that, in order to operate from the BCC for a prolonged period, the facility must: • Have a control room as similar to the MCC as possible in order to reduce the potential for error when switching facilities – configuring the BCC as the control room simulator was seen as the best way to do that • Have a complete data center capable of accommodating current and future system requirements • Be at a distance to mitigate the risk of a single point of failure with the primary control center (power, water, and telecom central office) • And be at a distance in order to have multiple access routes to ensure that the current NERC two-hour time requirement could be met as well as recognizing that the time constraint could become shorter in the future • After taking all of the requirements into account it was determined that the optimal distance for a new BCC was between 10 and 20 miles from Holyoke
Outline of BCC Options • Expanding the existing BCC is not an option • There is no available room to meaningfully expand • ISO-NE evaluated five different building scenarios: • Scenario 1 - BCC & Data Center Only - Minimum BCC Footprint • Scenario 2 - BCC/Simulator, Data Center, and Incident Management Team Space • Scenario 3 - BCC/Simulator, Data Center, and emergency room for 150 first responders as well as certain basic structural requirements to enable future expansion equivalent to Scenario 4 • Scenario 4 - BCC/Simulator, Data Center, and permanent office space for 145 staff • Scenario 5 - BCC/Simulator, Data Center, permanent office space for 145 staff, and full meeting space functionality for NEPOOL • Further options include building versus renovating, and leasing versus purchasing
Scenario Analysis: Costs The following summarizes the capital costs to build each of the scenarios and assumes that the decision is made to purchase rather than lease, given that overall long-term costs are lower as a result of purchasing. These capital costs would be funded through bond issuances or other debt, which would be recouped from participants through the ISO’s annual revenue requirement. The related annual incremental cost increases are reflected in the last column in the table below.
Scenario Analysis: Selected Option • After a thorough analysis, discussions with the ISO-NE Board, the NEPOOL Budget & Finance committee, and other stakeholders, ISO-NE has concluded that Scenario 4, a 70,200 SF ($32.7M) building offers the best value • This scenario avoids an approximate $4M construction cost premium that would be required to expand the smaller building described previously as Scenario 3 into the larger Scenario 4 building at some later date while also serving as a hedge against future ISO-NE expansion needs • In addition to the Operations Training personnel, this building will be able to accommodate additional staff on site in permanent office space and will provide sufficient space to support all of the functions required by the ISO-NE Business Continuity Plan
Project Schedule • The schedule for the 2012 will focus on site acquisition, facility design, and project financing • The project is scheduled to be complete at the end of end of February, 2014 • All scheduled activities are on track
Project Budget • The project is scheduled to be completed by the end of February, 2014 • The impact on the ISO 2014 Operating Budget, as well as the impact in subsequent years , is shown on the next page • The Preliminary Capital Budget (excluding capitalized internal labor) is $32.7M – all activities are on budget
The drop in the annual revenue requirements is primarily a result of the end of depreciation for short lived (7 years and less) property $’s Millions
Project Status Completed Milestones • ISO-NE Selected Northstar Project and Real Estate Services as Project Manager • Northstar will manage the design, engineering, and construction of the project • The Northstar principal on the project is Anthony Warren, who was also the project manager on our last building project in Holyoke • On March 16,2012 ISO-NE signed a non-binding Letter of Intent (“LOI”) to purchase property in Windsor, Ct. • ISO-NE has selected the project Architect, AV consultant, and Civil Engineer • Spagnolo Gisness & Associated (“SGA”) the project Architect-SGA was also the project Architect for the Holyoke building • SG&A has begun design work • Shen, Milsom, & Wilke (“SMW”) will provide Control Room and other AV design services • AHA will be the Civil Engineer
Project Status-cont’d • Upcoming Milestones • Sign the Purchase and Sales Agreement (including a $50K deposit), and begin our due diligence on the Windsor, Ct. site (April) • Arrange for financing and prepare the required FERC Section 204 filing documents (May)
Other ISOs/RTOs • A comparison of the current BCC with the solutions in place at other ISOs/RTOs indicates that the current BCC is out of sync with the rest • The current ISO-NE BCC is a fraction of the size of its peers – even when compared to BCC facilities that are only tasked with reliability functions (BCP functions being handled at additional sites)
NYISO • Is nearing completion on a new MCC ($34.5M) and renovation/conversion of the current MCC into a new BCC and data center ($13.4M) • The project broke ground in August, 2011 PJM • Has replicated its MCC to operate as an alternate (not back-up) control center (58,370 SF) for both reliability and markets utilizing a converted former AT&T Long Lines underground bunker to function as a BCC
CAISO • Operates 2 MCC sites (Folsom & Alhambra) with complete failover capabilities (reliability and markets) to one another
MISO • Operates 2 MCC sites and a complete BCC – uses the BCC as the simulator – only the wallboard is different (simpler, front projection) ERCOT • Operates a 61,000 SF BCC – 33,000 for reliability the balance for business continuity