360 likes | 759 Views
FEP Diversity Project. RTS System Admin Group RTUs, Communications. Substation – EMS Communications Overview Traditional 4-wire communications FEP Diversity Project Motivators Pilot Project - POC Technical Solutions Communications Architecture AREVA Database Modifications/Changes
E N D
FEP Diversity Project RTS System Admin Group RTUs, Communications
Substation – EMS Communications Overview Traditional 4-wire communications FEP Diversity Project Motivators Pilot Project - POC Technical Solutions Communications Architecture AREVA Database Modifications/Changes Additional Benefits Future Direction – Expansion to other Substation needs Agenda
Substation to EMS data flow Substation
SCADA protocols • Protocol is a way of organizing and presenting data in a consistent, repeatable format • Bit and byte oriented protocols • Byte - Harris, DNP • Bit - Conitel, CDC2, TRW S9000, TRW S9550 • Special interfaces - Intrac and MOSCAD radio systems • All master station SCADA protocols do the same basic functions • analog data conversion • status point monitoring • support “select before operate” function for controls • provide analog output capability • provide different types of scans to capture transient events between scans and minimize bandwidth usage • provide some form of “cyclic redundancy check” CRC so that if the received message has been corrupted during transport it will be rejected.
FE RTU numbers and types • Approximately 2000 RTUs on FE system • 650 in west, 1350 in the east and growing • Diverse hardware platforms • Diverse protocols – including: • DNP 3.0 • Harris 5000 • CDC II • TRW S9000 / 9550 • Conitel 300/3000 • Modbus • Intrac/Moscad
RTU communications • Primarily four wire leased circuit • 1200 baud modem • Radio used extensively in east ~ 650 radio equipped RTUs • Primarily found at line switch controllers • Dialup - Approximately 100 dialup RTUs in west • Primarily found at line switch controllers
Substation to EMS data flow Substation
What is a FEP? • Windows 2k3 server OS • Standard Infrastructure server Hardware • One exception, the FEP has additional Digi PCI cards so that server has 128 COM ports • Runs the AREVA e_terraControl suite of applications • Process Starter • CFEReader – Polls RTU’s • SCADA – Contains the database • ISD – Connects the FEP to the EMS • Ancillary processes: • AlarmPager (provides method to pass FEP alarms to EMP) • SpaceSaver • DBUpdate • Key: No FEP’s = No Data entering the EMS
Project Motivators • Issue identified by NERC as a disaster recovery limitation • Loss of DataCenter, either from a physical or communications perspective, essentially eliminates the EMS system. • Upgrade the method of communications to critical transmission substation RTU’s. • Network change without causing system software changes • Keep serial connected FEP ports and serial connected RTUs • Improve Communications Network Reliability
Project Overview • Proposal to operate selected critical transmission RTU’s from physically diverse FEP’s • 3rd/4th Quarter 2005 performed R&D proof of concept testing • Production deployment began 1st quarter of 2006 • Initial sites include interconnection and critical transmission sites • By the end of 2006 we successfully cut over 54 production sites • By the end of 2007 we successfully cut over 104 production sites – All 345 and 138kV Tx sites in FE’s MISO area • 111 Transmission sites in PJM are scheduled for completion though 2012 • All new FE RTU sites are being installed on this technology
Pilot Objectives - What we wanted to prove? • Demonstrate that the AREVA system can support data acquisition using diversified FEP’s at the same level of reliability as currently observed with co-located FEP pairs. • Verify the proposed network communications architecture will support geographically distributed FEP’s. • Demonstrate that in using diversified FEP’s there is no degradation in AREVA system level performance (FEP and EMP). • Demonstrate that alarm processing performance is equivalent to that found with legacy designs - throughput. • Verify system responds properly to the loss of inter-FEP heartbeat.
Required work activities in R&D Pilot period • Build new servers (Windows and Unix) • Develop a point lists for each RTU • Develop software configuration for each RTU • Develop a strategy for modeler configurations • Develop a standard network hardware configuration • Configure AREVA Systems • Configure, debug, and test pilot system • Develop Cut-over strategies for production
Measure of success in all pilot phases? • Burn in test using all RTU’s for a one week period of local continuous polling is successful • No FEP process shutdowns • Minimal missed scans • Alarm propagation and data integrity - 24 hour test • All status points change state at least once per scan interval • All analog points change each scan interval • No missed scans or missed alarms • System stability is maintained for following failover scenarios: • Loss of one FEP • Loss of inter FEP heartbeat • Loss of one EMS server
FEP Diversity Pilot Phases • Phase 1 - 10/3/05-10/13/05 • Performed local proof of concept testing using new FEP’s and 55 logical RTU’s. Using DYMEC supplied network hardware, failure analysis was be conducted and logical RTU’s polling statistics were gathered. FRAME network is all mocked up locally at DataCenter B. • End Result- Satisfactory test results based on operational performance and failure analysis. All testing performed locally at DataCenter B.
FEP Diversity Pilot Phases -CONT. • Phase 2 - 10/14/05-10/25/05 • DYMEC Equipment was extended to DataCenter A to allow controlled diversified FEP testing. Required relocating a FEP server to DataCenter A, and the associated FRAME network connectivity. • End Result- Satisfactory test results based on operational performance and failure analysis.
FEP Diversity Pilot Phases - CONT. • Phase 3 -10/26/05-11/16/05 • Selected substations (6 site’s) will be dual ported and connected to digital circuits. Digital FRAME circuits will appear at DataCenter A and DataCenter B where the diversified FEP’s and EMS servers reside. • Includes site’s on the FE, and the Telco (leased) FRAME network’s • End Result- Satisfactory test results based on operational performance and failure analysis.
Results of Pilot • Project team successfully installed and configured the new versions of the SMP software on FEP pair. • Latest Windows O/S • Latest AREVA SMP Software • New hardware components • Verified successful RTU communications using current serial communications. • All three phases of RTU testing completed successfully, including phase III’s real-life field tests • Developed Cutover Plan
Production Implementation Project • Order communication circuits as required • A majority of circuits require High Voltage Protection (Positron) • Make necessary RTU configuration changes • Baud Rate / Modem settings • Complete AREVA modeler work to allow cutover to occur • Run ManageSiteAssignments.pl script to use Primary/Secondary SITE ID • Make PATH changes for new communication circuit • Cutover RTU’s to Diversified FEP • Perform full station check-out on new circuit
Cut-over Strategy (part1) • Need to have both existing and diversified RTU modeled for cutover in AREVA EMS • Model an additional TFE structure for new diversified FEP • Changes to BAUD, PTMO1, PTMO2 and NoReplyTimeout • Additional RTUC Record pointing to same RTU record as 4-wire circuit in production • Utilize SITE_* and SITE2_* fields to facilitate the cutover • Each RTU has potentially hundreds of POINT/ANALOG/COUNT records associated with it. • These all need their SITE_* field set to the ID_SITE of the new diverse FEP. • SITE2_* field set to the ID_SITE of the current 4-wire FEP in production • Manage those individual data items that do NOT allow automatic SITE/SITE2 selection. • Selection is done automatically by SCANNER based on quality. Items with no status, (CTRL only) do not have a good quality code to follow (TAP Changers) • Areva TFE tree changes made, No RTU tree changes made. • Limit the risk of the change – No production affects
Cut-over Strategy (part2) • RTU Changes • New FRAME T1 Circuit can be installed at any time on spare pairs • Make RTU configuration changes and load onto spare D20 processor board • Swap out board for cutover, if there are issues original board can be immediately put back • No Point mapping changes made
POST DESIGN BENIFITS • DR Environment • Remote RTU can be pulled from either DataCenter • Eliminates costly – Out-of-date Analog 4-wire circuits • Increase RTU speed/poll interval • 1200bps to 9600bps (max.: 38400 bps) • Ability to pull byte and bit-oriented RTUs • Remote Support • Allow for mobile workforce support/configuration/data analysis – Secured IP Based • Security • Serial Connectivity – Isolation of corporate substation traffic • HP/OV Support • Complete Network end-to-end alarming • “Smart” Substation platform • Ability to add additional PVC’s for other needs • IP enabled Security Cameras for monitoring site • Swipe card entry systems • CIP Cyber Security full compliance • Non-Routable Transport