1 / 40

Control Status Readiness Report

Control Status Readiness Report. Eugenia Hatziangeli on behalf of the CERN Accelerator and Beams Controls Group. Outline. LHC controls infrastructure – overview Status Report on Core Controls Front ends Hardware and Software Databases Industrial Controls Machine Interlocks

george-erickson
Download Presentation

Control Status Readiness Report

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Control Status Readiness Report Eugenia Hatziangelion behalf ofthe CERN Accelerator and BeamsControls Group LHC – MAC

  2. Outline • LHC controls infrastructure – overview • Status Report on Core Controls • Front ends Hardware and Software • Databases • Industrial Controls • Machine Interlocks • The LHC timing system • Core Services and Applications • Post mortem • Sequencer for HWC • See next talk for Beam Sequencer • Logging • Software Interlock System • LSA (see next talk) • Controls Security • Role Based Access • Management of Critical Settings • LHC Controls Security Panel • Controls Infrastructure Tests • Deployment on LEIR, SPS, LHC TL • Dry runs - Commissioning • Scalability Tests • LHC Timing Crash Tests • RBAC tests • Monitor and Diagnostics • LASER • DIAMON • Injector Renovation • Summary E. Hatziangeli AB/CO

  3. Outline • LHC controls infrastructure – overview • Status Report on Core Controls • Front ends Hardware and Software • Databases • Industrial Controls • Machine Interlocks • The LHC timing system • Core Services and Applications • Post mortem • Sequencer for HWC • See next talk for Beam Sequencer • Logging • Software Interlock System • LSA (see next talk) • Controls Security • Role Based Access • Management of Critical Settings • LHC Controls Security Panel • Controls Infrastructure Tests • Deployment on LEIR, SPS, LHC TL • Dry runs - Commissioning • Scalability Tests • LHC Timing Crash Tests • RBAC tests • Monitor and Diagnostics • LASER • DIAMON • Injector Renovation • Summary E. Hatziangeli AB/CO

  4. LHC controls infrastructure – Overview CTRL CTRL • The 3-tier architecture • Hardware Infrastructure • Software layers • Resource Tier • VME crates, PC GW & PLC dealing with high performance acquisitions and real-time processing • Database where all the setting and configuration of all LHC device exist • Server Tier • Application servers • Data Servers • File Servers • Central Timing • Client Tier • Interactive Consoles • Fixed Displays • GUI applications • Communication to the equipment goes through Controls MiddleWare CMW Applications Layer Client tier DB Business Layer Server tier CMW Hardware Resource tier E. Hatziangeli AB/CO

  5. Outline • LHC controls infrastructure – overview • Status Report on Core Controls • Front ends Hardware and Software • Databases • Industrial Controls • Machine Interlocks • The LHC timing system • Core Services and Applications • Post mortem • Sequencer for HWC • See next talk for Beam Sequencer • Logging • Software Interlock System • LSA (see next talk) • Controls Security • Role Based Access • Management of Critical Settings • LHC Controls Security Panel • Controls Infrastructure Tests • Deployment on LEIR, SPS, LHC TL • Dry runs - Commissioning • Scalability Tests • LHC Timing Crash Tests • RBAC tests • Monitor and Diagnostics • LASER • DIAMON • Injector Renovation • Summary E. Hatziangeli AB/CO

  6. Front-Ends Hardware and Software • Hardware Installations • All Front-end controls equipment in place (> 200 VMEBus systems and > 250 industrial PCs) • WorldFIP infrastructure operational for PO, QPS, Cryogenics and BI (400 km network, 20.000 nodes) • General Machine Timing (GMT) network operational, including transmissions for LHC Collimators and for LHC Experiments • Front End Software Architecture (FESA) • FESA V2.10 Framework operational, including support for machine critical settings, transactional commands and in-depth diagnostics of RT behavior (via Alarms system LASER) • Front-End FESA classes developed by AB equipment groups (> 250 classes deployed on > 400 front-ends) • Deployment process =>AB-CO supports 3 last FESA releases • All industrial PCs running now Linux O/S • On-going Actions • Two major tendering exercise for the procurement of AB front-end hardware (adjudication during CERN FC in September 2008) E. Hatziangeli AB/CO

  7. Accelerator Databases Readiness Off-line Databases Online Databases Operational Settings Data model enhanced to cover functional extensions for Role Based Access (RBAC) , XPOC, Sequencer PS Controls Renovation requirements Logging Service New database hosting since Mar. 08 Common logging infrastructure for the complete accelerator chain Sustained increasing logging requirements for HWC& beam data Improved data retrieval tool Layout • Racks & electronics incorporated up to a high level of detail • Layout data is now used as foundation for the controls system Still to do • More data is being captured relating layout and assets information • Tools for data maintenance still to be put in place E. Hatziangeli AB/CO

  8. Service Availability and Data Security CTRL CTRL CTRL CTRL Additional server for testing: Standby database for LSA HWC MeasurementsMeasurements Logging Controls ConfigurationLSA SettingsE-LogbookCESAR 2 x quad-core 2.8GHz CPU 8GB RAM 11.4TB usable Clustered NAS shelf14x146GB FC disks Clustered NAS shelf14x300GB SATA disks • Service Availability • New infrastructure has high-redundancy for high-availability • Deploy each service on a dedicated Oracle Real Application Cluster • The use of a standby database will be investigated • objective of reaching 100% uptime for LSA • Secure database account granting specific privileges to dedicated db accounts • DIAMON agent on Oracle Application Servers • For all CO databases • CO puts the UR, and pays for the hardware • IT chooses the hardware, hosts, supports and maintains E. Hatziangeli AB/CO

  9. Industrial Controls • Industrial Controls for LHC have reached a high level of maturity • All systems, fully deployed for HWC in 2007, are presently in their operational version • Machine protection (PIC, WIC, QPS, Circuits) • Collimator Environment Monitoring Package (temperature, water cooling) • Survey • Cryogenic controls • Cryogenics Instrumentation Expert Tools (CIET) • Most of the SCADA applications have been ported to Linux • The front-end FESA software has been ported to Linux • Migration to last version of FESA (v2.10) to be done for next shutdown • The interface toward logging database has been consolidated • DIAMON is used for diagnostics • PLC agents are available, tested and ready to be deployed • PVSS diagnostics will be soon available E. Hatziangeli AB/CO

  10. Cryogenics Control System SCADA Data Servers Local & Central Control Rooms LHCA LHCCA LHCB LHCCB Return Module S78 & S81 Comp 1.8K LN2 Buffer Comp 4.5K Main Dryer Comp 4.5K Comp 1.8K Surface Main Dryer RM QSCCA QSAA QSCA QSDN QSCB QSCCB QSAB UCB 4.5K Cold Box 4.5K QSKA QSRA QSRB Shaft QURA Cavern CB 1.8K CB 1.8K QURCA QUI QURCB Connection Box Profibus DP PA WorldFIP Alcoves Sector 81 (3.3 Km) Sector 78 (3.3 Km) RM81 RM78 Tunnel E. Hatziangeli AB/CO

  11. Cryogenic Controls Reliability • The operation of cryogenics sectors has revealed a high risk dependency of the cryo control system on the reliability of the Technical Network • Steps taken to reduce the dependencies • PLC architecture was rationalized  no dependency on Ethernet of the cryogenics control loops for production equipments • Architecture of the network components was optimized  minimum dependency on communications equipment (switches) • Powering of network component was checked homogenization where possible with the cryo powering • Work ongoing • Identify the weak network components and improve (fiber–copper) • Consolidate the restart of communication after a network failure • Ensure interventions on Technical Network (hardware & software) are carefully planned and agreed with Operation E. Hatziangeli AB/CO

  12. Machine Interlocks for Protecting Supra-ConductingMagnets and Normal Conducting Magnets for Protecting the Equipments for Beam Operation Beam Interlock System (VME based) Powering Interlock System (PLC based) + Safe Machine Parameters system (VME based) Warm Magnet Interlock System (PLC based) Fast Magnet current Change Monitors (FMCM) E. Hatziangeli AB/CO

  13. Powering & Warm Magnets Interlocks Powering Interlock Controllers • 36 units of PLC based system protecting ~800 LHC electrical circuits • monitored via PVSS Supervision • Operational and daily used during HWC • Warm magnet Interlock Controllers • 8 units of PLC based system protecting ~150 LHC normal conducting magnets • monitored via PVSS Supervision • Operational and daily used during HWC E. Hatziangeli AB/CO

  14. Beam Interlock System  Individual System Tests successfully performed Beam Interlock Controllers 19 VME systems and ~200 connections with most of the LHC systems Monitored by Operational Application • on going BIS Commissioning (involving all User systems) • done in // with HWC • 3/8 points already performed BIS will be ready for the machine checkout… E. Hatziangeli AB/CO

  15. (TT40 incident in 2004) No marks or damage on magnet flanges Beam Vacuum chamber cut (outside view) ~110 cm (inside view) Ejected material opposite cut (inside view) E. Hatziangeli AB/CO

  16. Fast Magnet current Change Monitors - FMCM • Successful collaboration with DESY • DESYdevelopment + CERN adaptation • First units successfully used during the SPS Extraction tests and CNGS runs in 2007 • currently being re-commissioned for 2008 runs • Installation and commissioning in progress • 12 monitors deployed in the LHC (+ 14 in Transfer Lines), including ALL septa families • LHC installations to be completed next month (12 devices) • 1st version of FESA class and Java supervision available since June 2007 • minor consolidation work in progress I (A) I (A) 10 ms 500 ms FMCM triggers @ 3984.4 <103 PC current View of FMCM board FMCM trigger  0.1% drop ! E. Hatziangeli AB/CO time (ms) time (ms)

  17. Timing System major components • The LHC central timing • Master, Slave, Gateway using reflective memory, and hot standby switch • The LHC Injector chain timing (CBCM) • Master, Slave and Gateway using reflective memory, and hot standby switch • Timing is distributed over dedicated network to timing receivers CTRx in front ends • LHC and SPS safe machine parameter distribution E. Hatziangeli AB/CO

  18. Safe Machine Parameters • The SPS and LHC safe beam flags and beam energy are distributed on the LHC timing network • Work needs to be done for the final system to be ready • The CTR timing receiver modules are able to distribute the beam energy and Safe Beam flags without any software to ensures higher reliability • All timing receivers are monitored by DIAMON • Powerful diagnostics for 1000+ receivers E. Hatziangeli AB/CO

  19. Outline • LHC controls infrastructure – overview • Status Report on Core Controls • Front ends Hardware and Software • Databases • Industrial Controls • Machine Interlocks • The LHC timing system • Core Services and Applications • Post mortem • Sequencer for HWC • See next talk for Beam Sequencer • Logging • Software Interlock System • LSA (see next talk) • Controls Security • Role Based Access • Management of Critical Settings • LHC Controls Security Panel • Controls Infrastructure Tests • Deployment on LEIR, SPS, LHC TL • Dry runs - Commissioning • Scalability Tests • LHC Timing Crash Tests • RBAC tests • Monitor and Diagnostics • LASER • DIAMON • Injector Renovation • Summary E. Hatziangeli AB/CO

  20. Post Mortem - Towards Beam Commissioning Global PM Analysis:Global Event sequence, summaries, advised actions, event DB,… Global event sequence Machine Protection OK Advised Actions • Validation of machine protection features • Pre-analysis of PM buffers into result files • Flagging of interesting systems/data reduction • Database catalogue Individual System Analysis & Checks IPOC-BIS Event Sequence BLM, BPM > threshold Circuit events I/XPOC Data completeness and consistencycheck at system and global level (minimum data, configurable) Upon beam dump / self triggering, systems start pushing data to PM system, Logging, Alarms, etc… BLM BPM PIC/WIC QPS FMCM FGC BIS XPOC … E. Hatziangeli AB/CO …

  21. Post Mortem Readiness • Many tools are ready for HWC and tested on sectors 5-6 and 7-8 • Implemented since December 2007 • Automatic Test Analysis on three applications (expert override possible) • Calculated result parameters sent to Sequencer for MTF upload. • Well defined GUI for each test step. • Test results electronically signed by role using RBAC • Event recognition with Event Builder • Redundant services for PM collection and data • Scalability tests with first beam clients started (BLM, BPM) • To do for 2008 • Parallel sector commissioning still to be tested • For the 600A circuits many steps are still to be automated. • Further validation tests with beam clients to be done (BLM, BPM, RF, etc…) • Extend framework from HWC to beam operation • Implement higher level of Automated Test Analysis • Data completeness checks & Individual system tests E. Hatziangeli AB/CO

  22. Readiness of HWC sequencer • First version of the sequencer deployed in early 2007 • Many new versions with improvements deployed since • HWC Sector 7-8 May-Jul 07 4-5 Winter 08, 5-6Spring 08 • ~ 35 sequences written and maintained by 3 HWC experts • Sector 4-5: over 1700 sequences executed, in 5-6 over 600 • Essential tool for HWC • Overall it works well and satisfies the requirements • Sequencer (the tool) is complete, no important new features needed • Sequences (the tests) are maintained by HWC experts • Ready for multi-sector / multi-front HWC • Used in multi-front operations for over a year • Recent experience in multi-sector operations • “normal” HWC is done in sector 78 • training quenches are done in sector 56 • No scalability issues are anticipated E. Hatziangeli AB/CO

  23. Logging Service Readiness • Logging for Operation • Data logged from PS Complex, SPS, CNGS, LHC HWC, LHC , any type of equipment • Processes run continuously on dedicated machines • Monitored through Alarms system LASER & DIAMON & diagnostic application • Logging for equipment commissioning • Dedicated service, running in the environment of the specialist • Aim: validation of the equipment behavior before operational deployment • No interference with Operational logging • Requirement for a watchdog system (coming weeks) • For critical data (INB, CNGS neutrino events, ...) continuous monitoring of data logged in DB, generation of a specific alarm E. Hatziangeli AB/CO

  24. Software Interlock System - SIS Overview • Very useful system to anticipate failures and gives early alarms • Accommodates complex interlock logic • Complements BIS (hardware) as protection system • Proved to be reliable tool for operations • Excellent experience in SPS (900 parameters monitored) E. Hatziangeli AB/CO

  25. SIS for LHC • Gives 2 Permits for Injection BICs(Beam1 & Beam2) • All PCs not HW interlocked (~ 800, orbit correctors, warm magnets) • Current of separation dipoles and MCBX orbit correctors • Ring & injection screens (only IN when mode inject-dump) • Extraction screens • Circulating beam intensity limit • Gives Permit for the LHC ring (dumps the beam - initially alarms) • Integrated field of orbit correctors (beam dump energy tracking) • Extraction screens combined with intensity + energy • Orbit at TCDQ • Future work • RBAC integration • Critical settings monitoring (MCS) from LSA • Refinement of the configuration as we progress with LHC E. Hatziangeli AB/CO

  26. Outline • LHC controls infrastructure – overview • Status Report on Core Controls • Front ends Hardware and Software • Databases • Industrial Controls • Machine Interlocks • The LHC timing system • Core Services and Applications • Post mortem • Sequencer for HWC • See next talk for Beam Sequencer • Logging • Software Interlock System • LSA (see next talk) • Controls Security • Role Based Access • Management of Critical Settings • LHC Controls Security Panel • Controls Infrastructure Tests • Deployment on LEIR, SPS, LHC TL • Dry runs - Commissioning • Scalability Tests • LHC Timing Crash Tests • RBAC tests • Monitor and Diagnostics • LASER • DIAMON • Injector Renovation • Summary E. Hatziangeli AB/CO

  27. Role Based Access (RBAC) Overview • Need to prevent • Well meaning person from doing the wrong thing at the wrong moment • Ignorant person from doing anything at any moment T Configuration DB Application RBAC T • Authentication: • User requests to be authenticated • RBAC authenticates user via NICE user name and password • RBA returns Tokento Application • Authorization: • Application sends token to Application Server (3-tier env.) • CMW client sends token to CMW server • CMW server (on front-end) verifies token • CMW server checks AccessMapfor role, location, application, mode • RBAC Token: • Application name • User name • IP address/location • Time of authentication • Time of expiry • Roles[ ] • Digital signature (RBA private key) Application Server CMW client T CMW server Access MAP FESA E. Hatziangeli AB/CO

  28. Management of Critical Settings - MCS Need to ensure • Critical parameters, which can compromise the safety of the machine • can only be changed by an authorized person and nobody else • are what they are supposed to be • MCS ensures • Critical parameters are only changed by authorized person • RBAC for Authentication & Authorization • It signs the data with a unique signature to ensure critical parameters have not changed since the authorized person has updated it • Public-private key digital signatures E. Hatziangeli AB/CO

  29. LHC Controls Security Panel - LCSP • The LHC Controls Security Panel is mandated to address all the technical and non-technical issues concerning AB security for Controls • Take responsibility for the RBAC data (ROLES and RULES) • Ensure all critical parts of the machine are protected • Take responsibility for the CNIC actions • reduction of Trusted list, change of operational account passwords,.. CNIC: Computing and Network Infrastructure for Controls E. Hatziangeli AB/CO

  30. Outline • LHC controls infrastructure – overview • Status Report on Core Controls • Front ends Hardware and Software • Databases • Industrial Controls • Machine Interlocks • The LHC timing system • Core Services and Applications • Post mortem • Sequencer for HWC • See next talk for Beam Sequencer • Logging • Software Interlock System • LSA (see next talk) • Controls Security • Role Based Access • Management of Critical Settings • LHC Controls Security Panel • Controls Infrastructure Tests • Deployment on LEIR, SPS, LHC TL • Dry runs - Commissioning • Scalability Tests • LHC Timing Crash Tests • RBAC tests • Monitor and Diagnostics • LASER • DIAMON • Injector Renovation • Summary See next talk E. Hatziangeli AB/CO

  31. LHC controls infrastructure Scalability • Tests =>Preliminary results, issues & foreseen solutions • Systems which scale to LHC full load • Software Interlock System SIS • Data Concentrators (BLMs, BPMs) • Alarm system LASER (new architecture) • Controls Middle Ware (CMW) /Java API Parameter Control (JAPC) • Diagnostic & Monitoring tool DIAMON • Systems potentially critical (tests ongoing- results mid June) • Post Mortem • Logging service • Scalable to LHC load from all clients except LHC BLM • Preliminary limit : ~ 5000 parameters/second • Bottleneck : SQL calls management by Oracle Server E. Hatziangeli AB/CO

  32. Failures in Central Timing • Tests have been performed to validate the behaviour of the Controls Infrastructure when the Central Timing crashes • These “crash” timing tests are on going • Results • The behaviour of the control system with no timing is correct • The application programs, servers and front ends recovered without manual intervention when timing returns E. Hatziangeli AB/CO

  33. RBAC Dry Runs • The LHC Controls Security Panel (LCSP) is preparing an RBAC dry-run end June/early July • The RBAC default behavior is changed to • “Access with no RBAC token is refused” • Property not protected is not authorized • All Equipment servers will be loaded with RBAC access maps • Typical applications will be tested • LHC 2-tier & 3-tier applications • LHC Core controls (LSA) • Background servers, concentrators • Fixed Displays E. Hatziangeli AB/CO

  34. Outline • LHC controls infrastructure – overview • Status Report on Core Controls • Front ends Hardware and Software • Databases • Industrial Controls • Machine Interlocks • The LHC timing system • Core Services and Applications • Post mortem • Sequencer for HWC • See next talk for Beam Sequencer • Logging • Software Interlock System • LSA (see next talk) • Controls Security • Role Based Access • Management of Critical Settings • LHC Controls Security Panel • Controls Infrastructure Tests • Deployment on LEIR, SPS, LHC TL • Dry runs - Commissioning • Scalability Tests • LHC Timing Crash Tests • RBAC tests • Monitor and Diagnostics • LASER • DIAMON • Injector Renovation • Summary E. Hatziangeli AB/CO

  35. Alarms (LASER) for LHC • An important increase in expected alarm events • Required availability 365 days/24h • Alarm console extended • Allows for dynamic grouping of alarms • New alarm definition database schema • Ensures the data quality by reducing redundancy and protecting against incomplete data • Alarm server modified fundamentally to allow • Fast response to increase in load • Increased resilience to external failures and improved diagnostics tools LASER console LASER DB LASER CORE LASER source LASER source LASER source E. Hatziangeli AB/CO

  36. DIAgnostic & MONitoring System - DIAMON • DIAMON provides • Software infrastructure for monitoring the AB Controls Infrastructure • Easy to use first line diagnostics and tool to solve problems or help to decide about responsibilities for first line intervention Group View Navigation Tree Monitoring Tests Details Repair tools E. Hatziangeli AB/CO

  37. Outline • LHC controls infrastructure – overview • Status Report on Core Controls • Front ends Hardware and Software • Databases • Industrial Controls • Machine Interlocks • The LHC timing system • Core Services and Applications • Post mortem • Sequencer for HWC • See next talk for Beam Sequencer • Logging • Software Interlock System • LSA (see next talk) • Controls Security • Role Based Access • Management of Critical Settings • LHC Controls Security Panel • Controls Infrastructure Tests • Deployment on LEIR, SPS, LHC TL • Dry runs - Commissioning • Scalability Tests • LHC Timing Crash Tests • RBAC tests • Monitor and Diagnostics • LASER • DIAMON • Injector Renovation • Summary E. Hatziangeli AB/CO

  38. Injector Controls Renovation - Status Report Injector Controls Architecture - InCA • Architecture validation with critical Use Cases • Check interfacing of the various components • LSA core • Standard CO components for Acquisition • Standard PS and LSA Applications interfaced to the core • Check data flow • Low-level trim and monitoring values of correctors • Orbit correction using LHC Beam steering application • High-level trim + drive a front end • Results • Whole data flow validated • Architecture closer to the final one Injector Complex FE Renovation – 2nd half of 2008 • A "strategic" plan for the renovation of the FE controls infrastructure is due by mid-2008 • Development and validation of new Front-end solutions in view of their first deployment in 2009 E. Hatziangeli AB/CO

  39. Summary • The LHC controls infrastructure had been targeted for readiness for an engineering run at 450 GeV in November 2007 - This goal has been met. • The ongoing hardware commissioning and the extensive use of programs and databases (“learn by doing”) have significantly changed the specifications and the resulting follow-up and work has been done. • Additional functionality has been prepared in 2008- network security (RBAC)- diagnostic tools (DIAMON) E. Hatziangeli AB/CO

  40. End E. Hatziangeli AB/CO

More Related