140 likes | 154 Views
Learn about the commissioning process for the cryogenic system at the European XFEL, including simulating I/O, testing before installation, diagnostics, record structure, and trust in control systems. Understand the importance of testing PLC programs and creating spare records. Discover best practices for EPICS/IOC versioning and implementing redundancy.
E N D
Commissioning the European XFEL cryogenic system and best practice EPICS Meeting @ Barcelona 7. October 2017 Matthias Clausen DESY Deutsches Elektronen Synchrotron
Commissioning IOCs before the I/O is installed • Everybody knows this situation: As soon as the last (or even the first?) cable is connected you get the question:„When can we switch on“? • A way to ‚getting closer‘ to the turnkey system is to simulate the I/O. In our case we simulate _all_ I/O connected to the Profibus. This makes: • The Profibus driver happy because it will see the complete bus • The IOC happy because all records get initialized properly • The engineer happy because he can troubleshoot his database • In the end you get close to a situation where the IOC is ready as soon as the ‚real‘ fieldbus is connected name, DESY title of presentation, 22-May-06
Simulating I/O WinMOD name, DESY title of presentation, 22-May-06
The simulation tool • WinMOD is a commercial tool • Runs on a Windows PC • Special Profibus card • Configured by XML file generated by the same CSS-based I/O configurator which is used to configure the Profibus. name, DESY title of presentation, 22-May-06
Testing I/O before it gets installed • All cryogenic controls racks for the XFEL tunnel were fully equipped with all componets before they got installed in the tunnel • This gave us the chance to test the I/O with the final production EPICS database • Hardware and software errors could be removed before installation • Only a few (per mill) errors were left in the final tests in the XFEL tunnel name, DESY title of presentation, 22-May-06
Diagnostics, diagnostics, diagnostics • Good diagnostic tools help to get started faster during commissioning. • We installed diagnostic tools on each Profibus line to diagnose the state and status: • On each Profibus line to diagnose protocol/ cable errors • On each Profibus line to collect status information from the installed equipment for condition monitoring name, DESY title of presentation, 22-May-06
Diagnostic Tools ProfibusDiagnostic Condition Monitoring name, DESY title of presentation, 22-May-06
Separation of Concern name, DESY title of presentation, 22-May-06
Record Structure name, DESY title of presentation, 22-May-06
Trust is good control is better … • Testing PLC programs before they get installed(Specially from 3rd parties) • What can go wrong? • Wrong version (not the new one) • New version based not on current but an older version • Structure of communication protocol changed without notice • New functions are in conflict with EPICS IOC • What to do? • Prepare PLC / IOC test bench with is always available to test new PLC versions – but also changes on the IOC side … name, DESY title of presentation, 22-May-06
Spare Records • Specially when IOCs have to run 24/7 for a long time: • Create spare records in each IOC. In our case: • A few AI ~ 10-20 • Some PID ~ 20 • Many CALC records ~ 30 • New functions/ feature can be implemented ‚on the fly‘ and do not have to wait until the next boot.(May be several months in the future) name, DESY title of presentation, 22-May-06
IOC Boot • What is the current version running in the IOC? • What is the next version?(Available – but not active) • What was the previous version?(The way back should always be possible) name, DESY title of presentation, 22-May-06
Redundancy • File Server (NetApp) • Network • UPS • IOC • Fieldbus (ring structure) name, DESY title of presentation, 22-May-06
Summary • Test as much as possible before it is installed in the field • Do not save on diagnostics • Create you own standards on EPICS/ IOC versioning(or copy a good one from somebody else) • Redundancy wherever you can effort it! name, DESY title of presentation, 22-May-06