Controls and Monitoring Implementation Plan

1. Controls and Monitoring Implementation Plan J. Leaver 03/06/2009

2. Implementation Issues • Organisation & responsibilities • General EPICS infrastructure • EPICS server / client organisation • Unification of control systems • Remote access • Monitoring • Controls • Configuration database • Schedule

3. Organisation of Control Systems • Original plan was for Daresbury Lab (DL) to provide all controls for the experiment • DL responsible for many existing C&M systems (excellent quality) • Unfortunately, recent funding issues have limited collaboration’s ability to pay DL for new work • DL to continue with current projects (where possible) • MICE community to take responsibility for additional C&M systems

4. Organisation of Control Systems • MICE Online Group (MOG) created in January • Aim: Organise data acquisition, C&M & online reconstruction • Controls & Monitoring Leader (JL) • Identify control requirements for each section of MICE • Decide on most appropriate solution • Coordinate the effort of those involved in implementing agreed solution

5. Organisation of Control Systems • MOG directly responsible for C&M infrastructure • Network/hardware organisation • Integration of control systems (with each other & the rest of MICE) • User experience (i.e. how operators interact with ‘global’ C&M system) • For individual projects, each group within MICE should be responsible for own system(s) • Contributing either EPICS development effort or funds for a 3rd party (e.g. DL) to complete required work • Where necessary, MOG contributes developer effort • However, very limited resources available (~1.5 man years per year) • Currently seeking additional support within the community

6. EPICS Client / Server Overview

7. EPICS Server / Client Organisation • Wide variety of EPICS server applications permitted • Typically connect to physical hardware • Impossible to enforce common interface/processor/OS specifications • Each server maintained by ‘owner’ of respective control system • Strict central administration unnecessary – ‘end user’ only concerned with availability of PVs on network • EPICS clients also varied, but must be uniformly accessible • Users should not have difficulty finding/launching clients • Applications should be consistently organised/updated • MOG responsibility

8. EPICS Client Organisation • All client-side applications run on miceecserv • Central installation repository greatly simplifies configuration/maintenance/backup • MOG collates individual applications, applies updates when available from control system ‘owners’ EPICS client applications miceecserv miceopi1 miceopi2 Controls Network EPICS server applications EPICS IOC EPICS IOC Portable CA Server Portable CA Server EPICS IOC

9. EPICS Client Organisation • Client control/monitoring GUIs viewed directly on miceecserv, or one of 2 ‘Operator Interface’ PCs • OPI PCs act as ‘dumb terminals’, running displays from miceecserv via SSH EPICS client applications miceecserv miceopi1 miceopi2 Controls Network EPICS server applications EPICS IOC EPICS IOC Portable CA Server Portable CA Server EPICS IOC

10. Unification of Control Systems • At user level: Simple ‘wrapper’ GUI provides menu for launching individual client applications • At system level: Employ 2 standard EPICS tools (running as background services on miceecserv) • Alarm Handler • Monitors all servers & warns operators of abnormal/dangerous conditions • Channel Archiver • Automatically records PV parameters to disk & provides several visualisation options • See P. Hanlet’s talk

11. User Interface

12. User Interface Message log Alarm Handler Large wall-mounted display Any important parameters for current run

13. User Interface Client application launcher Client GUI Standard desktop monitor

14. User Interface Connected to miceecserv

15. User Interface Connected to miceopi1 Connected to miceopi2

16. Remote Monitoring: General Principles • Remote users should have simple, easily accessible interface for routine monitoring • ‘Expert’ remote users should have access to monitoring displays which match those in MLCR • No machine on Controls Network should be directly accessible over the internet • System load generated by remote monitoring should have minimal impact on control & monitoring services

17. Remote Monitoring: Web Server RAL Gateway Java Archive Viewer Data Server Web Server NFS Mount CGI Export miceecserv PPD Network Web browser Channel Archiver PV Archive Internet Controls Network EPICS IOC EPICS IOC Portable CA Server Portable CA Server EPICS IOC

18. Remote Monitoring: Direct PV Access • Could recreate normal client displays using web interface, but would involve impractical development overheads • Provide direct read only access to PVs so actual client GUIs may be run remotely miceecserv RAL Gateway Standard client GUI running on remote PC (read only) CA Gateway (read only) CA Gateway (read only) Controls Network EPICS IOC EPICS IOC Portable CA Server Portable CA Server EPICS IOC

19. Remote Monitoring: Direct PV Access • CA Gateway makes PVs available across subnets (with full access control), while minimising load on underlying servers • To simplify end-user support, virtual machine disk image containing EPICS + all client applications will be made available miceecserv RAL Gateway Standard client GUI running on remote PC (read only) CA Gateway (read only) CA Gateway (read only) Controls Network EPICS IOC EPICS IOC Portable CA Server Portable CA Server EPICS IOC

20. Remote Control • Where possible, operations affecting the state of any MICE system should only be performed within MLCR • Remote users accessing controls can lead to unknown/unexpected running conditions – should be discouraged • If necessary, off-site experts will be permitted to run control client applications on miceecserv, via SSH through RAL Gateway • Each expert will have an account on miceecserv which only contains client applications for their designated system

21. Configuration Database • Necessary to integrate control systems with central MICE Configuration Database • Read set point values from database • Upload PV values to EPICS servers • Modify PVs with client GUIs • Download PV values from EPICS servers • Write new set point values to database • For (2) & (4), could use standard EPICS Backup & Restore Tool (BURT) • Backup/restore PV values to/from ‘snapshot’ files • However, interfacing snapshot files with database introduces significant overheads • Propose creation of custom backup/restore client

22. Configuration Database • Simple client application • Read/write PV values via MICE C++ wrapper for CA C-bindings • XML configuration file specifies PV names, correct sequence for write operations • Import/export sets of PV values from/to XML string • Read/write XML string from/to database via Configuration Database API • Manual backup/restore • State tagged with time, user-generated identification string, etc. • Monitoring of DATE DAQ state • Automatic backup at start of each run

23. Configuration Database • Additional requirements • Throughout each DAQ run, all set point values should be held in state defined by the last Configuration Database ‘snapshot’ • If values change, system in unknown state • Cannot perform automated analysis of run data • While DAQ in run state, client monitors all set point values • If any parameters are modified • Set PV to indicate invalid run state (read into DAQ stream) • Set warning on Alarm Handler display

24. Configuration Database • Configuration Database interface still in early design stages – work not commenced • J. Leaver/P. Hanlet to develop EPICS client • D. Forrest to implement database API functions for parsing/formatting EPICS set point XML strings • Details of run state PV monitoring to be confirmed

25. Infrastructure Schedule