Soft-IOC Based Alarm Handler

1. Soft-IOC Based Alarm Handler Pam Gurd July 14, 2006

2. Why? • Wanted to be able to incorporate alarm summaries in edm screens • And to call edm screens from alarm screens.

3. Where? • Standard soft IOC • Alarm_softIOC/Devel…/commonApp/ • src contains the template for the logging sequences • Db contains the database templates. • srcOpi holds the common alarm screens and the templates to be used to build specific alarm screens. • Alarm_softIOC/Devel…/alarmTopApp/srcOpi contains specific menus (for example, for IOC heartbeat screens).

4. Where? continued • iocBoot/ics-ioc-linux-alrm soft IOC for common stuff (now alarmModes, later summaries of everything.) • Alarm_softIOC/Devel…/xml_handlers contains the scripts that follow. • Alarm_softIOC/Devel…/alarmCfg contains the alarm configuration .xml files. Presently defined: • ICS_IOC_HBt.xml – IOC heartbeats (non-RF) • ICS_IOC_HBt_Soft.xml – Soft IOC heartbeats • ICS_RF_IOC_HBt.xml – RF IOC heartbeats • PPS_ICS_Chmk_list.txt – chipmunks • ICS_RF_Util.xml – RFQ LLRF IOC1 utility status • CF_CU_config.xml – example xml produced from a CF .alhConfig file • CHL.xml – CHL alarms

5. What? • Set of scripts to create soft IOCs from alarm configuration files. • make_all_alarm_apps.sh – calls make_alarm_app for each configuration file • make_alarm_app.sh – makes an alarm app (<s><ss><n>App) and an iocBoot directory for a soft IOC (<s>-<ss>-ioc-linux<n>) using the standard makeBaseApp templates. • SNS-name-centric-ness has been reduced somewhat. (eg, you can define the “App” and “ioc” names you want.)

6. What’s in the application? • Scripts define a soft IOC from an alarm configuration file • make_db_files.pl – makes all the EPICS database files: summaries, disable bits, timestamps, alarm logs, etc. • make_screen_substitutions.pl – makes a substitutions file to be used to create the alarm edm screen. • make_alarm_seqs.pl – makes the EPICS image file including the sequence to handle the alarm log • make_arreqs.pl – makes the archive req files (includes, mostly, the alarm log.) • make_ioc.pl – makes the autosave .req file and adds the databases, sequences and the autosave .req to the soft IOC startup command script.

7. Other scripts • Utilities for scripts to create soft IOCs from alarm configuration files. • get_alarm_app.pl – translates S, SS and N to sSsN for app name (eg, S=PPS, SS=ICS, N=_Chmk translates to ppsICS_ChmkApp) • (We’ll call that sSsNApp later on.) • get_ioc_name.pl – translates S, SS and N to <s>-<ss>-ioc-alrm<n>, converting “_” to “-” for app name (eg, S=PPS, SS=ICS, N=_Chmk translates to pps-ics-ioc-linux-alrm-chmk)

8. Configuration File • Top-level Summary definition • Mid-level Summary definitions (up to 144 for each top-level summary) • PV lists for each (up to 144 to each mid-level summary) • + all specifications needed to build all the goodies described previously

9. Configuration File Elements • XML files parsed using XML:Simple. An <alarmconfig> contains 1 or more <top_alarm> and/or <summary> and may contain a <display>. A <top_alarm> contains 1 or more <alarm> and/or <summary> and may contain a <display>. A <summary> contains 1 or more <alarm> and may contain a <display>. An <alarm> contains 1 or more <pv> and may contain a <display>. A <pv> specifies a channel to watch and may contain a <display>. A <display> specifies a file to show the device or system status and the parameters to use in calling it.

10. Configuration File: <top_alarm> attributes • sns_top_s – usual SNS system • sns_top_ss – subsystem • sns_top_n – device instance • Or top_pv – to make this less SNS-specific • eg <top_alarm sns_top_s=“PPS” sns_top_ss=“ICS” sns_top_n=“_Chmk”> • These are used for the App name, the soft IOC name, and the PV names for the top-level summary and alarm log.

11. Configuration File: <alarm> attributes • Up to 144 <alarm>’s for each <top_alarm> • sns_s – usual SNS system • sns_ss – subsystem • sns_n – device instance • Or name • mode – EPICS PV to define machine mode. • mode_desc – Short string to describe machine mode. • eg <alarm sns_s=“PPS” sns_ss=“Lin” sns_n=“_ChmkA” mode=“Tgt_ICS:Summary:MachMode” mode_desc=“Tgt”>

12. Configuration File: <pv> attributes • (up to 144 <pv>’s for each mid-level summary) • name – EPICS PV name • Or sns_hbt_name and sns_hbt_sig for IOC heartbeats • desc – text description of alarm • eg <pv name=“ICS:IOC_linux1:HBtNLOK_FE1” desc="Front End IOC 1“>

13. Configuration File: <display> attributes • file – edm screen filename for details • parameters – macro definitions for calling the detail screen • type – “edm_file” or “edm_menu” (allows bringing up vxStats/utility/timing screens, for example. Defaults to “edm_file”. • eg <display file=“pps_frntend_phase1_3” parameters=“S=Cryo,SS=ICS,N=57,DI=''” type=“edm_menu”>

14. Not complete • There are other attributes, but this is long enough already.

15. WARNING Don’t edit the stuff “Under the hood” and then go back and run the script again. Your changes will be deleted.

16. How to Run the Script • In $IOCTOP/Alarm_softIOC/<vers> make alarm_apps Performs this command: commonApp/src/make_all_alarm_apps.sh creates an app for each *xml file in alarmCfgs. Or commonApp/src/make_alarm_app.sh <full pathname>/<your file> creates one app. Don’t forget the warning!

17. Under the Hood – EPICS Database Files (sSsNApp Db directory) • alarmModes.db – Defines machine modes for mid-level summaries (loaded by the soft IOC ics-ioc-linux-alrm.) • <S>_<SS>_Alarm<N>.substitutions – defines records using the following VDCT templates: • alarmLatch.template • alarmLog.template • sumLatch.template • topSumLatch.template • <S>_<SS>_ Alarm<N>_top.db – records that are directly written by the script • <s>-<ss>-ioc-linux-alrm<n>.substitutions – standard linuxStats soft IOC status.

18. Under the Hood – Top-Level Summary PVs • <s>_<ss>:Summary<n>:StsCurr – current highest severity of included PVs (masked as specified in disable bits) • <s>_<ss>:Summary<n>:StsLtch – summary of all lower-level latches • <s>_<ss>:Summary<n>:StsNotOK– actual current highest severity of included PVs, not masked by disable bits • <s>_<ss>:Summary<n>:ClrCmd – command to clear latches, including all lower-level latches • <s>_<ss>:Summary<n>:Ltch – highest alarm severity since last latch clear command • <s>_<ss>:Summary<n>:LtchCh – 1 when the latch is becoming set • <s>_<ss>:Summary<n>:LtchTS – timestamp the last time the Latch value changed. • <s>_<ss>:Summary<n>:DisSumSts – summary of all lower level disables

19. Under the Hood – Logging PVs(for each Top-Level Summary) • <s>_<ss>:Summary<n>:AlarmLog – Description of last PV of which the alarm severity increased (masked by disable bits) • <s>_<ss>:Summary<n>:AlarmLogTime – the time that the alarm severity increased • <s>_<ss>:Summary<n>: AlarmLogSevr – the new alarm severity • Similarly, <s>_<ss>:Summary<n>:NotOKLog logs the unmasked alarm PV, and <s>_<ss>:Summary_ioc:AlarmLtchLog logs the latch state • Each Log PV is rolled into a set of PVs with a number from 00 to 19 appended to the PV name so the last 20 alarms can be viewed on the Alarm Log screen.

20. Under the Hood – Mid-Level Summary PVs • <s>_<ss>:Summary<n>:StsCurr – current highest severity of included PVs (masked as specified in disable bits) • <s>_<ss>:Summary<n>:StsLtch – summary of all lower-level latches • <s>_<ss>:Summary<n>:StsNotOK– actual current highest severity of included PVs, not masked by disable bits • <s>_<ss>:Summary<n>:LtchTS – timestamp the last time the Latch value changed. • <s>_<ss>:Summary<n>:Ltch – highest alarm severity since last latch clear command. • <s>_<ss>:Summary<n>:ClrCmd – command to clear latches, including all lower-level latches • <s>_<ss>:Summary<n>:Dis – disable at this level • <s>_<ss>:Summary<n>:DisSumSts – summary of all lower level disables and the disable at this level • <s>_<ss>:Summary<n>:StsDis – summary of all lower level disables • <s>_<ss>:Summary<n>:ModeSts – state of mode status PV • <s>_<ss>:Summary<n>:LtchCh – 1 when the latch severity increases.

21. Under the Hood – for each PV • <pv>AlarmSts – current severity, masked by disable bits as specified. • <pv>AlarmLtch – highest alarm severity since last latch clear command • <pv>AlarmNotOK – actual current severity, not masked by disable bits • <pv>AlarmDis – disable alarm for this PV • <pv>AlarmCnt – counts how many times in a row the PV severity has been greater than the latched severity value. Used to delay latch. • <pv>AlarmLtchCh – 1 when the latch changes • <pv>AlarmLtchTS – timestamp the last time the Latch value changed • <pv>AlarmStsCalc – using the “OUT” of this calcout allows unconnected PVs to be masked. • <pv>AlarmOKCalc – similarly to the above, but for unmasked value. (?) • This might look like a lot, but PVs in soft IOCs are pretty cheap. • Only 1 CA connection to each real PV.

22. Under the Hood – Alarm Log Sequence (sSsNApp/src directory) • The sequence <S>_<SS><N>_alarmLogSeq.st is created to make alarm logs for edm display. • The PV descriptions go there. • Called 3 times, to make 3 logs corresponding to current status after disables are applied, latched status and unmasked status. • Plus <s><Ss><N>Include.dbd and Makefile in support.

23. Under the Hood – Archive reqs (sSsNApp/srcCfg/arCfg directory) • <S>_<SS>_Alarm<N>_Summary.arReq to specify archiving the alarm logs.

24. edm Screens – Overall Summary • This one isn’t getting generated automatically yet.

25. edm Screens – Top-Level Summary (in sSsNApp/srcOpi directory) This circle includes current status, latch status, disable status and unmasked status. You can copy it to your own screen. Click for details (Alarm details or edm screen from system) Click to change disable status

26. edm Screens – Mid-Level Summary Click for alarm configuration details Here’s another circle you could copy to your own screen. Click for details (edm screen from system) Click to change disable status

27. edm Screens – Alarm Circle Legend

31. Alarm Logs

32. edm Screens – Alarm Log

33. edm Screens – Alarm Configuration (needs work)

34. iocBoot Directory • ics-ioc-linux-alrm soft IOC for common stuff (now alarmModes, later summaries of everything.) • iocBoot/<s>-<ss>-ioc-linux-alrm<n> directory • “_” in <n> are changed to “-”. • eg pps-ics-ioc-linux-alrm-chmk • contains <s>-<ss>-ioc-linux-alrm<n>.req (autosave req file for alarm parameters) and <s>-<ss>-ioc-linux-alrm<n>_st.cmd (startup command script.) • IOC name can also be specified by using the ioc=“<ioc name>” attribute.

35. Future Developments • Configuration screen needs work, and need different screens for different record types. • Create top-level summaries of everything. • Automatically generate a top-level screen. • Big-font screen for good visibility • Clear alarms when okay • More examples to find other needs and shortcomings (and, hopefully, because they’d be useful.) • Add enable/disable commands to logs. • Clear messages when okay.

36. More Future Developments • Interface with phone dialer & alarm annunciator • Maybe specify using a <command> directive. • Alarm priority levels. • Link with Oracle database for configuration. • Provide a way for Ops to do configuration and boot soft IOCs. • Document heartbeat and operating status definitions. • Define a “flatline” alarm. • Integrate and document .alhConfig conversion.