Alarm Training Making the transition from alh to CSS Alarm Tools

1. Alarm TrainingMaking the transition from alh to CSS Alarm Tools EPICS Collaboration Meeting Aix-en-Provence June 1st , 2010 Matthias Clausen - DESY

2. Alarm Training • Record Processing • Monitor Types • Setting Alarms • Alarm Handling

3. EPICS Record Processing EPICS (Experimental Physics and Industrial Control System) Client (Configuration- and Operator Interfaces) Channel Access IP based Communication Protocol I/O Controller is independent from clients I/O Controller (control loops, Sequence programs) I/O System (Field bus) P T

4. EPICS Information exchange Client initiating basic requstes: get/ set/ monitor • Client • Register for Changes • Display-Deadband • Archive-Deadband • Alarm-Status Sending request via Channel Access Register Client for chanegs in specified record for Alarm changes Whenever Displa/ Archive/ Alarm-Deadband/ Status changes -> check whether clients have been registered. If true: -> send updated data P T

5. get/ set register for monitors New/ Changed Alarm Limits Alarm/Status P T EPICS default: focus on ‘requesting’ data-> not ‘expecting’ data How to stay ‘in synch’ with alarm deadbands/ status in IOC? Client initiating basic requstes: get/ set/ monitor

6. register for monitors New/ Changed Alarm Limits Alarm/Status P T Introducing a Message based Protocol: JMS(keeping Channel Access for ‘basic’ monitors) JMS Basic: get/ set/ monitor InterConnection Server Accepting any alarms from IOC get/ set

7. register for monitors New/ Changed Alarm Limits Alarm/Status P T JMS: For Alarms and all Log-based Messages JMS Basic: get/ set/ monitor InterConnection Server get/ set SysLog-Messages PutLog-Messages SNL-Log-Messages

8. register for monitors P T Existing Log Server Implementations Sys-Log Message Server Put-Log Message Server Basic: get/ set/ monitor get/ set PutLog-Messages SysLog-Messages

9. register for monitors New/ Changed Alarm Limits Alarm/Status P T JMS does not add more levels of complexity(just new ones …) JMS Basic: get/ set/ monitor InterConnection Server get/ set SysLog-Messages PutLog-Messages SNL-Log-Messages

10. How to implement the new Logging Mechanism • Using the existing Alarm Hook in Record Processing • Using the existing Hooks in System Logging • Using the existing Hooks in (ca)Put Logging • Using the new Function in SNL Logging

11. Alarm Logging • Changes in iocCore for Alarm Logging • very small hook in recGbl  recGblResetAlarms • /* Hook Routines */ • RECGBL_ALARM_HOOK_ROUTINE recGblAlarmHook = NULL; • ....... • loading and starting the alarm logging software redefines recGblAlarmHook • ....... • if(sevr!=nsev || stat!=nsta) { • /* HOOK for alarm logging */ • if (recGblAlarmHook) (*recGblAlarmHook)(pdbc, stat, sevr); • .......

12. Alarm Logging The special queue has sequential input .... add Hook Routine replace and parallel access if space is rare and the record is stored in the overflow area. SCAN context implenented as ring buffer Control task Message tasks Beacon task Port B Port A

13. EPICS Channel-Access-Put-Logging asTrapWriteRegisterListener(caPutLog2Array) caPutLog2Array (...) (caller context) caPutLog2Q (...) (interrupt context) caPutLogTask () register Hook Routine for caPut operation write caPut-Message into Array timeout copy into Queue index queue send Message to Network

14. SNL Log • ‘Just’ a new supported function in the snl executive. • Code example: • when(TV001zu){ • snlMsgLog( SEVR_INFO, "<Start> TV001 has been closed"); • } state Auf50 • Sent to the log table in CSS. • Different severy types can be set -> used to color-code the messages

15. EPICS IOC-Logging Overview Alarms caPut SysLog SNL-Log ioc log control Command Reply Alarm caPutLog sysMsg snlMsg Beacon Alarm ICS ICS ICS

16. Alarm Table

17. Looking back into the Archive (24h)

18. Selecting the Time Span

19. Any Message in the System

20. Log Messages (System, Put, SNL)

21. Properties View (default Eclipse view)

22. Alarm System - Alarm Tree view • Shows the current status of the alarm state from the IOC • Delete and create records and subcomponents by context menu • Configuration-Changes are stored in the LDAP server (or in XML file) • Alarm status is propagated to root component • Property view to display and edit tree items

23. Demo Alarm Views • Changing / setting alarm Limits • Severity Settings • Different kind of Displays

24. How to get there without JMS? • The internal Alarm Model in CSS (DESY) has been modified to accept Alarm-Messages from two different sources: • JMS • DAL (Channel Access) • (currently only available in the development environmentforeseen for the next release)

25. How to display an Alarm Tree without LDAP? • Use the new alternative to configure your LDAP Tree from an XML file • Create XML file by Hand • Create XML file from DCT (next release) • Create XML file from ‘your’ database creation tool • Use LDAP Tree to manually configure your tree and save your configuration from tree ‘save as’

26. How to integrate IOC’s without iocLogClient-Support? • A new headless application will handle this: • Reading an XML config file • Use channel access to register for Alarm Changes • Send alarm changes to JMS

27. The transition Plan: • Start using the ‘DAL’ option for Alarm Tables and the Alarm Tree • Get your configuration file e.g. from DCT • Get the hierarchy visualized in the Alarm Tree • Get all configured alarms in the Alarm Table • Install JMS Servers • Immediately get your CSS log messages into you CSS Alarm views • Use dal2jms to send Alarm State changes to JMS • Install iocLogClient-Support on your IOCs • Install interConnection-Server(s) • Get ALL your messages (alarm and log) from your IOCs • Install jms2Ora (rdb) to store your messages

28. Alarm System - Acknowledgement CSS Instance Acknowledge Alarm message Ack. Message (JMS) Update Ack Persistant Store (LDAP) JMS Server Ack Ack Ack Ack CSS Instance CSS Instance CSS Instance CSS Instance

29. Alarm System - Alarm Management System (AMS) Alarm Message (JMS) CSS Alarm Configu- rator Filter Manager Filter Write Configuration Read configuration Action DB JMS JMS JMS SMS Connector Voice Mail Connector Mail Connector Voice Mail SMS Mail

30. Alarm Management System • Slides from Markus Möller

31. T ALARM T T T T T T Derby Db Derby Db T_AMS_CON_SMS T_AMS_CON_VOICEMAIL T_AMS_CON_MAIL T_AMS_MESSAGEMINDER T_AMS_DISTRIBUTE T_AMS_CON_JMS Filter-Konfiguration Filter-Konfiguration AMS - Overview Message Sources ActiveMQ JMS-Server DepartmentDecision (FilterManager) SmsConnector EMailConnector JmsConnector ActiveMQ JMS-Server MessageMinder VoicemailConnector Distributor

32. AMS - Applications • DepartmentDecision (FilterManager) Compares the incoming message with the filter configuration • MessageMinder Reduces message storms • Distributor Creates the alarm messages and sends them to the responsible Connector • Connector Sends the message to the user via SMS e-Mail telephone call JMS message producer to a user defined JMS topic

33. User – Configuration (GUI)

34. User – Configuration • Name - The name of the user • Status code - To deactivate a user or a group • Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) • Confirm code - To confirm a alarm chain • User is active - Only active users receive a alarm message • E-Mail - E-Mail address of the user • SMS number - Number of the user‘s cell phone • Voicemail number - Number that is used for the telephone call • Prefered alarming type

35. Group – Configuration (GUI)

36. Group – Configuration • Name - The name of the group • List of group members - Every member of a group can be set active or not active - Only active member of a group receive messages • Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) • Minimale Anzahl aktiver Mitglieder - Minimum number of active users in this group • Wartezeit bis Rückmeldung - Number of seconds to wait for an alarm chain answer • Alarmgruppe aktiv - Only the member of an active group receive messages

37. Condition – Configuration (GUI)

38. Condition – Configuration • Name - The name of the condition • Compare Key - Name of the message property that is compared by this condition • Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) • Operator - Type of operator that is used to compare the value • Description - Description for this condition • Compare Value - The value that is used for comparison • Filtertype - Type of this filter (String, String Array, PV, Time based)

39. Filter – Configuration (GUI)

40. Filter – Configuration • Name - The name of the condition • List of receivers - Name of the receiver - Type of action (SMS, Mail, Voicemail, JMS) • Rubrik (rubric, category) - Name of the configuration rubric (just for display puposes) • Operator - Type of operator that is used to compare the value • Templates - Message template (not used yet) • Compare Value - The value that is used for comparison • Default Message - Alarm message that will be send to the users • Filterconditions - The configuration / combination of the conditions

41. Questions?

42. Active ICS Connection OverviewICS: InterConnection Server (between IOC and JMS)

43. Disabling Messages (for Testing/ Reboot …)

45. Alarm system - Structure Alarm / Log message Sources EPICS IOC D3 PCM CSS Instance Updated from IC Archive DB Persistent Store (LDAP) JMS Server Alarm Management System CSS Alarm Tools (Views, Con- figuration, …) Message Table Message Archive SMS Mail AMS Configuration Alarm Tree

46. Message sources EPICS IOC • EPICS IOC and D3 PCM send alarm messages in special format • Interconnection Server (EPICS) and D3 Alarms (D3) translates alarm messages in JMS format • CSS uses log4j and sends log messages in JMS format • Generic message system for alarm messages • Easy to add other sources D3 PCM Special Format Special Format JMS Communication Interconnection Server D3 Alarm Reader Other Sources JMS Server (Active MQ) CSS Instance

47. Alarm System - AMS Filter • Filter: • Checks if the filter matches • Creates a new message with the relevant information of the alarm message • Forwards the message to an action • Filter condition: • A Filter is a combination of filter conditions • Filter conditions can be connected with AND and OR • Available condition types are: Compare strings, Check current PV, Time based condition, …

48. Alarm System - AMS operators and groups • Operators: • Receive alarm messages via mail, sms, … • Status active or inactive can be set • PIN Code to acknowledge alarm messages • Groups: • Operators responsible for specific facilities • Defines priority who should be informed first, second, … • Maximum delay for acknowledgment of alarm messages

49. Logging : 2X Log Server Verbindung • CSS mit direct Channel Access (DAL) connection • As a transition to JMS • Demo DAL • Demo JMS • ICS und JMS Folien • AMS Folien • Config in CSS Log Table  Topics

50. Application Development Configuration of I/O Hardware And Process Controls Integrating new I/O devices Hardware Engineer Process Engineer Programmer Prepare new IOC Load Image Get driver from EPICS site Configure Profibus Configure Database Profibus Configuration Tool Database Creation Tool Write text file to load directory Load IOC with -Operating System -EPICS core -Database