Instruments and Sensors on the Grid

1. EU FP6 Project Instruments and Sensors on the Grid The GridCC Project Gaetano Maron gaetano.maron@lnl.infn.it Istituto Nazionale di Fisica Nucleare – Laboratori di Legnaro Legnaro (PD), Italy

2. Outline • GRIDCC project: goals & objectives • Architecture overview • Strategic targets • Pilot applications • Focus on the new provided capabilities • Instrument Element • RMM-JMS as fast publishing system • Virtual Control Room • Conclusions

3. GRIDCC project Grid Enabled Remote Instrumentation with Distributed Control and Computation • It is a 3-years EU FP6 project started in September 2004 • Web site: www.gridcc.org

4. GRIDCC: Goals & Objectives 1. Remote control and monitoring of complex and distributed instrumentation 2. Tight integration between instrument grid and classical computational grid 3. Human interaction with Grids via Virtual Control Room (collaborative environment) 4. Enactment of complex workflows

5. Instruments Grid Computational Grid CE IE IE DATA Instrument Element CE Instrument Manager Information & Monitoring System Grid SE Problem Solver VCR Workflow VCR ES VCR VCR Collaborative Environment Execution Services GRIDCC: Architecture (I) • The GRIDCC services are Web Services compliant, according to the WS-I convention. • Any type of client WS-I compliant, can access the GRIDCC services without any specific software library. • X509 based security is used • When performances are required , a kerberos based mechanism is used to access IEs. Service Oriented Architecture – SOA

6. Instrument elements (IE) Storage Element (SE) Instrument elements (IE) Storage Element (SE) Instrument Element (IE) Storage Element (SE) Virtual Control Room Storage Element (SE) Storage Element (SE) Storage Element (SE) Grid Virtual Control Room Compute element (CE) Compute element (CE) Computing Element (CE) Execution Service WfMS WMS AS GRIDCC Architecture Information and Monitoring Services (IMS) Global Problem Solver Security Services Collaborative Services (CS) Web Service Interface WMS Work Management System WfMS Work Flow Mng System AS Agreement Service Information System (BDII) Broker

7. GridCC PermanentTest Bed VCR, IE WfMS VOMS IE, CE/CREAM,SE, LFC AS, KrbServer WMProxy VCR, SE, IE, BDII

8. Strategic plan: 3 main target areas Remote process control Accelerator control (Tele-) Biomedicine Robotics Automotive Electronic microscopes (Large-scale) scientific experiments High energy particle physics (Radio-) Telescopes GRIDCC Middleware Widely Sparse Instrumentation Power Grids Monitoring of the territory Monitoring of the sea Geo-hazard prediction Distributed laboratories Transportation monitoring Sensor network

9. GRIDCC pilot applications PowerGrid Particle Accelerator High Energy Physics CMS Meteorology Device Farm Geohazard Monitoring

10. DAQ IM DAQ RS DAQ IMS FB RB FF xdaq xdaq xdaq Pilot applications: CMS - I • The GridCC middleware has been deployed to control the run of the CMS (Compact Muon Solenoid), one of the four high energy experiments in LHC (Large Hadron Collider) at CERN laboratory. • CMS Magnet Test and Cosmic Challenge (MTCC), a milestone in the CMS construction, positively carried out. CMSDetector CMS Control Structure User Interface Top IE CSC IE Tracker IE HCAL IE DAQ IE RPC IE Trigger IE DT IE ECAL IE

11. Virtual Control Room Virtual Control Room Pilot applications: Power grid • GRIDCC deployed to monitor: • A 50kW generator • A 1 kw Photo-Voltaic array Power Grid. GRIDCC deployed to monitor a 50kW generator and a 1 kW Photo-Voltaic array Instrument Manager Power Grid V.O. Instrument Element Solar Panel ... Gas

12. Elettra Synchrotron Pilot applications: Remote Operation of an Accelerator

13. GRIDCC: other applications • Meteorology (Ensemble Limited Area Forecasting) • Weather forecasting system to detect hazardous weather • Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories • The Device Farm consists of a pool of Measurement Instruments for Telecommunication Experiments • Geo-hazards: Remote Operation of Geophysical Monitoring Network • The monitoring net will be characterized by different levels of activity: stand-by, pre-alert, alert, plus a control modality • An event worth to be monitored is for example the evolution of a “landslip”

14. Focus on the new capabilities provided by GRIDCC • Instrument Element • Complete integration of Instruments into SOA. Instruments become “services” accessible via standard Web Service • Complete integration into the computational and storage GRID with the embedded Sorage Element (SE) • High virtualization and abstraction of diverse physical devices • A very scalable and platform-independent framework (Java based) • Fast Data Publishing System • High-performance reliable multicast via RMM-JMS • Virtual Control Room • Allows effective human interactions with the grid • Provides a homogeneous collaborative environment

15. Instrument Element: Versatile I/O multichannel • The IE permits • SOA accessible operations to control and monitor the instruments (via VIGS), such as: • execute a command • get / set parameters • different data outputs: • data mover to/from a grid Storage Element (via SRM) • high bandwidth channel for data publishing (via IMS). • low bandwidth channel for logs, states etc. (via IMS). Logs, Errors, States, Monitors Data Subscribers VCR IMS Grid Interaction IE Commands Storage Element (SE) Storage Element (SE) VIGS Storage Element (SE) SRM ES Status Parameters Instruments VIGS Virtual Instrument Grid Service IMS Information and Monitor Service SRM Storage Resource Management

16. Service Oriented Access Web Services Instrument Element Any Protocol or physical connection Instrumentation Sensor Network Instrument Instrument Instrument Element: Instruments as Services • The Instrument Element (IE) is one of the novel elements of the GRIDCC architecture. • It offers a standard web service interface to integrate scientific and general purpose instruments and sensors within the grid. • An IE can control a set of instruments with the possibility to decide on which sub-set to operate. • The IE is the key decoupling element between real instruments and the grid

17. IMS IE VIGS SE Instruments Instrument Element: Instrument Virtualization Instrument Parameters • The IE grid instruments representation is basically based on: • providing the XML description of the instruments managed by the IE • providing the Finite State Machine that defines the dynamic behaviour of the control performed on the set of instruments managed by the IE • simple grid accessible operations to control and monitor the instruments like execute a command, get or set a parameter or a set of parameters of the instrument (VIGS). • providing different data acquisition output paths ranging from the accessing of a Grid Storage Element (SRM/SE) to a very efficient communication channel when an high bandwidth output stream is required (via IMS). Attributes Control Mode XML Based Language • : Perform a measure

18. Access Control Manager Data Flow State Flow Error Flow Monitor Flow Control Flow Instrument Element Architecture IMS The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments. IMS Resource Service SRM/SE GridFTP SE Instrument Element Problem Solver VIGS Data Publisher IMS Instrument Manager IMS Proxy Control Manager Data Collector Control Manager Event Processor FSM Engine Instrument Protocols Input Manager Resource Proxy Real Instruments

19. IE Instrument Instrument Instrument Instrument Instrument Element Framework Characteristics Resource Service IMS • The IE framework is modular, fully customizable and can be adapted to different environments and different type of instruments to be controlled. • Customizable plug-in modules are used to interface the framework with the communication protocol (both hardware and software) used by the instruments. • the dynamic behaviour of the control that has to be performed is programmable via the built in Finite State Machine (FSM). • relatively fast control feedback loops can be provided using the customizable built in Event Processor (EP). • Error repairing actions due to malfunctioning of the instruments can even be handled by EP or by the Local Problem Solver (LPS). Data Mover Local Problem Solver VIGS Access Control Manager Instrument Manager Instrument Manager Data Collector IMS Proxy Control Manager Control Manager Event Processor FSM Engine Input Manager Resource Proxy

20. Access Control Manager Instrument Manager IMS IMS Resource Service SRM/SE GridFTP SE Instrument Element Problem Solver Data Publisher IMS VIGS Instrument Manager Instrument Manager IMS Proxy Control Manager Data Collector

21. Instruments Instrument Manager Control Manager IMS Proxy Event Processor FSM Engine Data Collector Input Manager Resource Proxy Instrument Manager Plug-in modules to interface to the instruments Customizable Control Flow Data Flow Monitor Flow State Flow Error Flow • IM is composed by 3 main components: • - Control Manager: • - Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs,errors/state/log/monitor messages. • - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability • - FSM. A finite state machine is implemented • - Resource Proxy. It handles all the outgoing connections with the resources. • Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen. • - IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS

22. Data Subscribers (Monitor, data analysis, etc.) Access Control Manager Control Manager Event Processor FSM Engine Instrument Instrument Instrument Input Manager Resource Proxy Instrument DB persistency Data Collector RMM-JMS Resource Service IMS Data Mover enable/disable data stream Local Problem Solver Instrument Element VIGS IE File System Instrument Manager IMS Proxy Control Manager Data Collector • Status • data collection • data publishing on RMM-JMS • data on file in progress • data on DB to be started • sync commands (enable/disable • done trough the “set param” of the • IE façade (VIGS) data stream

23. Access Control Manager Resource Service IMS IMS Resource Service SRM/SE GridFTP SE Resource Service Instrument Element Problem Solver Data Publisher IMS VIGS Instrument Manager IMS Proxy Control Manager Data Collector

24. Resource Service Architecture Partition/Configuration retrieve methods Discovery Manager Available Resources Partition and Lock setting methods Subscribe Manager Partition Definitions RS Data Bases • The Resource Service (RS) handles all the resources of an IE and manages their partition (if any). • A resource can be any hardware or software component involved in the IE (instruments, Instrument Managers, IMS components) • RS stores the configuration data of the resources and download them to resource target when necessary • Resources can be discovered, allocated and queried. • It is the responsibility of the RS to check resource availability and contention with other active partitions when a resource is allocated for use. • A periodic scan of the registered resources keeps the configuration database up to date. Configuration setting methods Partition&Lock Manager Configuration Definitions Discovery methods Configuration Manager

25. Resources Discovery • Technologyusedhttp://www.jxta.org/

26. Access Control Manager Information and Monitor Service IMS IMS Resource Service SRM/SE GridFTP IMS SE Instrument Element Problem Solver Data Publisher IMS VIGS Instrument Manager IMS Proxy Control Manager Data Collector

27. Information and Monitor System (IMS) IMS Proxy • The Information and Monitor Service (IMS) collects messages and monitor data coming from the IMS Proxy of the IMs. • The messages are catalogued according to their type, severity level and timestamp. Data can be provided in numeric formats, histograms, tables and other forms. • The IMS collects and organizes the incoming information either in a database or a pub/sub system or both. It publishes the data to subscribers. These subscribers can register for specific messages categorized by a number of selection criteria, such as timestamp, information source and severity level. Publish / Subscrib IMS Proxy PERSISTENCY SUBSCRIBERS SUBSCRIBERS IMS Proxy

28. IE IMS Fast Data Publishing: RMM-JMS Same data are sent to several subscribers. Multicast protocols can have a benefic impact on performances IE Data Producer Data Subscribers • JMS provides a standard set of • APIs for the communication • Many commercial and academic • JMS implementations • both in C/C++ and Java • (NaradaBrokering, Sun, IBM) • GRIDCC (IBM Haifa lab) has • implemented a Reliable Multicast • protocol (RMM) JMS compliant • RMM-JMS works within a LAN • but an efficient bridge technology • has been developed to allow • inter-LAN multicast communication

29. Access Control Manager Problem Solver IMS IMS Resource Service SRM/SE GridFTP SE Instrument Element Problem Solver Problem Solver Data Publisher IMS VIGS Instrument Manager IMS Proxy Control Manager Data Collector

30. Instrument Manager Instrument Manager Instrument Manager Instrument Manager IMS Proxy IMS Proxy IMS Proxy IMS Proxy Control Manager Control Manager Control Manager Control Manager Problem Solver Step 3 On-line information can be analyzed in order to detect possible malfunctions Step 1 The control manager can perform an autonomous recovery action where the cost for the determination it is not so heavy . Problem Solver Pub/Sub On Line Analisys DB Data Mining Tools State Flow Error Flow Monitor Flow Algorithms evaluations : Rule Induction, Tree, Functions, Lazy, Clusters and Associative Step 2 Persistent information can be analyzed in order to extract knowledge

31. Access Control Manager Embedded SRM compliant Grid Storage Element (SE) Resource Service IMS SRM SE Backend GRID accessibile data Local Problem Solver Instrument Element VIGS IE File System Instrument Manager • Status: installed and running • SRM 2.2 • STORM implemetation (INFN) • Bestman implementation (Berkeley)) • XFS IE file system

32. Custom Board Xilink Virtex IV Grid Custom Electronics FPGA PPC 405 JavaVM Web Service GridCC IM Linux Instrument Element: Scalable on embedded systems Instrument Manager on a chip … … towards a grid on a chip 1 Gbps Ethernet IE Instrument Manager Custom Logic Standalone Axis Montavista www.montavista.com USER INTERFACE JamVM http://jamvm.sourceforge.net

33. IE technologies • Web Service compliant (WS-I) • Tomcat + Axis (and Java) and Axis standalone are the main technologies of the IE • All the services are deployed on a single or multiple instances of Tomcat, according to the needs of the application • Message oriented middleware (Pub/Sub) is based on the Java Messaging System (JMS). The following implementations are used in the project • Sun • Narada Brokering • RMM - JMS (GridCC IBM) • MySQL and Oracle are used as Data Base for the RS • SRM interface version 2.2 used

34. Grid Virtual Control Room • The VCR is a collaborative web portal • Enable the user to run, aggregate and display plug-ins that act as “mini applications”

35. Conclusion - I • The GridCC project is integrating instruments into the “classic” computational/storage Grids. • Novel concepts introduced by GridCC are: • The Instrument Element, allowing a virtualisation of the instruments to be controlled and their insertion in a Grid • The Virtual Control Room, providing an highly interactive environment with IEs/CEs/SEs. VCR even provides cooperative tools to allow the cooperation (logbook, chat, videoconf., etc.) between remote users • hard (reservation of IE) and soft (statistical prediction) guarantees of the IE’s methods execution times • Fast Data Publication via a Message Oriented middleware (RMM-JMS) to distribute data and information from an IE to the world wide Grid. • The IE is highly customizable and can be adapted in different environments.

36. Conclusion - II • The IE can be shrunk down into a chip allowing grid enabled embedded control of the instrumentation • Several heterogeneous pilot applications are deploying and running the IE middleware • We support and encourage the adoption of our middleware in other projects/experiments.

37. Thank you for your attention • Any Questions?

38. Spare

39. http://gladgw:2002/InstrumentElementKeeper/ da dove vedere la lista degli strumenti http://gladgw.lnl.infn.it:2002/SeedsFinder/ da dove scaricare l’IE da web-start Tecnologie usate http://www.jxta.org/

40. UniUD GridCC Scalability Demo Imperial Brunel VCR Monitor GUI Elettra INFN IASA IBM GRNet Network Core Machine CNIT Instrument Element IMAA Network Information Provider

42. 2 107 electronics channels 40 MHz 100 Hz Pilot applications: CMS - II • O(104 ) distributed Objects to be • controlled • configured • monitored • On-line diagnostics and problem solving capability • Highly interactive system (human reaction time - fractions of second) • World Wide distributed monitor and control CMSDetector

43. Data Consumer Fast Data Publishing Fast Data Display IE VCR Control Panel IE Existing Grid Elements Control and Status WorkFlow Mng. System IE Computing Element (CE) Storage Element (SE) Log Persistency Log Display Information and Monitor System Instrument Element – Interconnections II

44. References • - WS-I, http://www.ws-i.org • - Java Message Service (JMS), http://java.sun.com/products/jms • - RMM-JSM, http://www.haifa.ibm.com/projects/software/rmsdk/gridcc.html • - gLite, http://glite.web.cern.ch/glite/ • - SRM, http://sdm.lbl.gov/srm-wg • - StoRM, http://grid-it.cnaf.infn.it/storm • - Montavista, http://www.montavista.com • - JamVM, http://jamvm.sourceforge.net

45. Acknowledgments The INFN team working in Legnaro, in collaboration with CERN at Geneva: • E. Frizziero • M. Gulmini • F. Lelli • G. Maron • A. Petrucci • S. Traldi

46. IE IE Storage Element (SE) Storage Element (SE) Storage Element (SE) Compute element (CE) Compute element (CE) Computing Element (CE) Execution Service WfMS WMS AS Instrument Element: Versatile I/O multichannel Data Subscribers Virtual Control Room DP Commands VIGS SE Grid Interaction Status IMS Logs, Errors, States, Monitors • A Virtual Instrument Grid Service (VIGS) interface has been defined. The VIGS provides a Web Service acccess to the instrumentation • The Data Publishing (DP) channel disseminates the acquired information to the data subscribers • A SRM/SE interface is provided to allow file movement with other Grid Element, like SE, CE and other IEs. • Information messages (logs, errors, etc.) are published from the IE (via IMS) VIGS Virtual Instrument Grid Service IMS Information and Monitor Service SE Storage Element DP Data Publishing

47. Fast Data Publishing: Some results Message Rate: Case many-to-one • 32 Dual Xeon 2.4GHz 1.5GB RAM machines, 1 GB Ethernet switch • At most 1 publisher, subscriber, or broker- (Sun MQ3.6) per machine • No message lost • RMM throughput: 75-90 Mbytes/sec. (for 5 and more publishers)

48. Architecture: New “instruments grid”