1 / 43

Monitoring and remote control of scientific instrumentation through the Grid

Monitoring and remote control of scientific instrumentation through the Grid C. Vuerli (1,2) and G. Taffoni (1,2) (1) INAF – Informative Systems Unit (2) INAF – Astronomical Observatory of Trieste. The Goal.

kemp
Download Presentation

Monitoring and remote control of scientific instrumentation through the Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Monitoring and remote control of scientific instrumentation through the Grid C. Vuerli (1,2) and G. Taffoni (1,2) (1) INAF – Informative Systems Unit(2) INAF – Astronomical Observatory of Trieste Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 1

  2. The Goal • Gridification of Telescopes and attached astronomical instrumentation to allow remote monitoring and control in Quasi-Real Time • Step 1. Remote monitoring of telemetry and scientific data under acquisition • Step 2. Remote control: the astronomer interacts with night assistant by remotely sending commands and receiving feedback Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 2

  3. The Origin: The GRID.IT Project • GRID.IT: National multi-disciplinary project to validate the Italian Grid infrastructure • INAF participates with 3 applications • Access to Astronomical Databases via Grid • Porting on Grid the VST Pipelines • Monitoring/Control of astronomical instrumentation Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 3

  4. Astro DBs in Grid: drivers • “Customer whishes”: • Transparent access to data sources (DBs); • Transparent access to computing resources; • Complex workflows; • IVOA standards for data access; • GRID computing and storing resources; Integration of the VObs with Grid technology is of key importance to provide users with proper computing power Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 4

  5. Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 5

  6. Our Grid • It is the Grid middleware that we used in the framework of GRID.IT Project • Set up by INFN (National Institute of Nuclear Physics) to run jobs of the LHC (Large Hadron Collider) experiments (CERN, Geneva) • LCG (LHC Computing Grid) is based on Globus 2.4  native computing and data storage resources • Tests are now in progress to migrate LCG to gLite (WebServices enabled) No native access to non-computational resources G-DSE: a low level integration frameworkIn Globus and not on Globus Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 6

  7. Control systems in Grid • Grid TeleControl Protocol (GTCP) • JavaServices + GT4 container • Xport (X-ray Cristallography) • WebServices • GridCC • Web Services G-ICS: a low level integration framework on a production grid Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 7

  8. Toward a Generalized Grid Element • To have the Generalised Grid Element as a new Grid element embedded in its middleware some components of it must be revised and modified • The new generalised element (and all of its specialisations) enhances the CE in terms of functionalities and intrinsic characteristics • These intrinsic characteristics are expressed by means of a set of new meta-data, that enhance the set of meta-data characterising traditional CE and a SE elements • The new Grid middleware has to be able to • Recognise this new set of meta-data • Ingest them in its Information System • Publish them to make users operating at client machines able to discover these new resources • Some components of the middleware have then to be revised and modified. They are those related with the Grid Information System, namely the BDII (Berkeley Database Information Index), the GRIS (Grid Resource Information Server), the MDS (Monitoring and Discovery Service) and the RB (Resource Broker) Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 8

  9. Recycling G-DSE to enable G-ICS • Provide a suitable formal definition of a Grid Abstract Computing Machine using Formal specification language • Review the architectural definition of a Data Source Engine (DSE), to build an Instrument Control System • Provide ICS integrated in the existing Grid Resource Layer Framework (G-ICS) • Extend GIS capabilities to understand and handle monitoring and control requests to ICS • Extend GIS capabilities to describe ICS Resources • Provide a new Manager for ICS integrated in Globus • Extend semantic and provide it to RSL to use G-ICS • Etc… Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 9

  10. ICSManger ICS plug-in ICS specific driver ICSProcess Instrument Control System ICS G-DSE and G-ICS GIS GRAM gatekeeper MDS GRIS ldif JobManger QueryManger Ldap Scheduler p-in query plug-in Grid Providers (snmp) Query DB specific driver Pbs/LFS JobProcess QueryProcess RDBMS RDBMS Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 10

  11. LCG-2: DB + ICS enabled middleware Application level services User interfaces Applications EU DataGrid +Glite + G-DSE + G-ICS “Collective” services App monitoring system Resource Broker Data management VDT (Condor, Globus, GLUE) + XMLDB + XMLICS “Basic” services User access Information system Information schema Data transfer Security PBS, Condor, LSF,… MySQL Ora, etc. NFS, … System software Scientific Linux File system RDBMS Local scheduler ICS Operating system HPSS, CASTOR… Hardware Instruments Computing cluster Network resources Data storage Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 11

  12. From G-DSE to QE • With the “G-DSE enabled” collective and resource Grid layers, the new QE Grid Element is introduced • QE is a CE able to manage Queries and to connect RDBMS resources via Grid • It speaks RSL to interact with Grid services, VOQL when exchanging data with RDBMS resources and processing query requests coming from UIs (Users) Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 12

  13. From G-ICS to IE • With the “G-ICS enabled” collective and resource Grid layers, the new IE of the Grid is introduced • IE is a CE able to manage, monitor and control requests to interact with ICS resources via Grid • It speaks RSL to interact with Grid services, and ICSL (Instrument Control System Language) when connecting an ICS Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 13

  14. From G-ICS to IE • What we need is: • ICSP: a ICS Protocol to make IE and ICS able to communicate each other • ICSL: a ICS Language used by IE to speak with ICS according to protocol ICSP • ICSC: an Instrument Control System Connector, being it a driver able to • speak ICSL and understand ICSP-compliant requests and map them in requests to the specific ICS sitting behind it • handle answers to these requests Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 14

  15. From G-ICS to IE Custom ICS 1 • Mapping the DSE schema into ICS: • The ODBC Driver translates standard SQL in the SQL (extensions included) of the specific DBMS to be contacted • Similarly, the ICSC Driver translates standard ICSL in the specific ICSL (extensions included) of the ICS to be contacted • Extensions retrieved from metadata repository via MDS ICSCDriver ODBCDriver ICSC clientIE ODBC clientQE Oracle ICS 2 ICS 3 Odbclang Mysql ICSL ICS 4 Postgres Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 15

  16. ODBC Manager ICSC Manager DSE Instance Man ICS Instance Man ODBC Driver ICSC Driver DSE Instance ICS Instance DSE ICS Tel CS User DB Synchro Synchro CS Internal DB ICS Grid Access Enabled Gridinfosystem Grid Instrument Control System GRAM Protocol Job Manager MDS Local Resource Manager snmpd LDAP snmpd GANGLIA Worker Node Meta Machine Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 16

  17. Long Term Goal: Grid and VObs External IVOA Resource (Registry) VOQL Data not available in Grid Grid Information and Brokering System RSL(VOQL) RSL(VOQL) UI QE Data found in Grid Data RSL MetaData RSL VOQL QE Outside Grid (VObs) QE VOQL VOQL VOQL AstroDB Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 17 AstroDB AstroDB

  18. Long Term Goal: Grid and VObs MetaData Grid Information and Brokering System Mon/Ctrl requests RSL(VOQL) UI QE Data RSL RSL IE IE ICSL ICSL ICSC ICSC Instrument Control System Instrument Control System Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 18

  19. In Summary: What is the IE? • The IE inherits from the CE, so it has all the characteristics of a classic computing element • The IE, moreover, makes possible to • Publish on the Grid any kind of scientific instrumentation. Telescopes and related astronomical instrumentation are just an example of it • Share on the Grid infrastructure any kind of scientific instrumentation • Get access to any kind of scientific instrumentation shared on the Grid infrastructure • Perform remote monitoring sessions in which the Grid technology only is used • Perform remote control sessions in which the Grid technology only is used Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 19

  20. Current Status • Institutes that collaborated in the framework of GRID.IT Project • INAF Trieste • IE project coordination and development/test • G-DSE development/test • INAF Padova • G-DSE development/test • INAF Capodimonte (Naples) • IE development/test • INFN-CNAF • G-DSE development/test Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 20

  21. Current Status • We are going on through subsequent refinement steps • For the time being monitoring only implemented • ICS not fully implemented on Grid yet • At present the worker node (at Telescope Site) and the IE speak HTML. IE does not use G-ICS yet Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 21

  22. Current Status • Two test-bed ICS identified (at Trieste and Naples observatories) • Tests successfully carried out on LCG implementation of the Grid middleware (no G-ICS extension) • GRID.IT project is now ending (31 Dec 05) • New projects proposed and funds requested to carry out the implementation of the Instrument Element Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 22

  23. Current Status • A FP6 Strep proposal has been presented to EU Commission in September 2005 • Answer expected by EU Commission by mid December • The focus of the FP6 Strep is on • Query Element • Instrument Element • Integrated Working Environment System • Partners • INAF-Trieste (coordinator) (IT) • Astronomical Institute of the Academy of Sciences (CZ) • Cosylab, Ljubljana (SME) (SI) • Elettra Sincrotrone Trieste (IT) • Instituto de Astrofisica de Canarias (ES) • INFN (IT) • Poznan Supercomputing and Networking Centre (PL) • XLAB Ljubljana (SME) (SI) Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 23

  24. Test-bed and the first prototype of UIF First prototype of UIF written in Java The Celestron C-14 located at theBasovizza branch station of INAF-OATs Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 24

  25. The final goal: An Integrated System Synchrotron DBMS SE IE QE Grid-UI IE The GRID IE DBMS QE CE CE Grid-GUI SE Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 25

  26. End of Presentation Thank you for your attention Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 26

  27. Ancestors • Remote observations project with ESO. Remote observations with NTT (La Silla, Chile) from Trieste  WSS designed taking into account this successful experiment • Experimental Remot Project (EU funded) with the Telescopio Nazionale Galileo • Dynacore Project (EU funded) again with TNG  Both based on Corba technology Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 27

  28. Why the Grid? • Former projects based on (custom) technological solutions • The GRID offers • A robust and reliable technological infrastructure to connect users and observing structures • Secure and controlled access to Resources by means of Grid authentication and authorization mechanisms • VOs may be set up by observing structures. Users may easily and dynamically join and leave VOs Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 28

  29. Why the Grid? • Usage of observing structures and their instrumentation is regulated by quite rigid time-allocation schedules • In some cases (unexpected technical problems, weather conditions) force changes in the time schedule • Remote control via Grid makes possible to exploit such events and optimize the usage of resources Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 29

  30. We want to use the GRID for… • Monitoring and controlling of instrumentation • Transparent access to telescope control systems; • Monitoring of telescope activity; • Complex workflows (data acquisition/reduction); • Computing and DBMS resources; No native access to non computational resources from our Grid! Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 30

  31. What is IVOA The International Virtual Observatory Alliance (IVOA) was formed in June 2002 with a mission to facilitate the international coordination and collaboration necessary for the development and deployment of the tools, systems and organizational structures necessary to enable the international utilization of astronomical archives as an integrated and interoperating VObs Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 31

  32. Application level services User interfaces Applications EU DataGrid “Collective” services App monitoring system Resource Broker Data management VDT (Condor, Globus, GLUE) “Basic” services User access Information system Information schema Data transfer Security PBS, Condor, LSF,… System software NFS, … RedHat Linux Operating system File system Local scheduler HPSS, CASTOR… Hardware Computing cluster Network resources Data storage The missing point(from Grid to IVOA point of view) No native DB access from our Grid! EGEE is a computational and data grid (European Standards) Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 32

  33. Possible Solutions • Different approaches on LCG/EGEE • OGSA • WS-RF • DataGrid spitfire • EGEE Fireman • Etc…. G-DSE: a low level integration frameworkIn Globus and not on Globus Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 33

  34. G-DSE: the integration steps • Provide a suitable formal definition of a Grid Abstract Computing Machine using Formal specification language; • Provide a suitable architectural definition of a Data Source Engine (DSE), • Provide DSEs integrated in the existing Grid Resource Layer Framework (G-DSE), • Extend GIS capabilities to understand a RDBMS query processor, DB memory model, RDBMS OS; • Extend GIS capabilities to describe RDBMS DBs Resource; • Provide a new Manager for DBs integrated in Globus • Provide new semantic for RSL to use G-DSE • Etc… Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 34

  35. Globus G-DSE integration GIS GRAM gatekeeper MDS GRIS ldif JobManger QueryManger Ldap Scheduler p-in query plug-in Grid Providers (snmp) Query DB specific driver Pbs/LFS JobProcess QueryProcess RDBMS RDBMS Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 35

  36. Application level services User interfaces Applications EU DataGrid +Glite + G-DSE Resource “Collective” services App monitoring system Resource Broker Data management VDT (Condor, Globus, GLUE) + XMLDB “Basic” services User access Information system Information schema Data transfer Security PBS, Condor, LSF,… MySQL Ora, etc. System software NFS, … Scientific Linux Local scheduler RDBMS Operating system File system HPSS, CASTOR… Hardware Computing cluster Network resources Data storage LCG-2: DB middleware Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 36

  37. Data Source Engine Grid Access Enabled Gridinfosystem Grid Data Source Engine GRAM Protocol Job Manager MDS ODBC Manager Local Resource Manager DSE Instance Man snmpd ODBC Driver LDAP DSE Instance snmpd GANGLIA Worker Node DSE Meta Machine User DB Internal DB Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 37

  38. From G-DSE to QE • New Calculus capabilities for GridNode (Inference, Query, Reasoning) like now binary code execution. • New Fabric-Element for Computing (RDBMS, REASONING SYSTEM) like now LSF, PBS etc • Prototype for new Information System through standard SemanticWeb specifications from W3C. The query element (QE) Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 38

  39. From G-ICS to IE • New “Calculus” capabilities for GridNode • New Fabric-Element: Scientific Instrumentation and related Control Systems • New Information System: extended for monitoring and controlling instrumentations. The instrument element (IE) Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 39

  40. Integration in Grid IE GIS: Get info from site GIS Publish info to UI/RB RSL/LDAP IE Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 40

  41. The final goal: An Integrated System • The IES is an “integrated working environment system” aiming at making available to scientific, industrial and business communities an exhaustive, user-friendly working environment that hides to final user all technicalities in using the underlying technology (the Grid) and allowing users to carry out their tasks using a single technology. The IES encloses all capabilities of the Grid, both those traditional as well as those innovative • The IES federates together: computing, data storage, access to databases, monitoring/control of scientific instrumentation all of this accessed through easy-to-use user interfaces Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 41

  42. Local Model Celestron CM1400 Audio/video system Control system IE Telescope site SE Monitoring site UI QuantumGRID Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 42

  43. Global Model RB Site 2 Monitoring controlling Monitoring controlling UI Site n Site 1 Scientific Instruments and Sensors on the Grid – Melbourne – 5 December 2005 43

More Related