1 / 26

Synchrotron SOLEIL

Synchrotron SOLEIL. Data Format and Reduction. Majid OUNSY : Responsible for « High Level Applications » development. Agenda. Introduction Data Production Process Integrating Data Reduction in Soleil Software Architecture Conclusion. Introduction. Publication. Analysis. Storage.

albin
Download Presentation

Synchrotron SOLEIL

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Synchrotron SOLEIL Data Format and Reduction Majid OUNSY : Responsible for « High Level Applications » development

  2. Agenda Introduction Data Production Process Integrating Data Reduction in Soleil Software Architecture Conclusion

  3. Introduction Publication Analysis Storage Systèmes d’Acquisition Systèmes d’Acquisition Equipments Observable

  4. Data Analysis Applications Scientifiques d’analyse PyMCA EDNA Data Mining Data Visualisation Mambo, baladi Data Reduction Foxtrot, flamenco Data Retrieval twist Contrôle/Acquisition services Supervision Service GlobalSCREEN Sequencing Service Passerelle Scanning Service Salsa Expert System Drools FileStorage Service NeXus Archiving Service HDB/TDB User environnements MATLAB, CLI, etc .. Composants graphiques COMETE Common Data Model Layer NeXus files Tango system DataStorage

  5. Provide Users with Graphical components to Interact with their equipments Connect GUI viewers to Tango Control System View/Store acquired Data on Detectors Connect GUI viewers to Data Storage System Monitor/Preprocess Data during acquisition process Connect GUI viewers to Data Reduction Engine Data Production ProcessThe needs

  6. Keep The Graphical Framework Component based Use of the same framework for all graphical concerns (control/acquisition/storage) Assert Independance of GUI Components from their source of Data Connection of Data to GUI Components is done by external modules (DAO,CDM) Do not handle data processing in the GUI Components Provide users the ability to dinamically load data processing macros to be performed before connection to the GUI components (ImageJ) Data Production ProcessThe constraints

  7. Software Architecture Comete Widgets Data Connector ImageJ Analyis Data Connector CDM Storage Acquisition Process Data Connector Tango DAO Tango Bus Equipments Diffracto Device CCD Device Mono Device

  8. Widget Library does not have to know where plotted data is coming from Data is accessed thrue abstracted layers hiding the effective source Tango DAO to access control system Common Data Model to access stored data Data Connectors are there to make the glue between data access layer and presentation layer. Architecture Phylosophy

  9. Storage of the Experimental Data:Data Recording Java applications : ArchivingService TangoJavaATK ScanService SCADA configuration, monitoring, logging, system administration TANGO Software bus Device Data Recorder Device Device Device Device Equipment & subsystem devices libData Storage Process & calculation devices Hardware access devices • Provided by a TANGO device server : the Data Recorder DS NeXus File Control & Data Acquisition Systemof the Beam Line Local access point of the Beam Line • Each NeXus data files is: • locally recorded then automatically moved to the storage facility, • indexed in a Oracle DB using a set of metadata (inside the file) describing the experiment. Experimental Data StorageSystem

  10. Data Access Challenge

  11. NeXus Files choice : Is it Enough ? File retrieval File browsing SAXS Data Analysis Application • NeXus is a good and efficient storage format • Thanks to a unique API and a « SOLEIL standardized internal data organization », we could : • develop common software solutions • Decouple the development of Acquisition softwares from Data Analysis software NeXus Application Interface SOLEIL NeXus Files The COMETE library of data visualization components http://comete.sourceforge.net 11

  12. Data Analysis Application C SAXS Data Analysis Application Data Analysis Application B File retrieval File browsing NeXus Application Interface ESRF Files DESY Files SOLEIL NeXus Files NeXus Files choice : is it enough ? 12

  13. Data Analysis Data Analysis Data Analysis Application n°1 Application n°2 Application n°3 Common Data Access API SOLEIL NeXus plugin data format X data format Y plugin plugin The software architecture EDF SOLEIL Nexus netCDF

  14. Data Analysis Data Analysis Data Analysis Application n°1 Application n°2 Application n°3 Common Data Access API SOLEIL NeXus plugin data format X data format Y plugin plugin EDF SOLEIL NeXus netCDF What is the work to do ? Adapt each application to the Common DataAccess API Implement plugin for each data format

  15. What is the role of a plugin ? • Provide Data thanks to keywords • Implement the standard definition of these keywords Let scientists agree on keyword definitions Data Analysis Application Common Data Access API getDataItem(« pitch ») SOLEIL NeXus plugin Nexus File Look in dictionary Double getDataItem(« pitch ») { pitch = dictionary.get(« pitch ») return pitch *100; } Pitch ->experiment/instrument/d13/pitch Roll ->experiment/instrument/d13/Roll * * * *

  16. Data file access abstraction in collaboration with ANSTO (GumTree Data Model) Project initiate by ANSTO (http://gumtree.codehaus.org/GumTree+Data+Model/) The GDM API allow to create generic data analysis tools without need to known about experimental data files format(s) A set of data format plugins have to be developed by each facility so a generic application become able to read and process data files from these facility Current plugins: NetCDF (ANSTO), NeXus (Soleil, V1) GDM API is currently available in Java Common Data Access API

  17. SOLEIL/ANSTO Collaboration • Share the Common Data Access API and • the graphical components (COMETE and GumTree UI) made on top of the “GumTreeDataModel API” • the graphical applications made on top of these graphical components • the data analysis algorithms used in the GumTree and COMETE frameworks Collaboration planning

  18. How computer division views data analysis problems Data Reduction Challenge

  19. Data Preprocessing Elementary operations on data during their acquisition Ex : noise subtraction Solution : embed it in the Tango devices Data Reduction More complex operations on data already acquired and stored toqualify measure validity Four kinds of data analysis

  20. Data Postprocessing Data treatments forscientific exploitation On-line Data analysis Do the three phases in sequence: Acquisition Reduction Postprocessing With a possible feedback on data acquisition Four kinds of data analysis

  21. DATA REDUCTION

  22. Handling transparently (Black boxes) for users Data Access Calculation Processing Data Visualization Provide users with hooks (open boxes) to Define which data to operate on Implement algorithms for data reduction Graphically show the results The goal of data reduction environment

  23. The COMETE/ImageJ Framework Get Data Common Data Access API COMETE Framework Write Reduction algorithms Perform Calculations ImageJ Macro Language ImageJ Runtime

  24. NeXus : data storage format for images, intensities and contextual data Detected : I(Q) Ii, l IT Transmitted intensity Incoming intensity • Several images are collected for each given sample • Monitoring intensities are also collected for each image • A single NeXus file is generated for the whole sequence

  25. Foxtrot : Graphical Application (Java) for Data reduction and first analysis • Mask, 2D  1D Radial averaging, Frame averaging, Frame subtraction • Determination of Guinier parameters (I0, Rg) • Macro language

  26. Output from Foxtrot : NeXus file + ASCII files to comply with users needs

More Related