1 / 24

ICAT Integration at DLS.

ICAT Integration at DLS. Alun Ashton. What were the requirements?. Integrate with current business system Collect Data and Metadata relating to a proposal Security Long term storage of data (Multi?) Institutional repository Searchable Metadata

makara
Download Presentation

ICAT Integration at DLS.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ICAT Integration at DLS. Alun Ashton

  2. What were the requirements? • Integrate with current business system • Collect Data and Metadata relating to a proposal • Security • Long term storage of data • (Multi?) Institutional repository • Searchable Metadata • Potentially extendable with future e-science infrastructure • Scalable

  3. What were the restrictions? • Only relies on Beamline Network. • Must not prevent scientist collecting data. • Data is collected by putting files onto a UNIX file system. • Use e-Science research output. • Interest, vision, priorities and finances.

  4. What was the solution?Diamond, Modified by e-Science Atlas Data Store DataPortal Active Directory Diamond Proposal Web pages People DB SRB DLS ICAT DUO DUO Desk Data / metadata IKitten DArc and Storaged Nexus & Data GDA

  5. What was the solution?Diamond, Modified by e-Science Atlas Data Store DataPortal Active Directory Diamond Proposal Web pages People DB SRB DLS ICAT DUO DUO Desk Data / metadata IKitten DArc and Storaged Nexus & Data GDA

  6. What goes into the ICAT/Ikitten • Who • When • Where • ‘what’ • Abstract • Visit • Triggers I think moves the data to the iKitten DLS ICAT DUO Desk IKitten

  7. What does the GDA read from the IKitten • Metadata read only • Fedid – the GDA knows the users Fedid and asks the DB for the current visit for that user, used for setting up data paths. • Instrument, abstract Data / metadata IKitten GDA

  8. Role Based access and application sharing. • Definable access • Definable role • If this is not right…

  9. What does the GDA write out? • GDA does not currently write a NeXus file, • Detector files and anything….. • Currently implemented only on MX beamlines Nexus & Data GDA

  10. ‘GDA’ needs to initiate next step • In the MX GDA whenever an image file is created it runs a script • Bash script creates an XML file as input to DArc • The GDA can produce a script itself but no beamlines currently do today… maybe next week? Nexus & Data GDA

  11. DArc DArc Watches a directory • Python Based EDNA • Server that checks <>5s • http://www.edna-site.org Initiates storaged Copies a filelist To a location Converts file to XML Ingest format Runs XML ingest Success? Moves XML to A log

  12. The XML file, minimum! <?xml version="1.0" ?><icat version="1.0 RC6" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="icatXSD.xsd"> <study> <investigation> <inv_number>MX307</inv_number> <visit_id>MX307-27</visit_id> <instrument>i04</instrument> <title>dont need it</title> <inv_type>experiment</inv_type> <dataset> <name>icr/fernando</name> <dataset_type>EXPERIMENT_RAW</dataset_type> <description>unknown</description> <datafile> <name>FG2_3_MS_3_154.img</name> <location>srb://srb-mcat-i18.esc.rl.ac.uk:5518/dls-2/i04/data/data/2009/mx307-27/icr/fernando/FG2_3_MS_3_154.img</location> <description>unknown</description> <datafile_version>1.0</datafile_version> <datafile_create_time>2009-08-11T02:58:21</datafile_create_time> <datafile_modify_time>2009-08-11T02:58:21</datafile_modify_time> </datafile> </dataset>

  13. What is our status? • https://dataportal.diamond.ac.uk

  14. New Eclipse Based GDA

  15. Viewing and choosing the Metatdata Relation to NeXus

  16. By the by…https://ispyb.diamond.ac.uk • Did try a view, now use a script ISPyB DLS ICAT DUO Desk IKitten

  17. ICAT Archive Generic ISPyB LIMS Domain specific ISPyB and ICAT

  18. How robust is the system? Atlas Data Store DataPortal Active Directory Diamond Proposal Web pages People DB SRB DLS ICAT DUO DUO Desk Data / metadata IKitten DArc and Storaged Nexus & Data GDA

  19. How robust is the system? Atlas Data Store DataPortal Active Directory Diamond Proposal Web pages People DB SRB DLS ICAT DUO DUO Desk Data / metadata IKitten DArc and storaged Nexus & Data GDA

  20. Future Work • Use it!!! • Portal and API integration into in house tools • Sample information from user office (top level information about the sample, and ISPyB (the instance of the sample). • Internal ICAT. • Reduced data and richer metadata. • Archive old data

  21. Acknowledgements • Diamond Light Source: Tobias Richter, Stuart Campbell, Karl Levik, Marc Basham, Karl Levik, Bill Pulford (and others in groups who have contributed at various times on various beamlines) • STFC e-Science: Michael Gleaves, Roger Downing, Rik Tyer, Glen Drinkwater, Shoaib Sufi, Phil Couch, Kerstin Kleese Van Dam, Keir Hawker, Carmine Cioffe, Gordon Brown, Lisa Blanshard, Kevin O’Neill. • STFC Daresbury: Steve Kinder and Karen Ackroyd • But especially, Roger, Keir, Carmine, Michael, Mark and Tobias. • BTW we have 64Tb of raw data already 

More Related