1 / 26

AstroGrid and OGSA-DAI: Early experiences

AstroGrid and OGSA-DAI: Early experiences. Kona Andrews Guy Rixon Institute of Astronomy University of Cambridge. Data Set Access: The DataCenter. DataCenter component. Data Access Plug-in. Web Services. Data. Tomcat / Axis. DataCenter JDBC plug-in. DataCenter component. JDBC.

helena
Download Presentation

AstroGrid and OGSA-DAI: Early experiences

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AstroGrid and OGSA-DAI: Early experiences Kona Andrews Guy Rixon Institute of Astronomy University of Cambridge

  2. Data Set Access: The DataCenter DataCenter component Data Access Plug-in Web Services Data Tomcat / Axis AstroGrid and OGSA-DAI: Early experiences

  3. DataCenter JDBC plug-in DataCenter component JDBC Web Services DBMS Tomcat / Axis AstroGrid and OGSA-DAI: Early experiences

  4. DataCenter custom plug-in DataCenter component File querier Web Services FITS files Tomcat / Axis AstroGrid and OGSA-DAI: Early experiences

  5. DataCenter proxy plug-in DataCenter component Service proxy Web Services Vizier Tomcat / Axis AstroGrid and OGSA-DAI: Early experiences

  6. DataCenter Component OGSA-DAI client plug-in Tomcat / Axis What we imagined Grid Service Ports Web Services OGSA- DAI Tomcat / OGSA AstroGrid and OGSA-DAI: Early experiences

  7. DataCenter Component OGSA-DAI client plug-in Tomcat / Axis Problem Grid Service Ports Web Services OGSA- DAI Tomcat / OGSA AstroGrid and OGSA-DAI: Early experiences

  8. DataCenter Component OGSA-DAI client plug-in Tomcat / Axis Problem Grid Service Ports Web Services OGSA- DAI Tomcat / OGSA AstroGrid and OGSA-DAI: Early experiences

  9. DataCenter Component OGSA-DAI “client” plug-in Tomcat / Axis Nasty solution Separate JVM OGSA-DAI client Web Services Grid Ports OGSA- DAI Tomcat / OGSA AstroGrid and OGSA-DAI: Early experiences

  10. Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL AstroGrid and OGSA-DAI: Early experiences

  11. Current use of OGSA-DAI activities Data Center OGSA-DAI SQL Postgres DBMS ADQL to SQL ADQL AstroGrid and OGSA-DAI: Early experiences

  12. Current use of OGSA-DAI activities Data Center OGSA-DAI SQL SQL sqlQueryStatement Postgres DBMS ADQL to SQL ADQL AstroGrid and OGSA-DAI: Early experiences

  13. Current use of OGSA-DAI activities Data Center OGSA-DAI SQL SQL sqlQueryStatement Postgres DBMS ADQL to SQL ADQL (to XML RowSet) Result rows AstroGrid and OGSA-DAI: Early experiences

  14. Current use of OGSA-DAI activities Data Center OGSA-DAI SQL SQL sqlQueryStatement Postgres DBMS ADQL to SQL deliverFromURL ADQL (to XML RowSet) Result rows AstroGrid and OGSA-DAI: Early experiences

  15. Current use of OGSA-DAI activities Data Center OGSA-DAI SQL SQL sqlQueryStatement Postgres DBMS ADQL to SQL deliverFromURL ADQL (to XML RowSet) Result rows xslTransform VOTable AstroGrid and OGSA-DAI: Early experiences

  16. Current use of OGSA-DAI activities Data Center OGSA-DAI SQL SQL sqlQueryStatement Postgres DBMS ADQL to SQL deliverFromURL ADQL (to XML RowSet) Result rows xslTransform VOTable deliverToFile / deliverToGFTP AstroGrid and OGSA-DAI: Early experiences

  17. Current use of OGSA-DAI activities Data Center OGSA-DAI SQL SQL sqlQueryStatement Postgres DBMS ADQL to SQL deliverFromURL ADQL (to XML RowSet) Result rows xslTransform VOTable Deliver results deliverToFile / deliverToGFTP AstroGrid and OGSA-DAI: Early experiences

  18. Comments Astronomers often need LARGE result sets Thousands to millions of rows Memory usage is a big issue OGSA-DAI 3.1 much better than 3.0 xslTransform activity hogging memory AstroGrid and OGSA-DAI: Early experiences

  19. OGSA-DAI speed penalty • Rough figures only • Measured without xslTransform activity • Will likely vary with available memory AstroGrid and OGSA-DAI: Early experiences

  20. Planned extension: User tables DBMS Data Center OGSA- DAI R/O Datasets Query Query results AstroGrid and OGSA-DAI: Early experiences

  21. Planned extension: User tables DBMS Data Center OGSA- DAI R/O Datasets GDS to GDS transfer Bulk load User tables OGSA- DAI Query Query results Data flow AstroGrid and OGSA-DAI: Early experiences

  22. Planned extension: User tables DBMS Data Center OGSA- DAI R/O Datasets Data Center DBMS User tables OGSA- DAI Table Manager Query Query results Data flow AstroGrid and OGSA-DAI: Early experiences

  23. Planned extension: User tables DBMS Data Center OGSA- DAI R/O Datasets Data Center DBMS User tables OGSA- DAI Table Manager Client User data Query Query results Data flow AstroGrid and OGSA-DAI: Early experiences

  24. Comments Will be relying on sqlBulkLoadRowSet (!) Table management facilities currently thin Not using X.509 auth at user granularity Still require W/S-based access to G/S AstroGrid and OGSA-DAI: Early experiences

  25. Our needs • Ability to handle very large result sets - Don’t rely on loading whole result set into memory! • Suite of DB/table management activities • List tables in DB • Describe table column types / space usage • Last access/modification time • Etc etc etc • Ability to create and delete indices on tables • Ultra-simple installation procedures AstroGrid and OGSA-DAI: Early experiences

  26. The End www.astrogrid.org AstroGrid and OGSA-DAI: Early experiences

More Related