1 / 39

Euro-VO DCA WP4 and GAVO and (a little) AstroGrid-D

Euro-VO DCA WP4 and GAVO and (a little) AstroGrid-D. Gerard Lemson Euro-VO DCA WP5 kick-off meeting Trieste, 2006-11-30. Overview. Euro-VO DCA WP4 Goal and motivation Theory in VO Theory in IVOA Theory in Euro-VO DCA: WP4 WP4 and WP5: theory WP5 and/or grid in WP4 ?

leda
Download Presentation

Euro-VO DCA WP4 and GAVO and (a little) AstroGrid-D

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Euro-VO DCA WP4andGAVO and (a little) AstroGrid-D Gerard Lemson Euro-VO DCA WP5 kick-off meeting Trieste, 2006-11-30 Euro-VO DCA WP5 kick off

  2. Overview • Euro-VO DCA WP4 • Goal and motivation • Theory in VO • Theory in IVOA • Theory in Euro-VO DCA: WP4 • WP4 and WP5: theory WP5 and/or grid in WP4 ? • GAVO (& AstroGrid-D) • GAVO I • GAVO II • Theory in GAVO • Grid in GAVO • AstroGrid-D • (GA)VO and AstroGrid-D Euro-VO DCA WP5 kick off

  3. Euro-VO DCA WP4 • Objectives • This task aims at assessing the inclusion of theory data and services in the Virtual Observatory • Description of work • The task is tackled by the Theory Expert Group, which has yearly meetings and email discussions to establish a framework for the inclusion of Theoretical Astronomy Services in the VObs. The TEG organizes a Workshop. • Deliverables • WP4-D10: Euro-VO Theory Workshop • WP4-D11: TEG Report: Framework for inclusion of Theoretical Astronomy Services in the VObs • Expected results • The final report will describe a framework for inclusion of theoretical data and services in the VObs • Motivation of the European Theoretical astronomy community to participate in the VObs endeavour. Euro-VO DCA WP5 kick off

  4. Reminder • Broad goal of VObs • Make results of astronomical research, data and applications, more readily available to larger community, and create value-adding services. • Facilitate results- • communication • checking • (re)use • comparison • combination • Organised in IVOA Euro-VO DCA WP5 kick off

  5. The observational VObs • Most VO efforts concentrate on observational data sets • simple observables: photons detected at a certain time from a certain area on the sky • long history of archiving • pre-existing standards (FITS) • valuable over long time (digitising 80 yr old plates) • long history of comparing different observations: interoperability • Standards observationally biased • common sky: cone search, SIAP, region • common objects: XMatch • data models: characterisation of sky/time/energy (/no polarisation yet) Euro-VO DCA WP5 kick off

  6. The same object X-Ray Radio Optical John Hibbard http://www.cv.nrao.edu/~jhibbard/n4038/n4038.html NASA/CXC/SAO/G. Fabbiano et al. Euro-VO DCA WP5 kick off

  7. Theory in VO • Apply same ideas to theoretical resources (datasets and services). • Plenty of resources available online. • Not standardised: • 2004, whitepaper detailing issues related to standardisation of theory Euro-VO DCA WP5 kick off

  8. Theory in the VO: issues • Simulations not so simple • complex observables • no standardisation (not even HDF5) • archiving ad hoc, for local use • Moore’s law makes useful lifetime relatively short: few years later can do better Euro-VO DCA WP5 kick off

  9. “Moore’s law” for N-body simulations Euro-VO DCA WP5 kick off Courtesy Simon White

  10. Standardisation issues • Current IVOA standards somewhat irrelevant • assume common sky • common objects • comparison direct • In theory • no common sky • no common objects • comparison statistical • requires new data models • content, physics, code Euro-VO DCA WP5 kick off

  11. Not the same object Toomre & Toomre, 1972 Courtesy Volker Springel Di Matteo, Springel and Hernquist, 2005 Euro-VO DCA WP5 kick off

  12. So why bother ? • Simulations are interesting: • For many cases only way to see processes in action • Complex observations require sophisticated models for interpretation • Bridging gap in specialisations: not everyone has required expertise to create simulations, though they can analyse them. • Many use cases do not require the latest/greatest • exposure time calculator • survey design Euro-VO DCA WP5 kick off

  13. Theory in the IVOA • IVOA Interest group on theory • founded May 2004 • “horizontal” • use cases • Current projects: • Theory UCDs/Semantics (Laurie Shaw) • SNAP (Claudio Gheller) • Simulation (meta-) data model (GL) • Meeting on SNAP in Garching, early 2007 Euro-VO DCA WP5 kick off

  14. Euro-VO DCA WP4 (repeat) • Objectives • This task aims at assessing the inclusion of theory data and services in the Virtual Observatory • Description of work • The task is tackled by the Theory Expert Group, which has yearly meetings and email discussions to establish a framework for the inclusion of Theoretical Astronomy Services in the VObs. The TEG organizes a Workshop. • Deliverables • WP4-D10: Euro-VO Theory Workshop • WP4-D11: TEG Report: Framework for inclusion of Theoretical Astronomy Services in the VObs • Expected results • The final report will describe a framework for inclusion of theoretical data and services in the VObs • Motivation of the European Theoretical astronomy community to participate in the VObs endeavour. Euro-VO DCA WP5 kick off

  15. WP4: tasks • Procurement WP4 personnel: • Draft whitepaper “Framework for inclusion of Theory data and services in the VObs” (End cycle 1) • EURO-VO Theory Workshop (Deliverable 10, Month 17) • Whitepaper ready (Deliverable 11, month 26) • Participation IVOA interop meetings with theory Euro-VO DCA WP5 kick off

  16. WP4 first steps • Formation Theoretical astronomy Experts Group (TEG) • CNRS Hervé WOZNIAK • ESA Pedro OSUNA • ESO None • INAF Santi CASSISI • INTA Miguel CERVIÑO • MPG Gerard LEMSON (Chair) • NOVA TBD (Joop SCHAYE) • LU TBD (?) • Census theory data centers (with WP2) • Cooperation in IVOA theory interest group activities • Coordination with WP 5 (which is why I’m here) Euro-VO DCA WP5 kick off

  17. WP4 and WP5: some first ideas • Grid support for WP4 • Grid must be addressed in WP4 whitepaper • Grid can facilitate/is required for implementations of prototype for standards (see below) • Theory use cases/requirements for WP5. • Asynchronous, queued querying of large simulation database • OGSA-DAI? (GAVO+AstroGrid-D) • SNAP on Millennium raw data • grid services wrapping custom data access modules (GAVO+AstroGrid-D) • Scientifically useful virtual telescopes • Parameter studies using “small” simulations • Large simulations ?? Euro-VO DCA WP5 kick off

  18. Example: Virtual Telescope Observed Simulated Thanks to Volker Springel Thanks to Alexis Finoguenov, Ulrich Briel, Peter Schuecker, MPE) Euro-VO DCA WP5 kick off

  19. Virtual telescope configuration Euro-VO DCA WP5 kick off

  20. Virtual telescope configuration • Components: • simulation of source, wrapped with “photon producer” service • spec and implementation for “pre-observation image” (POI) • “foreground” accepting, reemitting POI • ISM/IGM • lenses • “software telescope”, simulating realistic observations • comparison software • Required (VO/Grid) infrastructure supporting this workflow • registry for publishing/discovering components • computational resources • scheduler for executing job • streaming support for flowing POI between components • storage for results • workflow engine? Euro-VO DCA WP5 kick off

  21. Discussion Suggestions very welcome Euro-VO DCA WP5 kick off

  22. Intermezzo Euro-VO DCA WP5 kick off

  23. GAVO and AstroGrid-D Euro-VO DCA WP5 kick off

  24. GAVO and AstroGrid-D • GAVO I • Funded by German Ministry of • Dec 2002 – March 2005 • 3 months extension • 3 FTEs • AstroGrid-D • 1 of 5 funded “communities” taking part in D-Grid project • O(13) FTEs • Autumn 2005 – 2007/8 • GAVO II • Delayed funding, due to need to distinguish from AstroGrid-D • June 2006-July 2008 • 5 FTEs (so far 3, ARI (>~1.5), MPE (1.5), TUM (.5), AIP (<~.5), Tübingen (1)) Euro-VO DCA WP5 kick off

  25. GAVO (I + II) projects • Archives: • ROSAT All Sky Survey • cone search: photons, sources • SIAP: fields • ROSAT pointed observations: SIAP • standardised approach to publish FITS images under SIAP • uses FITS metadata repository (SQLServer) • SQL based mapping and SIAP implementation • easily generalised to other datasets and protocols (SSA) • RAVE • CDFS optical follow up spectroscopy: SSA • Heidelberg catalogues and GAVO archive: SCS/SkyNode • Data mining • Cross-matcher (also on Grid, TUM) • SED classifier • Cluster finder Euro-VO DCA WP5 kick off

  26. GAVO (I + II) projects ctd. • Grid • evaluation Globus • Gadget 2 on Globus MPI (AIP + MPE) • cluster finder on the grid • VOQL/SkyNode • VOQL tiger team (Benjamin Gufler) • HiSbase peer-to-peer approach to multi catalogue cross-matching • Mock catalogues for testing • Theory • IVOA: • Initialised Theory Interest Group (chair) • SNAP • Data models • Semantics • Millennium database • (proto-) Virtual telescopes (planck, MoMaF2, Dolag et al) Euro-VO DCA WP5 kick off

  27. AstroGrid-D • Part of D-Grid • Community project • Provide • use cases • middleware • adaptation of existing applications/data to the (D-)Grid Euro-VO DCA WP5 kick off

  28. AstroGrid-D partner projects (incomplete) • Astrophysikalisches Institut Potsdam • project management • robotic telescopes on the grid • galaxy merger simulations • MPE: • cluster finder • catalogue cross-matcher (with TUM) • adapt simulation visualisation to the Grid • access to persistent data (in database) • MPA • Planck Process Coordinator on the Grid • Technischer Universitat München-Informatik • StarGlobe: streaming data querying/combination Euro-VO DCA WP5 kick off

  29. AstroGrid-D partner projects, ctd. • Zuse Institut Berlin • Grid Application Toolkit • GridSphere web portal • Information Services (“registry”, RDF+SparQL) • Albert Einstein Institute, Berlin • Cactus tool kit • Grid based simulation monitoring • Astronomisches Rechen Zentrum, Heidelberg • Stellar dynamics: Grape on the Grid • GaiaGrid • Planck ProC collaboration Euro-VO DCA WP5 kick off

  30. GAVO + AstroGrid-D, synergies • Groups involved in both projects: AIP, ARI, MPE, TUM • VO offers different kind of use cases: • VObs aims to enable external users to use resources • Heterogeneous world: Interoperate with resources outside of AstroGrid-D (but within IVOA standards) • Grid can help implement VO services Euro-VO DCA WP5 kick off

  31. GAVO + AstroGrid-D ctd. • Planned (from GAVO point of view) • Cluster finder (MPE) • SNAP implementation (MPA) • OGSA-DAI (GSM?, DSA?, CAS Jobs) access to Millennium DB (MPA) • Real virtual telescopes (MPA+...) • Cross matching (MPE+TUM) • Possibly • VOEvent and Robotic Telesopes ? • AstroGrid-D Information Services as IVOA Registry Euro-VO DCA WP5 kick off

  32. SNAP using grid services • Simple Numerical Access Protocol • Project of IVOA theory interest group • Based on simulation data model (input welcome !) • Query interesting simulations • Select and retrieve subset • Implementation: SNAP on Millennium snapshots • Custom access code deployed near large datasets • Authenticated access • Standardised publication for use in Grid workflows (ProC? Streaming?) • Data staging and movement Euro-VO DCA WP5 kick off

  33. Globus container SNAP web service Grid Service getBox() C-module Information Service Millennium snaphots IS aware Workflow Engine Euro-VO DCA WP5 kick off

  34. Millennium website and OGSA-DAI • Dataset • postprocessing products of Millennium simulation (http://www.mpa-garching.pg.de/millennum) • halo/galaxy catalogues • density fields (environment queries) • Functions • support generic SQL queries on large (O(5Tb)), complex database • support mydb • some graphics script based access Euro-VO DCA WP5 kick off

  35. Implementation • Microsoft SQLServer database (SkyServer inspired) • data model for efficient retrieval merger tree histories • spatial indexing • TOMCAT web server • http://www.g-vo.org/MyMillennium • authentication using RDB • wget based access, IDL+R modules • Graphics • VOPlot applet (small results, browser) • TOPCAT (plugin) • Synchronous queries • JDBC • 7 min time limit • Issues • I/O bottle necks when > 4 users query same table • short maximum query time • no asynchronous access • simplistic user management (compared say to CAS Jobs) Euro-VO DCA WP5 kick off

  36. Euro-VO DCA WP5 kick off

  37. Why Grid interesting? • Need • Asynchronous access • Queuing • Improve result management, data delivery • Bulk uploading into MyDB • More generic authentication (X.509) • Will try CAS jobs (SDSS/JHU) • Investigate OGSA-DAI • Map portal to OGSA-DAI functionality • Monitoring ? Logging ? Performance ? • Will try AstroGrid-UK’s DSA • Other suggestions ? Euro-VO DCA WP5 kick off

  38. Euro-VO DCA WP5 kick off

  39. Thank you ! Euro-VO DCA WP5 kick off

More Related