1 / 14

OpenSkyQuery (Parallel Scanning with Cooperative Agents for the NVO)

OpenSkyQuery (Parallel Scanning with Cooperative Agents for the NVO). Making SkyQuery Truly VO-Compliant Ani Thakar (JHU) / Peter Kunszt (CERN). SkyQuery. Distributed query prototype for VO Federation of astronomical archives Dynamic cross-match between catalogs Hierarchical Web Services

Olivia
Download Presentation

OpenSkyQuery (Parallel Scanning with Cooperative Agents for the NVO)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. OpenSkyQuery(Parallel Scanning with Cooperative Agents for the NVO) Making SkyQuery Truly VO-Compliant Ani Thakar (JHU) / Peter Kunszt (CERN)

  2. SkyQuery • Distributed query prototype for VO • Federation of astronomical archives • Dynamic cross-match between catalogs • Hierarchical Web Services • SkyQuery Portal • SkyNodes • Prototype developed for .NET contest • Won second prize (2002) AISRP PI Meeting 2003, Pittsburgh

  3. SkyQuery Architecture • Client • ASP page • SkyQA Java app • Portal • .NET Web service • SkyQL parser • SkyNodes • .NET Web services • DBs • SQL Server • HTM-indexed Client connects to portal, which executes the distributed query on the SkyNodes using WebMethods that send SOAP requests and receive XML DataSets Results: XML table of values, JPEG cutout of area AISRP PI Meeting 2003, Pittsburgh

  4. SkyNode4 Dropout Count Count Count Cross-Matching Algorithm Cross-Match Query Pre-Query Partial Results Xmatch RPC 1,065,100 SkyNode1 Partial Results Xmatch RPC Client Portal Final Results SkyNode2 7,004 305,200 Xmatch RPC Partial Results Xmatch RPC SkyNode3 AISRP PI Meeting 2003, Pittsburgh

  5. Original SkyQuery AISRP PI Meeting 2003, Pittsburgh

  6. SkyQuery Today • Several new SkyNodes added • SDSS SkyNode upgraded to DR1 • 2MASS, FIRST SkyNodes upgraded to full survey data • SkyNode template published • More sample queries to be added AISRP PI Meeting 2003, Pittsburgh

  7. OpenSkyQuery • VO-Compliance via IVOA standards • Interoperability is the key • Query language – ADQL/VOQL • Output format – VOTable • Automated discovery – VO Registry • 2 levels of SkyNode compliance • Core and Advanced • High speed data access for VO Data Grid • Caching, partitioning, Grid interface AISRP PI Meeting 2003, Pittsburgh

  8. OpenSkyQuery Proposal AISRP PI Meeting 2003, Pittsburgh

  9. ADQLVOQL • ADQL – Astronomical Data Query Language • See poster • Three layers of VOQL • VOQL1-Web Services (Basic SkyNode) • ADQL and VOTable to exchange info between machines • VOQL2-Federation, SkyQL (Full SkyNode) • SQL-like query language and federation system • Combination of SkyQuery, JVOQL and VO standards • VOQL3 – SkyXQuery • Future XML-based query language • VO services can choose level of compliance AISRP PI Meeting 2003, Pittsburgh

  10. VOQL Levels VOQL 1 VOQL 2 VOQL 3 ADQL Basic SkyNode Query Full SkyNode SkyQL Query (Future) SkyXNode Full VOQL Query AISRP PI Meeting 2003, Pittsburgh

  11. OpenSkyQuery Development • IVOA Compliance • ADQL & OpenSkyNode IVOA Working Drafts submitted (O’Mullane, Budavári et al.)  • ADQL Parser (W.O’Mullane, V.Haridas)  • VOTable output (S.Carliles, W. O’Mullane) • Prototype Registry @ JHU (G.Greene, W.O’Mullane)  • Speeding up distributed queries • Faster HTM v2 (G.Fekete, A.Szalay)  (poster) • Bypass caching (T.Malik) • Partitioning, Grid interface (M.Nieto-Santisteban) • Ferris Wheel proposal (Szalay et al.) • Other • SkyNode template published (N.Purger)  • Enhancements to cross-match algorithm • Joint development with ROE (B.Mann) AISRP PI Meeting 2003, Pittsburgh

  12. Bypass Caching • Assumption • SkyQuery is network-bound • Not sensitive to latency (for reasonable latencies) • Features • Altruistic, not greedy caching • Good network citizenship • Reduces overall network traffic rather than increasing local traffic by caching large amounts of data locally • Only high-yield queries cached • Queries that are frequent and return a lot of output • Yield metric BYHR (Byte Yield Hit Rate) • Size of output and frequency of query both factored in • Data subset for queries with high BYHR cached AISRP PI Meeting 2003, Pittsburgh

  13. Development Timeline • IVOA compliance • ADQL/VOTable output - End 2003 • Registry/VOQL – Spring 2004 • Enhancement to cross-matching algorithm • Spring 2004 • Caching/partitioning/parallel data access • Summer/Fall 2004 • Grid interface • Fall 2004 AISRP PI Meeting 2003, Pittsburgh

  14. Publications and Links • Budavári, T. et al. 2002, ADASS XII, 12, 31, “SkyQuery - A Prototype Distributed Query Web Service for the VO” • Malik, T., et al. 2002, CIDR `03, 17, “SkyQuery: A WebService Approach to Federate Databases” • Szalay, A. S., et al. 2002, SPIE, ”Web Services for the Virtual Observatory” • http://www.SkyQuery.net • http://skyservice.pha.jhu.edu/develop/vo/ • http://skyservice.pha.jhu.edu/develop/vo/adql/ • http://www.ivoa.net/twiki/bin/view/IVOA/IvoaVOQL AISRP PI Meeting 2003, Pittsburgh

More Related