1 / 21

VO Standards – Catalog Access

VO Standards – Catalog Access. Tamás Budavári Johns Hopkins University. Background. Virtual Observatory Query Language Level 3. Semantically rich XML language (TBD) Level 2. Portals combining astro archives e.g. SkyQuery and Open SkyQuery Level 1. Query a single database

zita
Download Presentation

VO Standards – Catalog Access

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VO Standards – Catalog Access Tamás Budavári Johns Hopkins University

  2. Background • Virtual Observatory Query Language • Level 3. Semantically rich XML language (TBD) • Level 2. Portals combining astro archives e.g. SkyQuery and Open SkyQuery • Level 1. Query a single database Astronomical Data Query Language VOTable • Start with something simple • Decided not to reinvent the wheel NVO Summer School, Aspen 2004

  3. SQL Primer How to query databases ADQL Extensions for astronomy Distributed resources SkyQuery SkyNode interface SkyQuery portal Outline NVO Summer School, Aspen 2004

  4. Simple syntax SELECT<columns>FROM<table>WHERE<conditions> For example: select ra, dec from galaxy where r<19 Joining tables More tables with common column (link, relation) Select from multiple tables with join criteria E.g. an object id: Structured Query Language select g.ra, g.dec, s.z from galaxy g, spec s where g.objid=s.objid and g.r<19 NVO Summer School, Aspen 2004

  5. Astronomical Data Query Language • SQL extended for astronomy • Added survey name in from clausee.g. SDSS:Galaxy g, DLS:obj o • Specify area on sky • Added “REGION” keyworde.g. Region('CIRCLE J2000 181.3 -0.76 6.5') • Formulate federated query • Added “XMATCH” keyworde.g. XMatch(g,o) using aliases NVO Summer School, Aspen 2004

  6. Representations String for humans /s XML for machines /x Helper service Converts both ways String to XML XML to String XML is pre-parsed So your service could accept both if needed ADQL Translator NVO Summer School, Aspen 2004

  7. ADQL/s ADQL/x XML ADQL representations <?xml version="1.0" encoding="utf-16"?> <Select xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.ivoa.net/xml/ADQL/v0.7.4"> <SelectionList> <Item xsi:type="columnReferenceType" Table="s" Name="objd" /> <Item xsi:type="columnReferenceType" Table="t" Name="id" /> </SelectionList> <From> <Table xsi:type="archiveTableType" Archive="SDSS" Name="Galaxy" Alias="s" /> <Table xsi:type="archiveTableType" Archive="TWOMASS" Name="PhotoObj" Alias="t" /> </From> <Where> <Condition xsi:type="intersectionSearchType"> <Condition xsi:type="xMatchType"> <Table xsi:type="includeTableType" Name="s" /> <Table xsi:type="includeTableType" Name="t" /> <Nature>&lt;</Nature> <Sigma xsi:type=“realType" Value="3.5" /> </Condition> <Condition xsi:type="regionSearchType"> <Region xmlns:q1="urn:nvo-region" xsi:type="q1:circleType" coord_system_id=""> <q1:Center ID="" coord_system_id=""> <Pos2Vector xmlns="urn:nvo-coords"> <Name>Ra Dec</Name> <CoordValue> <Value> <double>180</double> <double>0</double> </Value> </CoordValue> </Pos2Vector> </q1:Center> <q1:Radius>6.5</q1:Radius> </Region> </Condition> </Condition> </Where> </Select> select s.objd, t.id from TWOMASS:PhotoObj t, SDSS:Galaxy s where XMatch(s,t)<3.5 and Region('Circle J2000 180 0 6.5') NVO Summer School, Aspen 2004

  8. SkyNode interface to archives Implements ADQL returns VOTable Basic node understands “REGION” Full node understands “XMATCH” SkyQuery portal Knows the SkyNodes from Registry Understands federated query Open SkyQuery NVO Summer School, Aspen 2004

  9. Architecture Web App Image cutout SkyQuery SkyNodeSDSS SkyNode2Mass SkyNodeFirst NVO Summer School, Aspen 2004

  10. Step-by-step SELECT o.objId, o.r, o.type, t.objId, t.j_m FROM SDSS:PhotoPrimary o, TWOMASS:PhotoObj t WHERE XMatch(o,t)<3.5 AND Region(‘CIRCLEJ2000…’) AND (o.i-t.j_m)>2AND o.type=3 • Analyze query • Cost estimates • Sort by size • Execution plan • Cross-match • Recursively from small to large db • Propagate necessary attributes only • Return output in VO format NVO Summer School, Aspen 2004

  11. Metadata functions Tables, Columns, Functions Uptime, Footprint, etc… Query functions PerformQuery (ADQL q) Simple query interface QueryCost (ADQL q) Cost estimate for query ExecutePlan (XPlan plan) Recursive probabilistic join SkyNode NVO Summer School, Aspen 2004

  12. Data flow Registry SkyQuery query SkyNode 1 SkyNode 2 SkyNode 3 http://openskyquery.net NVO Summer School, Aspen 2004

  13. Cross-matching • Probabilistic spatial join • Likelihood that all measurements are actually observations of the same object • Consider all objects from all surveys • For surveys A, B and C, there are NANBNC combinations,NX is the number of objects that meet criteria in survey X • Massive optimization problem for true positionson the sky (x,y,z) NVO Summer School, Aspen 2004

  14. XMatch variants • XMatch (r, g, w) • Mandatory matches in all surveys • XMatch (r, g,!w) • Look for dropouts in certain surveys • Remove objects if found in w • XMatch (r, g,?w) • Return matches in w if found but don’t discard if not • Essentially an “outer join” NVO Summer School, Aspen 2004

  15. Hides complexity Simple web services Methods Get to metadata of nodes Find location of nodes Send query to one node Send federated query Create execution plan Submit execution plan SkyQuery portal NVO Summer School, Aspen 2004

  16. NVO Summer School, Aspen 2004

  17. And more… • Cross-match your own catalog • Upload file to temporary table, e.g. MyTable • Use MyData:MyTable in federated query • Browse metadata on SkyNodes • Sample queries of all types NVO Summer School, Aspen 2004

  18. Download template SkyNode written in C# Uses SQL Server 2000 HTM for fast searches Where? http://openskyquery.net Ready to use web service http://skyserver.org Templates for database From scratch Write it in any language C++, Java, C#, Perl, Python Use ADQL Translator? Open source .NET http://go-mono.org C# w/ DB access and WS Runs on your favorite OS Build your own SkyNode NVO Summer School, Aspen 2004

  19. Summary • Astronomical Data Query Language • Working VO standard for catalog access • Constructs for area and cross-match • Open SkyQuery • Dynamically federates astronomical archives • Uses VO Registry to lookup SkyNodes • On-the-fly probabilistic cross-matchingw/ dropout and outer join capabilities NVO Summer School, Aspen 2004

  20. Future Works • Things to add • Implement missing outer join • Full featured region specs • Footprint services • Test survey boundaries • Automated intersection of footprints • And more… NVO Summer School, Aspen 2004

  21. Where to go • Links in the talk • http://openskyquery.net/ • http://openskyquery.net/adqltranslator/ • http://skyserver.org/ • http://skyquery.net/ • Other relevant links • http://us-vo.org/ • http://voservices.org/ • http://www.ivoa.net/ NVO Summer School, Aspen 2004

More Related