1 / 33

Virtual Observatory Architecture Data Services Registry Services Compute Services

Virtual Observatory Architecture Data Services Registry Services Compute Services Roy Williams Caltech US-VO co-director. Trends. Future dominated by detector improvements. Moore’s Law growth in CCD capabilities Gigapixel arrays on the horizon

abiola
Download Presentation

Virtual Observatory Architecture Data Services Registry Services Compute Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Virtual Observatory Architecture Data ServicesRegistry ServicesCompute Services Roy WilliamsCaltech US-VO co-director

  2. Trends • Future dominated by detector improvements • Moore’s Law growth in CCD capabilities • Gigapixel arrays on the horizon • New Detector Technologies (e.g., STJ) • Improvements in computing and storage will track growth in data volume • Investment in software is critical, and growing Total area of 3m+ telescopes in the world in m2, total number of CCD pixels in Megapixels, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels.

  3. Astronomical Data • Image • Standard file format: FITS • Standardized c.1980 • Keyword-value dictionary + binary block • Catalog • Derived from image • Connected set of bright pixels • “Table of stars” • Standard format: VOTable • Standardized 2002 • XML with remote binary • Spectrum, Time series, ... new instrumentsnew astronomynew requirements -> moreDIVERSITY

  4. First NVO Discovery Database crossmatch of two massive databases creates new science “The sum is greater than the parts”

  5. Crossmatch Services NVO protocols SDSS database query Crossmatch service 2MASS database query query scientific knowledge!

  6. Networks of Services Catalog Service Query Check Service Query Estimator Archive Service User’s code Crossmatch Service Archive Service Storage Service What is the meaning of the service? Who is responsible? How can I use the service from Perl/Java/C++/IDL/IRAF? Is there a simple web client for the service? What is the request and response syntax? What authentication do I need?

  7. Discover Compute Publish Collaborate Portals, User Interfaces, Tools Topcat VOPlot SkyQuery OASIS Mirage DIS Aladin conVOT interfaces to data Registry Layer Data Services Compute Services HTTP Services SOAP Services Grid Servicesstateless, registered & self-describing & persistent, authenticated Semantics (UCD) crossmatch visualization Bulk Access OAI ADS image data mining source detection OpenSkyQuery SIAP, SSAP VOTable FITS, GIF,… Virtual Data Digital LibraryOther registriesXML, DC, METS Workflow (pipelines) Authentication & Authorization Existing Data Centers My Space storage services Grid MiddlewareSRB, Globus, OGSASOAP, GridFTP Databases, Persistency, Replication Disks, Tapes, CPUs, Fiber

  8. VOTable • Full metadata representation • Hierarchy of RESOURCEs • containing PARAMs and TABLEs • UCD (unified content descriptor) • a has unit meter • a has UCDORBIT_SIZE_SMAJ (Semi-major axis of the orbit ) • Can reference remote and/or binary streams • Table can be • Pure XML • "Simple Binary" • FITS Binary Table

  9. Cone/SIAP/SSAP Data Services • Simple, pragmatic solutions • quickly Specified, Created, Registered, Utilized! • Cone • request is cone, response is VOTable with RA, Dec • many of these since 2/02 • SIAP • request is cone, response is VOTable of image links • SSAP • under development

  10. Simple Image Protocol Data Services • Specify box by position and size • SIAP server returns relevant images • Footprint • Logical Name • URL Can choose: standard URL: http://....... SRB URL srb://nvo.npaci.edu/…..

  11. Unified Content Descriptors Data Services • UCD is a “semantic type” PHOT.INT-MAG.B Integrated total blue magnitude ORBIT.ECCENTRICITY Orbital eccentricity STAT.MEDIAN Statistics Median Value INST.QE Detector's Quantum Efficiency • Can be resolved by web service • to description, examples, etc • Base + Specifiers • eg error in default right ascension • POS.EQ.RA, MAIN, ERROR

  12. OpenSkyNode Data Services • Exposes a relational DB • select* from tables • select * from columns of table • select a,b,c where d>3 and e<4 • select ra, dec where REGION(ra, dec, .....) • select from Xmatch (SDSS, 2MASS) .....

  13. Registry Services Registry Services • Publish • Caltech, NCSA registries • Query • ADQL (borrow from OpenSkyNode) • XQuery/XPath • Harvest • OAI from NCSA, Caltech, JHU, Vizier (France) • What entities are described by registry? • Service • VO standard or arbitrary • Project, Data Collection • (person, community, VD object, etc etc ...?)

  14. Registry OAI Query Registry Registry OAI OAI Publish Publish Data Inventory Service Data Services Registry Services • Federates multiple cone, SIAP services JHU/StSci NCSA 4 Caltech Goddard DIS 2 1 3

  15. Data Inventory Service Data Services Registry Services Relevant Images and Catalogs NVSS Image ROSAT catalog

  16. Authorization WSDL Description Service Semantics CURATION VOResource Registry Services A mandatory form plus other supporting forms

  17. Service Paradigm Registry Services VOResource + VOStdService + WSDL Registry 1. publish fill in forms standard service types 2. find by white, yellow, green pages Provider Client 3. bind: request & response

  18. VO Identifiers Registry Services • URI form • Resolved by registries ivo://mydomain.com / mySkySurvey # file00037.fits • Authority ID • Registered with IVOA • Must correspond to a registry • Resource ID • Created by Authority • Resolved by registry • Record ID • Not known to registry delimiter delimiter

  19. Registry Registry Registry Publish Publish Publish Query Query Query OAI OAI OAI Open Archives Initiative (OAI) Registry Services • For harvesting registries • allows distributed control/fault-tolerance • Queries • Changes since last week • Give metadata in different views • Dublin Core, VOResource, VOVD? • Examples • Heasarc, Vizier, NCSA

  20. Services Compute Services • Why services • distributed & relocatable • workflow components • described by request/response protocols • self-describing (WSDL etc) • architecture independent • Questions • What’s wrong with remote objects? • Security framework • On-ramp for the normal human? • Toolkits for services • Bulk data and SOAP

  21. The Sky is a Database:Catalog Space Compute Services

  22. Figure 1. A schematic illustration of the problem of clustering analysis in some parameter space. The outliers, which do not fit well into any of the existing clusters, may be the new, previously unknown, and interesting objects. Catalog Space: The Final Frontier Compute Services

  23. Statistical Services Compute Services Convert pointset to density plus outliers 50000 stars in color-color space

  24. Image Federation Compute Services

  25. Multispectral Imagery Compute Services Crab Nebula.3 channels: X-ray in blue, optical in green, and radio in red. Moffet Field California. 224 channels from 400 nm to 2500 nm

  26. Multi-Wavelength Image Morphology Compute Services DPOSS-2MASS Image Mosaics J F N J H K J F N J H K Galaxy identifcation, galaxy clusters Pattern matching with shape AND color

  27. Images of the same galaxy taken several days apart are automatically subtracted from one another, and remaining bright spots may be supernova candidates. (NEAT project) detection Image subtraction allows detection of narrow-line features that are not also wide-band (eg Hα but not R-band) Image Federation Compute Services Stacking allows detection of faint sources. A 1-sigma detection in each of many bands becomes a 3-sigma detection. It's A New Window!

  28. Atlasmaker Compute Services Federated Images: wavelength, time, ... VO Registry SIAP SWarp Hyperatlas source detection average/max subtraction

  29. Atlasmaker DAG Compute Services Dependency from Scale Dependency from Compression

  30. ID RA DEC x y z Atlasmaker DAG Compute Services Dependency from Federation Dependency from Data Mining

  31. User request Request manager Mosaicked data is on file 2d: Store result & return result 2a. Mosaicked data is not on file 2b. Get raw data from NVO resources AtlasmakerVirtual Data System Compute Services Metadata repositories Federated by OAI Data repositories Federated by SRB 2c: Compute on TG/IPG Compute resources Federated by TG/IPG

  32. Grist Compute Services • Grid Data Mining for Astronomy • Williams, Djorgovski, Graham, Jacob, Katz, Mahabal, Miller et al • Architecture • Persistent Grid Services • VO Registry • Virtual Data • Distributed file system Grid Desktop Triana, Viper, ...? Teragrid, UK Grid, ...?

  33. GRIST Objectives Compute Services • Workflow • Portal, Batch, Grazing • Virtual Data • VO Data services • OpenSkyNode for crossmatch • Palomar-Quest exposure • SIAP exposure • Mining Palomar-Quest • Hi-z Quasar candidates • Cluster/outlier/correlation • Image Processing • Hyperatlas library • Faint source detection • Education • Grid computing and massive data • Teragrid

More Related