Virtual Observatory Developments in Europe and in the US Rudolf Albrecht

1. Virtual Observatory Developments in Europe and in the US Rudolf Albrecht Space Telescope European Coordinating Facility European Space Agency Robert J. Hanisch Space Telescope Science Institute/NVO Paolo Padovani Space Telescope European Coordinating Facility European Southern observatory/AVO

2. Virtual Observatory: motivation • State-of-the-art observing facilities in space and on the ground produce an ever-increasing amount of high-quality data • Due to the high cost of the facilities and due to the limited observing time these data are very valuable • All facilities collect the data in digital archives and make them available for further scientific exploitation. This includes investigations not envisaged by the original observer, combining data from different programs, or examining the data for changes over time. • The Virtual Observatory is the logical next step: it allows to combine data from several such archives to perform multi-wavelength, synoptic investigations.

3. Decadal Survey Recommendation • National Academy of Sciences Decadal Survey recommended NVO as highest priority small (<$100M) project “ Several small initiatives recommended by the committee span both ground and space. The first among them—the National Virtual Observatory (NVO)—is the committee’s top priority among the small initiatives. The NVO will provide a “virtual sky” based on the enormous data sets being created now and the even larger ones proposed for the future. It will enable a new mode of research for professional astronomers and will provide to the public an unparalleled opportunity for education and discovery.” —Astronomy and Astrophysics in the New Millennium, p. 14

4. 1 microSky (DPOSS) Astronomy is Facing a Data Avalanche Multi-Terabyte (soon: multi-Petabyte) sky surveys and archives over a broad range of wavelengths Billions of detected sources, hundreds of measured attributes per source 1 nanoSky (HDF-S)

5. A PanchromaticApproach to the Universe… …reveals a more complete physical picture The resulting complexity of data translates into increased demands for data analysis, visualization, and understanding

6. NVO History • 1990s: NASA establishes wavelength-oriented science archive centers; multiple large ground-based digital sky survey projects initiated • April 1999, Decadal Survey Panel on Theory, Computation, and Data Discovery met in Los Alamos • Szalay, Prince, and Alcock coin the name “National Virtual Observatory” • November 1999, NVO organizational workshop at JHU • February 2000, 2nd NVO workshop at NOAO-Tucson • June 2000, conference held at Caltech, “Towards a Virtual Observatory” • June 2000, ad hoc steering committee formed • February 2001, AASC/NAS report “Astronomy and Astrophysics in the New Millennium” released • April 2001, proposal submitted to NSF ITR program, 17 collaborating organizations, led by A. Szalay (JHU) • September 2001, NSF announces proposal selection • January 2003, first NVO science prototypes shown at Seattle AAS

7. Virtual Observatory in Europe • AVO Project started November 2001 • Three-year, 5 M€, Phase A project, funded by the European Commission (Fifth Framework Programme [FP5]) and six organizations: ESO, ESA, AstroGrid, CNRS (CDS, TERAPIX), University Louis Pasteur, JBO • Manpower: ~ 17 FTEs/yr total, 50/50 EC and partners, ~ 1 FTE/yr ECF • Next step: EURO-VO, implementation phase (subject to EC funding)

8. AVO (Phase A) Goals • Definition of Science Reference Mission • Definition of scientific requirements • Implementation of selected science cases through demonstrations • Science Working Group established two years ago to provide scientific advice to AVO Project

9. What the Virtual Observatory is… • A suite of international standards for the discovery, exchange, intercomparison, and analysis of network-accessible astronomical data • A data access and analysis environment that exploits the emerging computation/software/data Grid • A framework for data processing that enables and encourages the re-use of algorithms • A tool for science planning: Identify gaps in coverage of parameter space. Which new missions, instruments, experiments will have largest impact? • A catalyst for world-wide access to astronomical archives • A routinely used tool of the research astronomer • A vehicle for education and public outreach

10. What the Virtual Observatory is not… • A replacement for building new telescopes and instruments • A centralized repository for data • A data quality enforcement organization

11. Portals, User Interfaces, Tools VOPlot Mirage Topcat conVOT DIS NVO Resource Discovery Registries NVO Data Access Layer HTTP, Web, & Grid Services Computational Services VOTable ConeSearch ADQL, OSQ Queries SIAP, SSAP Responses FITS,GIF,… Virtual Data Catalogs, Archives, Collections, Models VO Architecture Data Models, UCDs, Metadata

12. VOTable • Reached international agreement on VOTable V1.0 specification in April 2002 • XML-based standard with in-line data or links to external data • Utilized for basic catalog and image access protocols • Merges “AstroRes” heritage with XML flexibility • Complements FITS • Multiple I/O libraries available (Java, Perl, C++, C#)

13. Data Models • Data modeling effort aimed at defining basic data types and relationships among them • High-level entities: image, spectrum, time series, catalog • Low-level entries: quantity, resolution, time of observation • Interfaces and protocols for other VO services derived from DM relationships

14. Data Access Layer • Data Access Layer is mediator between NVO data requests and data delivery • Defined “Cone Search” protocol and have ~100 implementations • Defined Simple Image Access Protocol (SIAP) and have 20+ implementations • Specification for Simple Spectral Access Protocol in development

15. Resource Metadata and Registry • Resource Metadata describes NVO data collections, services; this metadata is collected into a Registry • Resource Identifiers are component of resource metadata; have agreed on syntax • Using Open Archive Initiative protocols for metadata harvesting • Now focusing on query mechanisms and general updating/synching options • Prototype registry utilized in science demonstration, Data Inventory Service

16. Unified Content Descriptors • UCDs provide common data dictionary for describing contents of catalogs • CDS initiative, now broadened to international VO discussion • Current discussion focusing on structure and extensibility

17. VO Query Language • Working on minimal extensions to SQL to support astronomical queries (e.g., spatial proximity)  Astronomical Data Query Language • Defining standard query service based on SDSS SkyQuery: OpenSkyNode and OpenSkyQuery • Investigating higher-level query languages; natural language • Xquery

18. Grid and Web Services • Increasing number of web services (cone search and SIAP wrappers, for example) • Registry services will be implemented as web services • Prototyped use of Grid in galaxy morphology science demonstration • ROME (Remote Object Management Environment) project provides stateful user interface to Grid-based or other compute-intensive applications • Working closely with Grid community to understand progress on Grid services, e.g., OGSA, and to determine best time to adopt

19. NVO Science Prototypes • Science prototypes guide and validate technical initiatives • NVO Year 1 • Brown dwarf candidate search • Gamma-ray burst follow-up • Galaxy morphology measurement (utilizing computational grid) • Year 1.5 • Data Inventory Service • Year 2 • Data Inventory Service with registry-based resource selection • Access to theoretical simulation (globular cluster) and virtual observations

20. Brown Dwarf Candidate Search Scientific Motivation: The search for brown dwarfs has been revolutionized by the latest deep sky surveys. A key attribute to discovering brown dwarfs is the federation of many surveys over different wavelengths. Such matching of catalogs is currently laborious and time consuming. This matching problem is generic to many areas of astrophysics. Data Resources: Sloan Digital Sky Survey (SDSS) Early Data Release (15 million objects) 2-Micron All Sky Survey (2MASS) 2nd Incremental Point Source Catalog (162 million objects) What the VO Brings: Today, doing datasets is user-intensive and is replicated by many different users. Also, the correlation of these two datasets can take years of CPU time if not done correctly. The NVO brings two key aspects to this problem. First, it removes the need for the user to download large data to their machine, making direct use of distributed data. Second, the matching algorithm used here is computa-tionally efficient and designed to give answers in minutes rather than hours; results can be returned to the user in real-time. Sloan z magnitudes vs. 2MASS J magnitudes, with brown dwarf candidates in red.  Data are from the SDSS Early Data Release and 2MASS 2nd Incremental Release. Future Prospects: Catalog matching of large datasets is a generic problem in astrophysics. Therefore, making the matching facility available to any user for use on any dataset will greatly enhance the productivity of scientists. Standard I/O formats allow developers to create tols to use the matched data and easily integrate with existing visualization and analysis tools (anomaly detector). Bringing these data together on remote machines with enough CPU to perform analysis (Grid technology) will allow cross-comparisons of unprecedented scale.

21. As a T dwarf becomes cooler (i.e., methane and water absorptions increase) or more distant… • SDSS detects it only at z’ band • 2MASS detects it only at J band

22. Demo Leads to Discovery! New brown dwarf candidate confirmed spectroscopically with Keck Observatory

23. January 2004 AVO Demo Overview • Two scenarios: • Extragalactic: Obscured (Type 2) Quasars • Galactic: Classification of Young Stellar Objects (YSO) • Multiwavelength, heterogeneous, and complex data: VLA, CGPS, ISO, 2MASS, USNO, 2.2m/WFI, VLT/FORS, HST/ACS, XMM, and Chandra (images, spectra, and catalogues) • Access to any VO-compliant data: seamless and transparent access to ESA ISO & XMM archives and ESO data products • AVO: from First Light to First Science!

24. January 2004: AVO First Science redshift = 3.046 Ly C IV

25. January 2004: AVO First Science ~ 30 new obscured QSOs in GOODS CDFS+HDFN x 5 increase

26. VO connecting all European astronomical data centres and providers The Future Grid+Net

27. Persistent • Data Standards • User support • IVOA link • Distributed • R&D • VO Technologies for new projects and facilities 2005+

28. International VO Alliance

29. International VO Alliance • The IVOA brings together the astronomers, developers, and managers of the VO initiatives world-wide • Agreements on standards for data access (VOTable, catalog queries, image retrieval, resource descriptions, etc.) • Coordination of development activities • Sharing of software • Sharing of experience • 14 participating organizations: Astrogrid, AVO, US-NVO, VO-Australia, VO-Canada, VO-China, VO-France, VO-Germany (GAVO), VO-Hungary, VO-India, VO-Italy (DRACO), VO-Japan, VO-Korea, VO-Russia • http:www.ivoa.net

30. The VO Vision • The VO is the “semantic web” for astronomy (Tim Berners-Lee) • The VO democratizes astronomical research • The VO brings the universe to your desktop • The professional astronomer • Graduate students • Undergraduates • K-12 • Amateurs • The public (the taxpayers) http://us-vo.orghttp://www.euro-vo.org/