Virtual Observatories

Virtual Observatories Wolfgang Voges Max-Planck-Institut für extraterrestrische Physik Garching Workshop ‘‘Astronomie mit Großgeräten‘‘ Am 17.Oktober 2003 in Potsdam Wolfgang Voges

Virtual Observatories Overview: Historical roots * What’s happening in the world: IVOA European VO-activities German VO-activities (GAVO) Federation of local data-sets Next-generation search engine Grid Theory in GAVO Outlook * Viewgraphs partly copied from other presentations Workshop ‘‘Astronomie mit Großgeräten‘‘ Am 17.Oktober 2003 in Potsdam Wolfgang Voges

Historical remarks Virtual Observatories • VO meeting at Caltec in Pasadena (June 2000) • Astronomers mostly from the US • Very enthusiastic talks • Big vision of the future • Foundation of the NVO (US) • Since then similar meeting in Europe (Garching) • Foundation of national European and later • Other VOs Wolfgang Voges

Data  Knowledge ? The exponential growth of data volume (and complexity, quality) driven by the exponential growth in information technology … … But our understanding of the universe increases much more slowly -- Why? • Methodological bottleneck  VO is the answer • Human wetware limitations …  AI-assisted discovery  NGVO? Wolfgang Voges

How and Where are Discoveries Made? • Conceptual Discoveries:e.g., Relativity, QM, Brane World, Inflation … Theoretical, may be inspired by observations • Phenomenological Discoveries:e.g., Dark Matter, QSOs, GRBs, CMBR, Extrasolar Planets, Obscured Universe … Empirical, inspire theories, can be motivated by them New Technical Capabilities Observational Discoveries Theory (VO) IT/VO Phenomenological Discoveries:  Pushing along some parameter space axis VO useful  Making new connections (e.g., multi-)VO critical! Understanding of complex astrophysical phenomena requires complex, information-rich data (and simulations?) Wolfgang Voges

Why is VO a Good Scientific Prospect? • Technological revolutions as the drivers/enablers of the bursts of scientific growth • Historical examples in astronomy: • 1960’s: the advent of electronics and access to space Quasars, CMBR, x-ray astronomy, pulsars, GRBs, … • 1980’s - 1990’s: computers, digital detectors (CCDs etc.) Galaxy formation and evolution, extrasolar planets, CMBR fluctuations, dark matter and energy, GRBs, … • 2000’s and beyond: information technology The next golden age of discovery in astronomy? VO is the mechanism to effect this process Wolfgang Voges

A Schematic Illustration of the VO-Based Science Primary Data Providers Secondary Data Providers VO Surveys Observatories Missions Survey and Mission Archives Follow-Up Telescopes and Missions Data Services --------------- Data Mining and Analysis, Target Selection Results VO as an integral part of the whole system … Digital libraries Wolfgang Voges

The Changing Style of Observational Astronomy Wolfgang Voges

This quantitative change in the information volume and complexity will enable the Science of a Qualitatively Different Nature: • Statistical astronomy done right • Precision cosmology, Galactic structure, stellar astrophysics … • Discovery of significant patterns and multivariate correlations • Poissonian errors unimportant • Systematic exploration of the observable parameter spaces (NB: Energy content = Information content) • Searches for rare or unknown types of objects and phenomena • Low surface brightness universe, the time domain … • Confronting massive numerical simulations with massive data sets Wolfgang Voges

Panchromatic Views of the Universe:A More Complete, Less Biased Picture Radio Far-Infrared Visible Dust Map Wolfgang Voges Visible + X-ray Galaxy Density Map

Examples of Possible VO Projects: • A Panchromatic View of AGN and Their Evolution • Cross-matching of surveys, radio to x-ray • Understanding of the selection effects • Obscuration, Type-2 AGN, a complete census Evolution and net energetics, diffuse backgrounds • A Phase-Space Portrait of Our Galaxy • Matching surveys: visible to NIR (stars), FIR to radio (ISM) • A 3-D picture of stars, gas, and dust, SFR … • Proper motions and gas velocities: a 6-D phase-space picture Structure, dynamics, and formation of the Galaxy • Galaxy Clusters as Probes of the LSS and its Evolution • Cluster selection using a variety of methods: galaxy overdensity, x-rays, S-Z effect … • Understanding of the selection effects Probing the evolution of the LSS, cosmology Wolfgang Voges

Exploration of new domains of the observable parameter space: the Time Domain Faint, Fast Transients (Tyson et al.) • Existing and Forthcoming surveys:  Microlensing experiments (OGLE, MACHO …) • Solar System patrols, GRB patrols …  DPOSS plate overlaps (Mahabal et al.) •  QUEST-2 and NEAT at Palomar • … and many, many others … •  The future: LSST ? Megaflares on normal MS stars (DPOSS) Wolfgang Voges

Data Mining in the Image Domain: Can We Discover New Types of Phenomena Using Automated Pattern Recognition? (Every object detection algorithm has its biases and limitations) – Effective parametrization of source morphologies and environments – Multiscale analysis (Also: in the time/lightcurve domain) Wolfgang Voges

Exploration of observable parameter spaces and searches for rare or new types of objects Wolfgang Voges

Advantages of a Virtual Observatory • new, more, better, faster, and easier science • comparative analysis of multi-instrument data, permit new approaches to research and multi-wavelength exploration, opening discovery capabilities not otherwise possible This is clearly the primary mandate of all VO efforts • minimise redundancy: data collected by a single telescope / instrument can be re-used multiple times by different teams and for different scientific purposes • data integrity: data are archived and documented in a controlled and uniform fashion, ensuring long-term scientific usage • improving calibrations and creating more higher-level data products to make data more science-ready Wolfgang Voges

Advantages of a Virtual Observatory • interoperability of archives: - strengthening connections to other archives, catalogues and abstract services for broader research parameter space and links to the literature • advancing technologies for computers, networks, data compression, and storage media: - to retrieve and analyse more information more readily at lower cost • efficient serving of data to the public: • - there will be different levels of end-user from professional astronomers to interested (high-school) students and enthusiastic amateurs – many of whom may undertake projects which are simply unrealisable by large institutes • data-mining with new software tools and new catalogues of object properties: - to enable higher-order research based on questions posed in scientific terms Wolfgang Voges

Advantages of a Virtual Observatory • improving the preparation, development, building of new ground-based and space-based projects • improving new observation proposals • comparison of real data with simulated data – to provide feedback to new insights, new models, new physics Wolfgang Voges

International Virtual Observatory Alliance Korea, Japan, China, Australia, India, Russia, Hungary, Italy, France, Germany, Europe (ESO++), Canada, USA Wolfgang Voges

What’s happening in the world: IVOA Virtual Observatories International STANDARDS are needed Registry Data-Models VO-Table VO-Query Uniform Content Descriptors (UCD) Simple Image Access (SID) GRID-standards Tools e.g. data-mining Wolfgang Voges

European VO-activities Virtual Observatories In the AVO (euro-vo.org) under the leadership of ESO/ESA the following institutes/groups are collaborating: ESO ESA/STECF University of Edinburgh CDS Strasbourg University Louis Pasteur Centre National de la Reserche Scientifique Delegation Paris The Victoria University of Manchester GAVO (RDS:MPE,AIP,HS,MPA) Wolfgang Voges

German Astrophysical Virtual Observatory Matthias Steinmetz (Co-I) Harry Enke, Detlef Elstner Astrophysikalisches Institut Potsdam Dieter Reimers, (Co-I) Dieter Engels, Peter Hauschildt Hamburger Sternwarte Simon White, Anthony Banday, Volker Springel Max-Planck-Institut für Astrophysik, Garching Other institutes are most welcome to join >>>www.g-vo.org<<< GAVO-Team: Wolfgang Voges (PI) Hans-Martin Adorf, Gerard Lemson, Achim Bohnet, Joachim Paul Max-Planck-Institut für extraterrestrische Physik, Garching Wolfgang Voges

Why do we need a German AVO? • to remain internationally competitive (proposals, data utilisation, quality of science output) • to make available VO services to everyone and provide support for the science community and public in Germany • to prepare and maintain datasets obtained from German facilities for GAVO and IVO • to establish a network, within which the needs of the German science community and public are coordinated • to obtain financial support from German agencies for such a national task Wolfgang Voges

Activities and responsibilities of partners Wolfgang Voges

Activities and responsibilities of partners • Main goal is science driven, but it will drive science, too • fast access to all kinds of astronomical and related data • capability to use highly sophisticated software tools for new studies • GAVO will provide interoperability of distributed archives over a high speed network through a set of interface/infrastructure tools • GAVO ultimately will be incorporated into larger IVO federation • Astronomical institutes will require expert data centres of different local character • e.g. for providing key data archives, documentations, “simple” analysis-, correlation- and visualisation tools • - computer science groups will develop data handling and novel analysis tools and are responsible for their maintenance Wolfgang Voges

Activities and responsibilities of partners • university institutes will be able to use GAVO for teaching and will provide a simple gateway to the public and to schools • the “service community” will be responsible for designing and developing the interface/infrastructure tools necessary for communication between the users Wolfgang Voges

Archive publication through GAVO • ROSAT source catalogs published in IVOA compliant manner: • simple cone search • webservices • RASS Photons stored in PostgreSQL database • Spatial index using HEALPix • Cone search, webservices • Federation: fast match between ROSAT source catalogues and RASS photons. • Published first proposal for unified datamodel to serve as an ontology for the IVOA. • Plans: • Extend query capabilities • Publish ROSAT fields and pointed observations • Federate with SDSS mirror at MPA • Federate ROSAT catalogues with external catalogues for classification of X-ray sources (in collaboration with ClassX team). Wolfgang Voges

The local GAVO activities Top priority during initial stage of development • federation of local key datasets and provision of key applications • ROSAT, SDSS, Planck, RAVE • development of meta-data standards, especially for simulations • development of common query tools for the local archives • need ability to query/compare both real sky and simulated data • post-processing tools - must be platform-independent • installation of visualisation packages existing software provides a strong foundation to allow extension to different types of data and archives Wolfgang Voges

The local GAVO activities Technical challenges and requirements • archive standards: rules for ingestion, data quality, associated meta-data schema, data attributes • archive maintenance/evolution: migration of data with new technology and enhancements in data attributes • meta-data requirements/standards for different data-sets (observations, simulation, calibration) • federation of archives, interoperability • high-speed networking, streaming formats for data • distributed processing power – GRID concept • seeking active cooperation with industry in many of these areas Wolfgang Voges

Next Generation Search Engine • Download Manager • Retrieves data from multiple distributed databases • Matcher • Matches sources based on sky-position (astronomical sources have no unique identifier) • Classifier • Uses multi-wavelength data for identification purposes Wolfgang Voges

NextGen Search Engine (cntd.) Wolfgang Voges

Download Manager • Features • Tool … • … accesses registry at JHU • User … • … selects distributed catalogues • … specifies one or more sky-locations • Tool … • … queries remote catalogues • … retrieves datasets for further processing Wolfgang Voges

Download Manager (cntd.) Wolfgang Voges

Matcher • Essential for data mining • Prototype features • Performs “fuzzy” match between pairs of source lists from different catalogues • Computes probability of real match • Moving matcher into production use • Collaboration with Canadian Virtual Observatory (CVO) • Feeding ROSAT source matches to classifier Wolfgang Voges

Matcher (cntd.) Wolfgang Voges

Classificationof ROSAT X-ray sources • ClassX: in collaboration with US-VO • Requires data from several large sky-surveys • X-ray: ROSAT (BSC + FSC) • Optical: SDSS, USNO B1.0 • Infrared: 2MASS • Radio: FIRST, NVSS, SUMSS • True multi-wavelength VO-application Wolfgang Voges

Correlation of ROSAT and SDSS data Probing the large-scale structure of the universe with clusters of galaxies • Project outline: • (ideally on single photon/galaxy basis) (Schuecker, Boehringer,Voges) • identify a sample of galaxy clusters using X-ray/optical correlation • >>>>>> see next 3 viewgraphs • utilise optical multi-colour images (u,g,r,i,z) to derive photometric redshifts • quantify completeness and selection limits by comparison to simulated cluster data • search for IR correlation and quantify galaxy evolution in clusters • determine correlation with radio surveys to identify the frequency of radio galaxies and AGN in clusters, search for radio halos • optical correlation to identify AGN in clusters • identify correlations with microwave/sub-mm data to search for the Sunyaev-Zeldovich (SZ) effect (distance measurements, velocity determination) Wolfgang Voges

Search for clusters of galaxies Maximum likelihood contours based on RASS-3 X-ray photons (upper panel, 1, 2 .. contours), SDSS galaxies (middle panel, >10), and the combined maximum likelihood contours of RASS-3 and SDSS data (lower panel, >10). Crosses mark the position of the deepest X-ray clustersamples available sofar (REFLEX-2, X-ray flux limit 1.8 .10-12 erg s-1cm-2). Squares mark the position of the X-ray clusters of the final sample. Wolfgang Voges

Search for clusters of galaxies Cumulative X-ray cluster number counts of the RASS/SDSSclusters (histograms) for a log-likelihood minimum of 15 applied to SDSS data (continuous line), for 25 (lower dashed line), and for 5 (upper dashed line). The RASS/SDSS cluster counts are compared with results obtained with other surveys (squares: RDCS, REFLEX,REFLEX-2). No corrections for variations of the angular survey-sensitivity (effective survey area) are applied to the RASS/SDSS and REFLEX-2 data. The figure shows that with the combination of RASS and SDSS data a 10 times deeper X-ray flux limit can be obtained compared to traditional X-ray cluster surveys like REFLEX. Wolfgang Voges

Search for clusters of galaxies General remarks: Our first results are quite important as a guideline for future X-ray missions like ROSITA and DUO. For the latter, about 8,000 X-ray clusters are expected to be detectable with standard methods. The application of the matched-filter technique allows the extraction of about 30,000 X-ray clusters with DUO. Such large numbers of X-ray clusters are needed for precise tests of the dark energy and alternative gravitational theories. Wolfgang Voges

Correlation of ROSAT and SDSS data • VO functionality required: • federation of relevant datasets including interchange/merging of meta-data • identification of candidate cluster members by appropriate query applications to optical and/or X-ray catalogues • acquire multi-colour information to determine photometric redshifts • identification of candidate radio galaxy cluster members by querying radio catalogues with search criteria (e.g. location) tailored to the derived cluster sample • identification of associated SZ by specific queries in existing catalogues; if no candidate SZ cluster can be identified apply suitable search algorithms to Cosmic Microwave Background (CMB) sky surveys to determine effect or limits thereon • visualisation of multi-wavelength cluster data • deprojection algorithms to allow study of morphology in survey data • conversion of simulation data to the space of observable parameters • 3D-interface for visualisation (to schools) Wolfgang Voges

Query example Correlation between radio, IR, optical and X-ray sources Search for SDSS QSO´s with 1 < z < 2, which are variable in one of the 4 wavelength-bands • search-engine: • which datasets do exist and in which archive? • multiple availability? parallel handling on different servers • data available for different epochs? comparison of fluxes, light curves, period-search • Source catalogues available? • radio: FIRST, NVSS, … • IR: IRAS, 2MASS, … • Optical: SDSS, Tycho-2, HST-GSC, USNO-2… • X-ray: ROSAT, ASCA, XMM-Newton, Chandra, … Wolfgang Voges

Query example • if no catalogue entry exists •  postage-stamp (pixel-image with/without contour-lines) • creation of light curves, fluxes, spectra, etc. by using original-data • high demand of CPU? •  GRID implementation - search for publications on derived variable SDSS QSO objects Wolfgang Voges

Grid Technology for GAVO GAVO grid : integration of all GAVO-workstations at MPE and AIP into a cluster Basic services on GAVO-grid: CertificationAuthority provides single-sign-on/access-all facility via proxy-ca Resource discovery and runtime information, network-weather for the grid Running distributed applications Running MPI-based applications on the GAVO-cluster Wolfgang Voges

Theory and GAVO Virtual Observatories Simulations Comparison of Simulations and observations Wolfgang Voges

Simulations in the Virtual Observatory • Merging of the Milky Way with the Andromeda galaxy (M31) • (3 Mio particles, cluster of 16 CPU’s, 1 week of CPU time) • (30 k particles would need 25 minutes of cluster-CPU time) Wolfgang Voges

Simulations in the Virtual Observatory COMA type cluster of galaxies (>1000 galaxies, 10^15Msolar, 7 Mio particles) 8 CPUs, runtime:2 days; Gravitation, Hydrodynamics, not included: cooling, star formation (Volker Springel, MPA) Wolfgang Voges

The Role of Datasets fromTheoretical Astrophysics • Direct Comparisons with Observations • Verification (or not) of Models • Data Mining for Both Observations and Theory • New Applications • Buried Physics • Resource for Education and Outreach Wolfgang Voges

Theory and the VirtualObservatory • Size of Datasets Appropriate to VO • Large Scale Simulations, Parameter Space Libraries Imply 10GB – 10 TB Datasets • Rich Complement to Observational Side • Same/Similar Tools as for Obs. Datasets • Use of VO Infrastructure • Grid Technology, Portals, etc. Wolfgang Voges

Conclusions • Theoretical Astrophysics is an Essential Part of the Virtual Observatory Concept • Provides Benefits to Theorists • Provides Benefits to Observers • Provides Benefits to Education/Outreach • Drives New Science Wolfgang Voges

GAVO efforts on the TVO • Published IVOA whitepaper on “Theory in the VO” • Leading theory subgroup in IVOA data modeling effort. • Chair in IVOA special interest group on theory • Plans: • Publish simulation archives at AIP, LMU-Obs., andMPA • Collaborate with UPitt on publishing services on theoretical datasets (NSF grant proposal) • Collaboration with Technion Haifa to publish observed and simulated Ly- forest spectra (GIF proposal) • Collaboration in RTN proposal for comparison of simulated and observed X-Ray clusters (Boehringer et al@MPE) Wolfgang Voges

Virtual Observatories