The Virtual Observatory and Ground-Based Data

1. The Virtual Observatory and Ground-Based Data John Huchra Harvard-Smithsonian Center for Astrophysics http://cfa-www.harvard.edu/~huchra http://cfa-www.harvard.edu/~huchra/zcat

2. “We have plenty of information technology, but not enough information.” S. Harris

3. Abstract: • The era of extremely large, public databases in astronomy is upon us. Such databases are opening the field to new research and new researchers. However it is important to ensure that the resources are available to properly archive ground-based astronomical data, and to include the necessary quality checks, documentation and calibrations.

4. A Virtual Observatory (VO) without proper archives will have limited usefulness. This also implies that with limited resources, not all data can or should be archived. NASA already has a good handle on US space-based astronomical data. Agencies and organizations that operate astronomical facilities, particularly ground-based observatories, need to plan and budget for these activities now.

5. We should not underestimate the effort required to produce high quality data products that will be useful to the broader community. • Currently the best way to fill archives is with data from surveys. • This will continue to be the case for most ground-based observatories in the near future.

6. Astronomy from 1970 to 2003 • Astronomy has been transformed by: Mainframes workstations Print media web Plates Digital detectors Punch cards Tapes Hard disk Mail Fax e-mail and this transformation is accelerating!

7. There are both new scientific fields and new ways of doing science. We are being driven by larger and larger projects and also the cost of doing business. • Cost is important! A key and *recurring* recommendation of the Astronomy Decadal Surveys has been to ensure the funding for infrastructure (read, in part, archives) and for the actual cost of doing science.

8. The Virtual Observatory • In recognition of this and of several other contributing factors (e.g. more papers are now being written with HST archival data than with newly taken data), the most recent US Decadal Survey ranked as its highest priority small project (both ground- and space-based) the creation of a National Virtual Observatory

9. The NVO is now under development and is to have 4 main functions: 1. Integration of Major Archives 2. Advanced Services 3. Standards and tool development 4. Education and Public Outreach and should be coordinated but distributed.

10. The NVO should/will be comprised of several “things”: A. Data Archives B. Software/Algorithm Suites C. Hardware D. People (a.k.a. Human Capital)

11. Filling the NVO: • What makes up the Data Archives? 1.Historical Archives (The Plate Stacks, NSSDC) 2. Knowledge Bases/Environments (including print & electronic media, e.g. ADS, NED, SIMBAD…) 3. Survey Data (DPOSS, 2MASS, FIRST, … 2dF, ROSAT, SDSS) 4. Mission Data (HST, ROSAT, IUE … JWST) 5. General Data (NOAO, ESO, NRAO, … Gemini, Keck, ALMA, GSMT)

12. Some of these items have been (more or less) taken care of. (1) is no longer growing since more and more of today’s data is going into (2). • Some have been well attended to, (3) especially by NASA and its activities like MAST, HEASARC and IPAC. • Some are best described as unmitigated archival disasters (5), especially at the US private observatories !!!

13. What is the best way to fill the VO today? Surveys!

14. Not counting the ‘bits’ in Knowledge bases, the vast majority of data available to the community today comes from survey databases, especially the all-sky maps from the optical and IR digitized sky surveys (DPOSS and DPOSS2, 2MASS) measured in multiple terabytes.

15. The philosophy of surveys is changing from cataloging objects (e.g. SAO stars, the APM catalog, and even the 2MASS PSC), to serving images. The sky, digitized at the current Nyquist resolution around 1 micron, i.e. 0.1” pixels, and at full photometric resolution is approximately 100 Terabytes per bandpass.

16. There are now maps of the whole sky at all wavelengths/energies from a few kev to a meter (ROSAT, DPOSS, 2MASS, IRAS, FIRST, DPOSSII, NVSS, SUMSS). More and better maps are coming (e.g. SDSS & Galex), but some wavelengths/energies are still only poorly studied. How are surveys handled?

17. The 2 Micron All-Sky Survey My favorite example of a modern all-sky survey is 2MASS and its associated redshift survey, 2MRS. 2MASS was conceived in 1988 as a precursor to SIRTF. Its goal was to produce a map of the sky 1000 times deeper than the original TMSS, and to cover both hemispheres. The survey specifications regarding uniformity, completeness and reliability were rigorous from the start. Strict controls were in place for the life of the project to ensure that those goals were met above all other desires (beware mission creep!).

20. 2MASS Project Timeline: • 1988 Initial SMEX proposal to NASA • 1991 Second NASA/NSF proposal (ground) • 1993 Initial funding, Prototype camera • 1995 IPAC start, Pipeline Build • 1997 First light (FLWO) • 1999 First Incremental Data Release • 2000 Second DR, Observations Complete • 2003 Final Data/Catalog Release

24. 2MASS Milky Way

25. 2MASS Galaxy Key Project Goals 1. Verify Galaxy Product Uniformity and Completeness Maps Bias Studies 2. Near-IR galaxy Properties 3. Galaxy Density Field to z ~ 0.1 Bright redshifts to K < 12.2, all, ~100,000 Faint redshifts, K < 13.5, 1-in-10, 150,000 4. Galaxy Velocity Field to z ~ 0.05, CMB Dipole Tully-Fisher to edge on sprials, Dn-sigma to E’s 5. AGN DARK MATTER!

34. The 2MASS Redshift Survey • Motivation for 2MRS is simple: (a) Previous optical surveys, although densely sampled, are biased (the other use of the word) by dust, star formation, etc. Most are based on crude photographic magnitudes, those that aren’t are not very extensive and are biased against the galactic plane. (b) ongoing deeper optical surveys (SDSS , 2dF) will only cover a portion of the sky and are still biased by dust, SF, etc.

35. ( c) Previous IR surveys were based on IRAS and are not very dense (e.g. smoothing lengths of 1000 km/s are needed, which are too large to see our motions inside the local supercluster, and are also biased by star formation --- the IRAS surveys have almost no early type galaxies which are the objects that dominate dense regions. They do not trace the local baryon/mass density. 2MASS does and does so through the plane of the Milky Way.

37. Magenta V < 1000 km/s • Blue 1000 < V < 2000 km/s • Green 2000 < V < 3000 km/s

38. Red 3000 < v < 4000 Blue 4000 < v < 5000 Green 5000 < v < 6000

39. Red 6000 < v < 7000 km/s Blue 7000 < v < 8000 km/s Green 8000< v < 9000 km/s

44. Beware the warts! Extended Sources, Not All Galaxies

47. 2MASS Sky Coverage Filter

48. Dollars and Sense: • A lesson can be learned from the cost breakdown of the 2MASS project: • Hardware Two 1.3-m telescopes --- $1.5 M Two 3-channel cameras --- $1.5 M Other Hardware --- $0.5 M Personnel + Science Operations for 5 years --- $1.5 M Team Science --- $1.0 M Management --- $1.5 M Software & Archiving Pipeline ---- $16 M Archive Build & maintain --- $17 M ------------------------------------------------------- Total ~ $41.5 M Project cost was driven by the archive!

49. Whence General Data? • For well over two decades, NASA has had a policy of archiving and releasing raw and pipeline reduced data from space missions. It is that legacy which has driven the 2MASS output. It is not cheap, however. The MAST archive at StScI was developed from the original DMF (Data Management facility) DADS (Data Archive and Distribution Service) and cost between $30M and $60M to produce (depending on real vs optimistic accounting).

50. Thus the much more general problem is what to do with the general data from ground-based telescopes. Some archives exist or are planned --- ESO for the VLT and Gemini will archive their data, and those observatories currently have budgets that can handle that load. Scaling from the SAO Telescope Data Center, a proper type 3 data archive (data quality control, media updates, indexed, retrieval software, etc.) at NOAO would cost between $2M and $3M/year and a few times that for initial construction. There is not yet full support for this activity at NOAO, at the Keck Observatory or at many other large public and private facilities. Does the VO need them?