1 / 30

Overview and Status (astrogrid.ac.uk) Nicholas Walton IoA, Cambridge

Overview and Status (http://www.astrogrid.ac.uk) Nicholas Walton IoA, Cambridge. N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p 1 Printed: 30/10/01. The Sociology of Astronomy. Continuing Collectivization Facility class (common user) instruments

ivy
Download Presentation

Overview and Status (astrogrid.ac.uk) Nicholas Walton IoA, Cambridge

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Overview and Status(http://www.astrogrid.ac.uk)Nicholas WaltonIoA, Cambridge N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p1 Printed: 30/10/01

  2. The Sociology of Astronomy • Continuing Collectivization • Facility class (common user) instruments • Central development of supporting s/w (e.g. Iraf) • Calibrated archives and access tools (e.g. IPAC) • Information services (e.g. ADS, NED, astro-ph) • Consortium projects (e.g. MACHO, SLOAN, VISTA) • Evolving Developments • Inter-operable archives, joint queries (e.g. MAST) • Data mining (exploration and analysis tools) • Information discovery tools N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p2 Printed: 30/10/01

  3. Major Background Drivers • The Growth of Data • Significant increase in number and size of telescopes • In the optical: ESO's 4x8-m VLT, Gemini's 2x8-m • In the x-ray: XMM-Newton, Chandra • In the mm: ALMA • Significant increase in size and multiplex capabilities of associated instrumentation and detectors, e.g.: • In the optical: VISTA will have a Gpixel array • In the radio: e-Merlin with data rates of 320 Gbps will generate N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p3 Printed: 30/10/01

  4. Major Background Drivers • The Growth of Data Archives • Many observatories hold multi-TB archives • New initiatives set-up to support new observing capabilities (e.g. TeraPix) • In the optical, need to ingress all-sky survey's • Whole sky at 0.1 arcsec/pix is 100TB • Increasing Importance of Archival Data • Time on expensive facilities (e.g. HST) only awarded if the archive has been searched before hand • This trend is continuing, driving observatory and user demand for more accessible archival data N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p4 Printed: 30/10/01

  5. Empowering Science Driven Observational Astronomy • Break down the 'wavelength' barriers • e.g. Greater focus on science driven proposals encouraging the use of data sets from across a wide range of wavelengths • Increase Access • e.g. East European countries may not be able to fund a major new telescope but can contribute to, and access 'Virtual Observatories' N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p5 Printed: 30/10/01

  6. Astronomical Drivers: Enabling New Science • Linking the near and distant Universe • Comparison of rest frame samples to study evolution of galaxies • Combination of UV, optical and IR datasets • Creating the 'Digital Sky' • Temporal data measures motions in the Galactic centre, probes the creation of our Galaxy • The search for extra-solar planets • The planet-transit technique using federated survey data for millions of stars N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p6 Printed: 30/10/01

  7. Astronomical Drivers: New Era of Surveys • SuperCOSMOS(UK: now till 2002) • Based on Schmidt plates - Science database ~2TB • Sloan Digital Sky Survey (US: now till 2005) • Dedicated CCD survey telescope: Science database ~10TB • UKIDSS (UK: from 2003) • IR camera on 4-m UKIRT: Science database ~30TB • VISTA (UK: from 2005) • IR camera on 4-m VISTA: Science database ~300TB • LSST (US: from 2007-8?) • Dedicated ~8-m telescope: All sky/few nights:~5000TB/yr N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p7 Printed: 30/10/01

  8. Astronomical Drivers: Pre-Discovery Mining • Investigating the progenitors of sources that show variability • Dark matter revealed by microlensing events • Planets revealed by stellar variability • Formation of neutron stars revealed by GRB's • Death of massive stars revealed by Type II SN The progenitor of SN1999gi is <9 M⊙: found from mining pre-discovery HST images. (Smartt et al, 2001, ApJ, 556, L29) N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p8 Printed: 30/10/01

  9. Astronomical Drivers: Rare Objects • Huge data sets open up the possibility to find rare objects • Those that are hard to find: • Brown dwarfs, have unique red colours • Those that are intrinsically rare • High-z quasars, stand out due to suppression of their blue colour by the Ly- forest Hi-z QSO's found from SDSS multi-colour data: the shaded area is domain of QSO's, solid line is track for increasing z (Fan et al, 2001, ApJ, 121, 31) N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p9 Printed: 30/10/01

  10. Astronomical Drivers: New Objects • Large datasets open the possibility to discover new objects • Those that have been missed before because they are extremely rare or short-lived • Those that have previously been misclassified, revealed as outliers in new parameter space correlations DPOSS group, during searches for high-z quasars, have discovered peculiar objects, this one perhaps a BAL QSO (Djorgovski et al, 2001, PASP, 225, 52) N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p10 Printed: 30/10/01

  11. AstroGrid: Meeting the Challenge • Recognising the specific need to support UK survey science and associated key datasets was the backdrop to the genesis of the AstroGrid program • Initiative driven by key dataset users and archive centres: • Cambridge, Edinburgh, Leicester, RAL, MSSL, Jodrell Bank • Major input from specialist s/w & CS groups: • QUB and RAL (Starlink) • Programme funded through the PPARC e-science line, beginning 9/01 for 3 years with ~£5M N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p11 Printed: 30/10/01

  12. The AstroGrid Project: Structure N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p12 Printed: 30/10/01

  13. AstroGrid: Project Finance & Staffing • Finance: • Nominal 3 year funding envelope ~£5M • Phase-A commitment ~£0.8M • AstroGrid funding includes element to produce the VISTA e-pipeline • Additional funding provided through the AVO • Staffing: • 10 FTE/yr (over 15 people) from Jan 2002 • Some new appointments, some displacement • Additional 3 FTE/yr employed via AVO N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p13 Printed: 30/10/01

  14. The AstroGrid Project: Timelines N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p14 Printed: 30/10/01

  15. AstroGrid: The Phase-A Study • Intensive one-year R&D phase • Science requirements analysis • Community consultation • Development of use cases • h/w & s/w benchmark tests • e.g. Beowulf vs SMP, Object vs Relational • Design benchmark problems and run tests • Pilot federations, e.g.: • SuperCOSMOS/ SDSS/ INT-WFS • Merlin/ VLA N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p15 Printed: 30/10/01

  16. Phase-A: DataBase Technology • Support data structures and indexing required by the problem domain • Application areas include: • Access to existing and interface to new archives (e.g. VISTA) • Support of data mining activities • Statistical queries & access to remote databases • Support of multi-dimensional indexing • Support for heterogeous range of data structure • Tasks include: • Development of functional requirements based on use-cases • Survey of existing commercial and open source products • Evaluation of sky indexing methods • Integration of DB's with h/w and grid technologies N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p16 Printed: 30/10/01

  17. Phase-A: Grid Technology • Evaluation of 'Grid' Technologies • Globus 'middleware' • XML & FITS: options for meta-data handling • Storage Resource Broker • Jini • CORBA & and object orientated approaches • Development of trial Data-Grids suitable for astronomy • Initially link two distributed data centres (Edinburgh, IoA) N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p17 Printed: 30/10/01

  18. Phase-A: Grid Technology Three Key Problems for Astronomical Data Grids: • Access to grid and authentication of users • Selection and location of relevant data • “Tell me what data you have relating to x, but don’t send me the data yet.” • Transport of data and metadata to the processing site • “Processing” includes visualisation. N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p18 Printed: 30/10/01

  19. Phase-A: Grid Technology Learn by experiment – the current test architecture: N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p19 Printed: 30/10/01

  20. Phase-A: Grid Technology An experimental portal: N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p20 Printed: 30/10/01

  21. Phase-A: Grid Technology Enable future development and deployment of AstroGrid and External (e.g NVO) tools: Automated detection of outliers in SDSS two colour data (Connolly et al, 2001, AJ in press) A NVO prototype of an automated discovery tool for arcs developed by A Szalay N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p21 Printed: 30/10/01

  22. Phase-A: Compute/Storage Options • Development of benchmark tests • Assessment of options • Availability • Performance • Scalability • Cost • Recommendations of Options for Phase-B • Enable integrated access to terascale computing and data facilities N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p22 Printed: 30/10/01

  23. Phase-A: The Pilot Programme • Federations of key datasets to assess performance • Large scale object catalogues: SDSS, SuperCOSMOS, INT-WFS, INT-IR data (Opt/IR) • Possible provision of limited access to pixel data • Event lists: XMM-Newton, Chandra (X-ray) • Generate event lists/spectra from image • Fourier Data (Radio) • radio/optical, postage stamp generation, deconvolution • Image and Movie Libraries (Solar) • Image transport, heliocentric coordinates, on-demand proc • Time Series Data (STP) • Tabular data, metadata, search by time period N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p23 Printed: 30/10/01

  24. Phase-A: The Pilot Programme • Problems in matching multi-λ survey data: Differences in angular resolution, s/n ratios, backgrounds, etc (Djorgovski et al, 2001, astro-ph/0108346) N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p24 Printed: 30/10/01

  25. The VISTA e-pipeline • VISTA major new UK 4-m IR-survey telescope www.vista.ac.uk • Located on Paranal and integrated into ESO VLT ops • IR camera consists of 16 off 2kx2k IR detectors • e-pipeline – use the 'grid' for distributed image processing • On the fly image processing, stacking/tiling, catalogue generation, object identification • Use for QC feedback for ESO operations N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p25 Printed: 30/10/01

  26. Towards Phase-B: The Science Case • Rapid development of science requirements • Emphasis on user input via consultation • Development of 'use cases' • Example: Formation of large scale structure • Construct unbiased cluster of galaxies sample over z to test various cosmological models of galaxy formation • Need to operate on large data sets, constructing simultaed surveys to test for selection effects • Generate predictions of observed sample properties and compare with observed samples • Development of 'White Paper' developing AstroGrid's role for Phase-B and in the context of larger VO picture N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p26 Printed: 30/10/01

  27. AstroGrid and the AVO • Astrophysical Virtual Observatory (AVO) http://www.eso.org/projects/avo • European Commission supported programme, ~€2M • 3 year Phase-A study of a European astronomicalVO • Major partners: ESO, ST-ECF, AstroGrid, CDS, Terapix, Jodrell • Has links with similar US and Australian initiatives • 3 initial areas of study • Science requirements (ST-ECF) • Interoperability (CDS) • Key relevant technologies (AstroGrid) • AstroGrid taking lead in `Key Technologies' N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p27 Printed: 30/10/01

  28. AstroGrid's AVO Workpackages • AstroGrid AVO work aligned with it's Phase-A • Grid Technologies • Compute/ Storage Technologies • Data Base Technologies • Other AstroGrid AVO inputs: • The science case, towards a 'Virtual Observatory' • Interoperability, integrate with Pilot Federations • AstroGrid receives funding for 3 EU PDRA's • Includes one specfically to Jodrell Bank N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p28 Printed: 30/10/01

  29. VO initiatives in the USA • NSF have recently funded a 5 year CalTech/JHU led VO initiative ~$10M http://www.us-vo.org • GriPhyn – The Grid Physics Network • Aiming to deploy Petascale Virtual Data Grids • Supporting a number of major experiments • CMS and ATLAS at the LHC (CERN) • LIGO http://www.griphyn.org • SDSS • SDSS SkyServer – public access http://skyserver.fnal.gov http://www.griphyn.org N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p29 Printed: 30/10/01

  30. Concluding Remarks • AstroGrid is a major new UK funded e-science initiative • In partnership with EU centres it will play a lead role in the realisation of a European Virtual Observatory • AstroGrid is poised to significantly enhance the opportunities of the UK astronomical research community N A Walton: AstroGrid Overview: PPGrid Collaboration Meeting, Nov 5-6, 2001 p30 Printed: 30/10/01

More Related