1 / 22

ASTRO-WISE- federation OmegaCEN

ASTRO-WISE- federation OmegaCEN. AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO. Edwin A. Valentijn. ((G)A)VO meeting. Goal: Connect raw data to end user in VO environment Exchange views on design level 3 years after Design – bound / unbound

gilon
Download Presentation

ASTRO-WISE- federation OmegaCEN

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO Edwin A. Valentijn

  2. ((G)A)VO meeting • Goal: Connect raw data to end user in VO environment • Exchange views on design level • 3 years after • Design – bound / unbound • Review the layers • Can we build a working near-VO system? • OmegaCAM VISTA Lofar SExtractor VOT GAVO AVO

  3. Programme – 2 rounds • AstroWise • Vision - Edwin • Implementation – Object maker– Danny • Prototype – demo - Ewout • Object viewer - Michiel • AstroGrid • AVO • Sextractor - AVO • GAVO

  4. VST

  5. Paranal Mid 2005 + 3 year

  6. Lofar IBM- Blue Gene/L

  7. Handling of the data is non-trivial • Pipeline data reduction • Calibration and re-calibration • Image comparisons and combinations • Working with source lists • Visualization • Science can only be archive based } ESO compliant Large Data Volume • Wide-field imaging instruments, vast amounts of data • VST sees equivalent of Southern sky 30 min exp, 300 nights/y in 3 years. 100 Tbyte of image data and Tbytes of source list data

  8. ASTRO-WISE Mission Virtual Survey System • Environment that provides systematic and controlled • Access to all raw and calibration data • Execution and modification image/calibration pipelines • Execution of source extraction algorithms- catalogues • Archiving or regenerate dynamically • federated to link different data centers • Dynamical archive continuously grows, can be used for • small or large science projects • generating and checking calibration data • exchanging methods, scripts and configuration raw pixel data  pipelines/cal files  catalogues

  9. Keys -Solution • Procedurizing • Data taking at telescope for both science and calibration data • Observing Modes: —Stare—Jitter —Dither —SSO • Observing Strategies: —Stan —Deep —Freq —Mosaic • Full integration with data reduction • Design- ADD • Data model (classes) defined for data reduction and calibration • View pipeline as an administrative problem

  10. Monitoring Photometric Calibration

  11. Calibration procedures Sanity checks Image pipeline Source pipeline Calibration procedures Quality control

  12. Bias pipeline Source pipeline Flatfield pipeline Photometric pipeline Image pipeline AstroWise Pipelines

  13. Tbyte source lists brains make the associations Link -lists as fast as possible

  14. DB – engine Oracle contract: – reference – licenses - consultancy Oracle 9i -> 10g: • federation through “Advanced replication” to evolve into cross site links (=references= pointers) • Python I/F SQL – OCI- Oracle db • Full oo support + Python to db persistency • Terabyte scalability through “partitioned tables” • Administration tools • Interoperability: STREAMS connects to others: SyBase, MySQL

  15. Share the loadAstroWise Survey System • Processing • Hardware • Beowulf processors – 32 (most cases) • Multi Terabyte disks (10 – 100) • Data reduction • Derive calibration • Run image pipeline (1 Mpx/s) • Archiving • Storage • Images (100’s Tbyte), Calibration files (10 Tbyte) • Source parameters (1-10 Tbyte) • Federate (network speed) • 5 Mb/s (24 hours/day) full replication (MegaCAM 10 Mb/s) • 200 Mb/s no replication, on-the-fly retrieval

  16. Contents of federation • Raw data • Observed images • Ancillary information • Calibration results • Calibration files time stamped • Reduced images • Single observation • Coadded images • Software • Methods (pipelines) for processing calibration • Configuration files • Source lists – catalogues • Extracted source information • Associated among different data objects

  17. Concepts of federation • Federation maintained by a single database- Oracle9i • Full history tracking • of all input that went into result • providing on-the fly reprocessing • Dynamical archive - Context as object attributes • Project: Calibration, Science, Survey, Personal • Owner: Pipeline, Developer, User • Strategy: —Standard —Deep —Freq — Mosaïc • Mode: —Stare —Jitter — Dither —SSO • Time: Time stamping VO interface • Software standards • Classes/data model/procedures • 00 – inheritance/ persistency • Python scripts/ c-libraries USER Python

  18. How to use the system? UI: Ingest raw data UI: Run standard pipelines – GUI UI: Modify pipeline/modules UI: Make/associate source lists 5LS: Query – whatever

  19. VST - Virtual Survey Telescope

  20. What ? science ? Collaborate - EU-wide • Projects • user defined qualifications • Surveys = projects • Archive studies = Surveys =projects

  21. New components • Python binding to Oracle – 5LS ۷ • Fileserver ۷ • Parallelisation code ۷ • Catalogue class/ associating HTM ۷ • Federating with Oracle Streams • User interfaces • Tell me everything tool<- >on the fly reprocessing

  22. Schedule • Hardware • OmegaCEN: 6Tbyte + servers + 8 node cluster operational ۷ • OAC , USM , Terapix + CVS federation ۷ • Software • Design – review Q2 2002- ۷ • Design – formalized July 2003 ۷ • WFI +INT Prototype operational Q2 2003 ۷ • Basic operations – Q4 2003۷ • Evaluate and prepare for mass production 2004 • Qualify and populate 2005 • Deliver survey system – satellites

More Related