1 / 19

A Virtual Survey SYSTEM

A Virtual Survey SYSTEM. Astro-Wise NOVA/Kapteyn – OA Capodimonte ESO – Terapix – US Munich/MPE National WFI datacenters NL-I-D-Fr/ESO EU – FP5 RTD programme parallel to AVO 5 year programme -> dec 2006. Edwin A. Valentijn. Basic objectives Wide Field imaging EU.

thompsonfrank
Download Presentation

A Virtual Survey SYSTEM

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Virtual Survey SYSTEM Astro-Wise NOVA/Kapteyn – OA Capodimonte ESO – Terapix – US Munich/MPE National WFI datacenters NL-I-D-Fr/ESO EU – FP5 RTD programme parallel to AVO 5 year programme -> dec 2006 Edwin A. Valentijn

  2. Basic objectivesWide Field imaging EU Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results 100’s Tbyte of image data and 10’s Tbyte of catalogue data With production spread over EU What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs) }(O)MegaCAM

  3. status • VO conference 2002 - design • Build information system - working • Implemented, qualified WFI@2.2m, WFC@INT, MDM, OmegaCAM@ILT • >>Qualify with OmegaCAM@VST-2006 • >>tune to run Public Surveys • >>quality control • >>Optimize federation/replication

  4. new paradigmtarget processing to: -hunting /full linking -data pulling -target processing -raw data -archive -all in archive -request driven from: -waterfall/ multi-tier -data pushing -raw data processing -raw data delete -result ->archive -”releases” • raw pixel data  pipelines/cal files  catalogues • all integrated in one information system • distributed services Virtual Survey Telescope • processing GRID • Storage GRID • Methods/services GRID

  5. Astro-Wise VO Properties Benefits integrated dynamic db • on-the fly re-processing • 5LS: 5 Lines Script • All bits are traced • Administration for parallel processing compute GRID SETI@home • Global solutions –astrometry/photometry • Build–in workflow • Fully user tunable – own provided script • Context: projects/surveys, instruments, mydb • Publish directly in EURO-VO

  6. components • Procedures + Cal plan at telescope • Data model -> object model ++ ->++db • Central db ; server/clients • All I/O except images • Meta data • Source lists = catalogues + associate lists • Links = references = joints • Fileserver – distributed- via db • Python clients • CVS distributed code base - opipe

  7. Bias pipeline Source pipeline Flatfield pipeline Photometric pipeline Image pipeline Astro-Wise Pipelines

  8. Target processing:++ the make metaphor awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382') The processing chain is ReadNoise <-- Bias <-- HotPixels • > class HotPixelMap(ProcesTarget): • > > def self.make() • > class ProcessTarget(): • > > def get(date, chip) # if not exist/up-to-date then make() • > > def exist() # does the target exist? • > > def uptodate() # is each dependency up to date? Fully recursive

  9. Intra-operability peer to peer • code base + docs : CVS • Db: “Advanced Replication” evolving to streaming WRITE WRITE • READ-ONLY • READ-ONLY • REPLICATION • REPLICATION • REPLICATION • REPLICATION • READ-ONLY • READ-ONLY

  10. Contents of federation • Raw data • Observed images • Ancillary information • Calibration results • Calibration files time stamped • Reduced images • Single observation • Co added images • Software • Methods (pipelines) for processing calibration • Configuration files • Source lists – catalogues • Extracted source information • Associated among different data objects

  11. Example 5LS #Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y)

  12. Astro-Wise PORTAL

  13. Web services- object viewer

  14. QC - calibration scientist monitoring

  15. QC - calibration scientist monitoring

  16. Web services- object maker

  17. VST - Virtual Survey Telescope

  18. Lofar IBM- Blue Gene/L

  19. Thanks! Welcome at next Astro-Wise tutorial! October 2005 Netherlands www.astro-wise.org

More Related