1 / 11

The Dark Energy Survey Middleware LSST Workflow Workshop 09/2010

The Dark Energy Survey Middleware LSST Workflow Workshop 09/2010. Michelle Gower NCSA o n behalf of the DES Data Management Team and DES Collaboration. DES Data Management. Observing from 2011-2016: 200 TB raw data, 4 PB of products 100 TB object catalog

zihna
Download Presentation

The Dark Energy Survey Middleware LSST Workflow Workshop 09/2010

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Dark Energy Survey MiddlewareLSST Workflow Workshop 09/2010 Michelle Gower NCSA on behalf of the DES Data Management Team and DES Collaboration

  2. DES Data Management • Observing from 2011-2016: • 200 TB raw data, 4 PB of products • 100 TB object catalog • 120+ CPU years/year of processing • Processing includes: • Removing the instrument signature • Removing artifacts (e.g. planes) • Calibrating/registering • Feature detection • Feature analysis • Monitoring/quality assessment

  3. Requirements • Level of parallelism changes through processing • Easily modified by scientists • Local and remote clusters • Work with project’s Archive system • Monitoring of jobs while they’re running • Less research/More production

  4. Processing Framework Overview Archive Nodes Target (HPC)‏ Machines Pipelines of AppModules Database Archive Portal Event Monitor DAF Notification Service Orchestration 4

  5. Middleware • Workflow • Condor DAGman • Job submission • Condor-G to pre-WS GRAM for TeraGrid resources • Condor (vanilla jobs) for local machines • File transfer • GridFTP using clients uberftp and globus-url-copy • Runtime Monitoring • Elf/Ogrescript

  6. Workflow Crosstalk CreateCor ImCorrect Make Bkgd Masking AstroRefine Remap PSFModel WeakLensing

  7. Job Generation • Run queries to get master input file lists • Stage input images to target machine • Generate Jobs: input lists, job descriptions, DAG • Setup target machine • Stage generated lists and files to target machine • Make timestamp on target machine • Target jobs run • Ingestion of new or modified files

  8. Upcoming Orchestration Work • Skipping modules • Repeatedly using set of blocks (Campaign processing) • Let condor manage vanilla jobs per machine • threaded vs serial • Finer grain restarts

  9. Notification Events System Portal Monitor Event Store AppModule (Elf/OgreScript) ActiveMQ Message Bus Science Code

  10. DES Monitor Portal • Current Technology • Drupal CMS / PHP, Javascript/Ajax • Tools for Viewing Active Processing Status • High level Alert Monitoring • Quick navigation to event logs • Quality Assurance Profiles • Histograms of QA metrics, outlier detection • Processing Timing Summary Report • Middleware level timing profile

  11. Acknowledgements • DESDM Team: • Jim Myers, Terry McLaren • Joe Mohr • Bob Armstrong, Dora Cai, Ankit Chandra, Greg Daues, Shantanu Desai, Michelle Gower, Wayne Hoyenga, Chit Khin, Kailash Kotwani • Past DESDM Team Members, DES Project Team and Collaboration Members • National Science Foundation • http://cosmology.illinois.edu/DES/

More Related