1 / 13

The Astro-Wise Pipeline Conceptual Design Review Roeland Rengelink rengelink@strw.leidenuniv.nl

The Astro-Wise Pipeline Conceptual Design Review Roeland Rengelink rengelink@strw.leidenuniv.nl. Requirements Development Framework Prototype. Baseline Requirements. Data Flow System User Requirements (URD) overview of the scientific requirements,

carter-burris
Download Presentation

The Astro-Wise Pipeline Conceptual Design Review Roeland Rengelink rengelink@strw.leidenuniv.nl

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Astro-Wise PipelineConceptual Design ReviewRoeland Rengelink rengelink@strw.leidenuniv.nl Requirements Development Framework Prototype

  2. Baseline Requirements • Data Flow System User Requirements (URD) • overview of the scientific requirements, • includes calibration and processing requirements • Date Flow System Calibration Plan (CP) • detailed description of all calibration procedures • Date Reduction Specifications (DRS) • specifies the data reduction tasks for the calibration and science data

  3. Additional Requirements • Multi-instrument support • Multi-site operations • Database Driven operations • Extendable • On the fly reprocessing

  4. Database Requirements • Persistence • Object Oriented DB technology • Hide schema (code is DB schema) • Hide ingest/update • Queries • Relational DB technology • Hide SQL • Dynamic (new) • User defined objects / tables • Multi-Site • Integrated bulk data storage

  5. User Interface Requirements • Developers • Use pipeline API to write scripts • Command Line users • Recipes • GUI • Pipeline operations • Quality Control • Processing status

  6. Pipeline components • Data reduction recipes • Data reduction tools • Infrastructure • Administration • User Interface Multiple developers Multiple users Multiple sites

  7. Development Framework Deliver usable code • Working • Understandable • Correct

  8. Python • Object Oriented Programming • Modularity • Reusability • Rapid Application Development • Incremental development • Inline documentation • Understandable code • Unit testing framework • Working code

  9. Unit testing • Write a test for each unit of functionality • To know that it works • To reduce the cost of modifying code • We find bugs (also in eclipse and LDAC) • But, • Requires discipline • 25 % overhead

  10. CVS • Essential for collaborative projects • Initial investment • A different mind set • Requires discipline • Commit/Update early and often • Only commit working code

  11. Astro-Wise Pipeline - Scope In: • Calibration operations defined in CP • Co-added, calibrated, pointings Out: • Mosaics • Masking, other than CCD defects • Homogenization PSF These could be covered in WP2

  12. Astro-Wise Pipeline Prototype - Scope • Reduce WFI data • Baseline implementation calibration procedures • Low-level data reduction routines • Technology demo OO database interface • Oracle9i back end • Technology demo parallel processing • New approach under development • Establish development framework

  13. Astro-Wise Pipeline Prototype - Status • See CVS • No photometric calibration yet • Limited set of recipes • Database back ends • Parallel processing • Documentation

More Related