1 / 5

DC3 Goals and Objectives

DC3 Goals and Objectives. Jeff Kantor DM System Manager Tim Axelrod DM System Scientist. DC3 Overall Goals and Objectives. Extend DC2 Application Framework and Middleware to support Data Release Pipelines

nay
Download Presentation

DC3 Goals and Objectives

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DC3 Goals and Objectives Jeff Kantor DM System Manager Tim Axelrod DM System Scientist DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008

  2. DC3 Overall Goals and Objectives • Extend DC2 Application Framework and Middleware to support Data Release Pipelines • Expand DC2 Nightly Pipelines functionality to address LSST instrument signature removal and dayMOPS, and to improve quality of data outputs • Provide Image Processing Pipeline for control and commissioning visualization requirements • Develop first release of Data Release Pipelines (coaddition, detection, photometric calibration, astrometric calibration) • Develop first release of Science Data Quality Analysis System (SDQAS) • Continue scaled tests of data transfer, data processing, database ingest (to 15% of final LSST requirements) • Conduct first scaled tests of data query (with map reduce/bigtable and DBMS SQL) • Integrate new team members • Answer key questions for PDR as documented in DM R&D Plan DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008

  3. DC3 Top-level Project Plan DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008

  4. For PDR - Monitor and evaluate computing, storage, and network resource price/performance trends and architectures Produce computing/storage, long-haul network acquisition plans 1/1/08 – 9/30/08 LLNL (mountain/base), NCSA (archive center), SDSC (data access center), NOAO (long-haul networks) Produce cyber-security plan 1/1/08 – 9/30/08 NCSA (centers), LLNL (mountain/base) Monitor and extrapolate computing, storage, and network architectures and price/performance trends; produce bi-annual report summarizing findings 9/1/05 – 9/30/10 LLNL (mountain /base), NCSA (centers), SDSC, NOAO Model and prototype LSST infrastructure architecture and design, including scalability and reliability features. 6/1/06 – 9/30/10 LLNL (mountain/base), NCSA (archive center), SDSC (data access center), NOAO (long-haul networks) Validate scalability of infrastructure via scaled performance loading tests of pipeline processing, data ingest, and data transfer (Data Challenge 3, 15% of LSST final requirements) 1/1/08 – 9/30/08 NCSA, LLNL, SDSC, NOAO DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008

  5. For PDR - Peta-Scale Database Architecture and Analysis Test the performance of open source versions of Map/Reduce/Bigtable (hadoop/hdfs) on ingest and external queries. Characterize performance for spatial, temporal, and ad hoc meta-data based queries 1/1/08 – 9/30/08 SLAC, JHU, Google Implement provenance ingest and re-creation of data products from raw data and provenance 1/1/08 – 9/30/08 SLAC, LLNL Expand the implementation of persistence to include deep detection objects 1/1/08 – 9/30/08 SLAC Evaluate relational database and map-reduce technologies for storage and query of various types of LSST data, including images, catalogs, and meta-data. Evaluate open source versus commercial technologies. 1/1/08 – 9/30/08 SLAC, Google R&D Plan contains 15 items with PDR significance DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008

More Related