1 / 21

Data Management Middleware

Data Management Middleware. NOAO Brown Bag Tucson, AZ May 13, 2008 Jeff Kantor LSST Corporation. Data Products. Pipelines. Application Framework. The DM reference design uses layers for scalability, reliability, evolution. Application Layer. Scientific Layer

mieko
Download Presentation

Data Management Middleware

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Management Middleware NOAO Brown Bag Tucson, AZ May 13, 2008 Jeff Kantor LSST Corporation NOAO Brown Bag May 13, 2008 Tucson, AZ

  2. Data Products Pipelines Application Framework The DM reference design uses layers for scalability, reliability, evolution Application Layer • Scientific Layer • Pipelines constructed from reusable, standard “parts”, i.e. Application Framework • Data Products representations standardized • Metadata extendable without schema change • Object-oriented, python, C++ Custom Software Middleware Layer Data Access Distr. Processing User Interface • Portability to clusters, grid, other • Provide standard services so applications behave consistently (e.g. recording provenance) • Keep “thin” for performance and scalability • Open Source, Off-the-shelf Software, Custom Integration System Administration, Operations, Security Infrastructure Layer Storage Computing Communications • Distributed Platform • Different parts specialized for real-time alerting vs peta-scale data access • Off-the-shelf, Commercial Hardware & Software, Custom Integration Physical Plant NOAO Brown Bag May 13, 2008 Tucson, AZ

  3. Middleware for Pipeline Processing Pipeline Control and Management • event system controls Pipeline Construction A configured pipeline Pipeline Execution • pipeline harness Data Access Services data • persistence • framework NOAO Brown Bag May 13, 2008 Tucson, AZ

  4. Where we’ve been; where we’re going • Middleware • In DC2, we proved out framework for automated processing • Pipeline Harness: framework for run application code in a parallel environment • Data Persistence framework: data I/O (Lim) • Event System: loosely coupled communications • For DC3/PDR, • Extend Harness, Persistence, & Event System to apply to more types of pipelines • Pipeline Control and Management NOAO Brown Bag May 13, 2008 Tucson, AZ

  5. Pipeline Harness • A processing framework for hosting scientific algorithms in a parallel environment • Developer creates pipeline pluggable modules in python • A module (i.e., a Stage) operates on a data-parallel chunk of data • Developer does not worry about managing parallel processes or data I/O • Provides facilities for configuration, accessing data, logging, etc. • Stages can be chained together into complete pipelines • Harness handles execution on parallel platform • Operates on a stream of data, routing data in and out of modules • Uses Persistence Framework to do data I/O • Uses MPI to exchange signals and data between nodes. • Multiple pipelines running simultaneously can coordinate via Event System NOAO Brown Bag May 13, 2008 Tucson, AZ

  6. Queue Queue Queue Queue Stage Stage Stage Slice Slice Slice Slice Queue Queue Queue Queue Queue Queue Queue Queue Stage Stage Stage Queue Queue Queue Queue Stage Stage Stage Queue Queue Queue Queue Stage Stage Stage Stage Stage Stage DC2 Pipeline Harness Pipeline Pipeline Manager Process • executes serial processing • controls the parallel slice workers Pipeline Manager Serial processing Slice Slice Slice Parallel processing Slice Parallel processing Parallel processing Slice Worker Processes • processes one data-parallel portion of the data (e.g. a CCD)‏ Parallel processing

  7. Event System • System that allows loosely coupled components to talk to each other • e.g., Control systems and pipelines • Systems running on different sites • DC2 use • Instructing the nightly pipelines that new data are ready for processing • Passing data between separately running pipelines • Capturing log messages from many nodes • Implementation • Based on widely used messaging framework • Leveraging some existing middleware that supports framework • All events captured into database NOAO Brown Bag May 13, 2008 Tucson, AZ

  8. Queue Queue Queue Queue Stage Stage Stage Queue Queue Queue Queue Stage Stage Stage Queue Queue Queue Queue Stage Stage Stage Coupling Pipelines via the Event Framework Event System Image/Detection Pipeline “New Detections available” Pipeline Manager Object Association Pipeline “New Moving Object Candidates Available” Pipeline Manager Moving Objects Pipeline Pipeline Manager NOAO Brown Bag May 13, 2008 Tucson, AZ

  9. Pipeline Harness: What we learned • Useful framework for developing pipelines • Successfully implemented and executed nightly pipelines • Need better handling of exceptions when things go wrong • Improvements needed in configuring and testing pipeline instances • Association pipeline required more data sharing across nodes • Expect to work well for data release pipelines • Good overall performance • Architecture makes it straight forward to measure time spent on different operations • Logging system provides timestamps • Input and output are separate stages from processing • Harness incurs low overhead • Gotchas: how framework can affect performance • Need care when deploying pipeline to a set of processors to balance load • Slower Nodes or data chunks that need more processing can hold back entire pipeline • When one pipeline requires information from another, their performance is coupled. NOAO Brown Bag May 13, 2008 Tucson, AZ

  10. Plans for DC3: Pipeline Harness and Control • Apply Pipeline Harness to Data Release pipelines • Non-nightly pipelines; for producing products for data release e.g., Astrometry, Deep Photometry, Data Quality Analysis, etc. • Apply Harness to less-data-parallel applications Association, WCS determination, Cross-talk correction • Requires greater amount of sharing data across nodes • Develop Pipeline Control and Management System • Configures pipeline, deploys it onto a system, executes it, monitors it, cleans up after completion. • Automated • Important for leveraging computing systems beyond the Base Facility and Archive Center • Can leverage existing Grid Middleware NOAO Brown Bag May 13, 2008 Tucson, AZ

  11. Plans for Data Challenge 3: Event System • Event Monitoring System • Carries out actions in reaction to complex conditions • Fault System: looking for signs of trouble • Issues Fault Events when… • Systems go down (fails to hear from running systems) • Pipeline stalls or runs too slowly (fails to receive sequence of status messages in required time) • Can result in corrective measures… • Alert an operator • Reschedule a pipeline • Logging System: calculating performance statistics in real time. • Eliminate post-processing analysis NOAO Brown Bag May 13, 2008 Tucson, AZ

  12. LSST Data Base Classes NOAO Brown Bag May 13, 2008 Tucson, AZ

  13. Data Access Base Classes NOAO Brown Bag May 13, 2008 Tucson, AZ

  14. Data Access Middleware Design • Goal: Flexible, reconfigurable, high-performance persistence for application objects. Also used for inter-pipeline communication. • InputStage and OutputStage handle persistence for pipelines. • Persistence object coordinates transactions. • Formatter objects for each persistable class translate C++ objects into calls to Storage objects. • Storage objects manage interactions with persistent stores: database, FITS files, Boost archives. LSST All-Hands Meeting National Center for Supercomputing Applications (NCSA) May 19 - 23, 2008

  15. Plans for DC3: Data Access • Add/modify Formatters • Objects for data release processing • ISR inputs • SDQA outputs • Build image archive • Raw and science images • Template images • Cached image products • Improve retrieval of split pixels/metadata • Keep as lightweight as possible • Remove CORAL • Investigate runtime loading of Formatters LSST All-Hands Meeting National Center for Supercomputing Applications (NCSA) May 19 - 23, 2008

  16. Additional Environment Elements • Version Control • Build System • Coding and Review Standards • Defect Tracking • Automated unit and integration testing • Requirements/Design Traceability • http://dev.lsstcorp.org/trac NOAO Brown Bag May 13, 2008 Tucson, AZ

  17. Validating the design - Data Challenges NOAO Brown Bag May 13, 2008 Tucson, AZ

  18. Validating the design - Data Challenge work products to date Full reports in LSST document archive NOAO Brown Bag May 13, 2008 Tucson, AZ

  19. Data Challenge 1 Results • Data Transfer • 70 megabytes/ second data transfers • >15% of LSST transfer rate • Pipeline Execution • 192 CCDs (0.1 - 1.0 gigabytes each) runs processed across 16 nodes/32 itanium CPUs with latency and throughput of approximately 141.5 seconds • >42% of LSST per node image processing rate • Data Ingest • 6.1 megabytes/ second source data ingest • >100% of LSST required ingest rate at the Base Facility NOAO Brown Bag May 13, 2008 Tucson, AZ

  20. Data Challenge 2 Pipeline Results • Performance Pipeline Runs • runId nVisits nAmps inputImages outputImages outputFrac • rlp0127 53 36 1908 1875 0.983 • rlp0128 62 36 2232 2228 0.998 • rlp0130 62 36 2232 2213 0.991 • Total - 108 6372 6316 0.991 Average time to process one visit* avg. time (s) 3sigma (s) Image Processing/Detection (IPD) 207.2 108.2 Moving Objects (MOPS) 52.9 4.1 Association 207.4 108.5 Note: # visits = 177 Average overhead time per visit for each pipeline avg. time (s) 3sigma (s) Image Processing/Detection (IPD) 1.17 1.03 Moving Objects (MOPS) 0.27 0.20 Association 0.34 0.26 Note: # visits = 177 * Note: Post-DC2 algorithm tweaks improved the IPD by factor of 6 NOAO Brown Bag May 13, 2008 Tucson, AZ

  21. Data Challenge 2 Data Access Results • Read one slice’s 10.3 MB Exposure and 10.3 MB template Exposure from FITS files on NFS file system - 9.0 sec • Write one slice’s 10.3 MB Exposure to FITS file on NFS file system and metadata to Database - 6.5 s • Write one slice’s 10.3 MB difference Exposure to FITS file on NFS file system - 6.4 s • Write one slice’s DIASources to database (average number of DIASources = 36) - 0.05 s • Read all DIASources from database (average number of DIASources = 1321) - 0.16 s • Write DIASource matches to database (average number of match pairs = 1082) - 0.03 s • Write new Object identifier pairs to database (average number of pairs = 537) - 0.02 s NOAO Brown Bag May 13, 2008 Tucson, AZ

More Related