1 / 46

NPP Atmosphere PEATE Preliminary Design Review

NPP Atmosphere PEATE Preliminary Design Review. Part 2: Science Processing System Scott Mindock. Presented by the Atmosphere PEATE Team Space Science and Engineering Center University of Wisconsin-Madison 10 April 2007. Definitions. UML - Unified Modeling Language

brendajbrown
Download Presentation

NPP Atmosphere PEATE Preliminary Design Review

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NPP Atmosphere PEATEPreliminary Design Review Part 2: Science Processing SystemScott Mindock Presented by the Atmosphere PEATE TeamSpace Science and Engineering CenterUniversity of Wisconsin-Madison10 April 2007

  2. Definitions UML - Unified Modeling Language Use Case - Captures requirements at functional level Activity - Describes the steps of a use case Package - Show software structure and dependencies Class - Show software structure and dependencies

  3. Sample Activity Diagram Like Flowchart Ovals = Activity Rectangle = Data Action A and B Decoupled Dot = start Circle = end

  4. Presentation Format Key Features and Flow Requirements

  5. Design Goals Maintainability - System lifetime spans years Reusability - Subsystems and design patterns Testability - Subsystem dependencies are managed Scalability - Design must scale to larger and smaller systems

  6. Design Methodology Learn from others - IDPS, Clusters, Ocean PEATE, Disney Parkwide Leverage existing proven technologies - Use successful open source Prototype key features - Demonstration Projects Model System - UML Activity Model, Java System Decouple Major Systems - Manage dependencies, Define interfaces

  7. Atmosphere PEATE Science Processing System (APSPS)

  8. Software Functional Grouping • APSPS Software Subsystems • Ingest • Data Management System • Algorithm Rule Management • Computation Resource Grid • Software Development Support • Configuration Management • Science Investigation Support • Product Creation

  9. APSPS, Atmosphere PEATE Science Processing System DMS: Data Management System Stores Data CRG: Computational Resource Grid Processes Data ARM: Algorithm Rule Manage Applies Product Rules to Data ING : Ingest System Brings Data into System

  10. Cloud EDR Assessment Scenario

  11. Cloud EDR Assessment Scenario

  12. ING Supports pull ingests Supports push ingests Verification functions localized Catalog entry function localized Notices missing or late data

  13. ING Requirements

  14. APSPS, Atmosphere PEATE Science Processing System DMS - Data Management System CRG - Computational Resource Grid ARM - Algorithm Rule Manager ING - Ingest System

  15. Cloud EDR Assessment Scenario

  16. DMS Provides well-defined interface DMS is autonomous Provides storage Provides catalog

  17. DMS Executive Purges old data Watchdogs Catalog Server Watchdogs Delivery Servers Advertises catalog

  18. DMS Executive Requirements

  19. DMS Catalog Catalog of files in DMS Health of DMS Information stored in database Apache Server

  20. DMS Delivery Supports raw access protocols (e.g. FTP,RSYNC,HTTP) Support intelligent access (e.g. OpenDAP, ADDE)

  21. APSPS, Atmosphere PEATE Science Processing System DMS - Data Management System CRG - Computational Resource Grid ARM - Algorithm Rule Manager ING - Ingest System

  22. Cloud EDR Assessment Scenario

  23. CRG Provides well-defined interface Accepts job request Provides Job Status Monitor Job State Scalable Testable

  24. CRG Host One or more per node Retrieves data Retrieves algorithm Runs Algorithm Saves results

  25. CRG Host Requirements

  26. APSPS, Atmosphere PEATE Science Processing System DMS - Data Management System CRG - Computational Resource Grid ARM - Algorithm Rule Manager ING - Ingest System

  27. Cloud EDR Assessment Scenario

  28. ARM Assigns jobs to CRG Monitors data in DMS Monitors the status of jobs in CRG Applies rules to data Generates notifications Volatile logic lives here

  29. ARM Manager Match Rules and Algorithms to available data. Allow users to start processing. Monitor job progress. Implements volatile logic.

  30. ARM Manager Requirements

  31. ARM Monitor Monitors the DMS Finds new data. Finds deleted data. Uses metadata encoded in filenames. Provides scalability.

  32. ARM Monitor Requirements

  33. APSPS, Atmosphere PEATE Science Processing System DMS - Data Management System CRG - Computational Resource Grid ARM - Algorithm Rule Manager ING - Ingest System

  34. Cloud EDR Assessment Scenario

  35. Configuration Management Subversion Bugzilla Apache Tomcat SSEC - Document Management Process

  36. Configuration Management Provides versioned access to software Provides defect-tracking facilities Used for source code Used for test scripts Used for binaries All open source

  37. CM Requirements

  38. Lunch Content

  39. Product Generation Support Algorithm Ingest - Importing, Developing, Wrapping Algorithm Qualification - Preparing for APSPS LEOCAT - Low Earth Orbit Cloud Algorithm Test bed

  40. Algorithm Lifecycle Algorithm can come from anywhere. Once qualified, the algorithm can be applied from ARM.

  41. Algorithm Ingest Algorithm entered into subversion Product created in bugzilla Algorithm is ported and wrapped Tests are created

  42. Algorithm Ingest Requirements

  43. Algorithm Qualification Write a script to execute algorithm Script manages execution environment Algorithm name, inputs and outputs entered into ARM

  44. Algorithm Qualification Requirements

  45. LEOCAT Reads in all data. Executes algorithm. Writes data.

  46. Cloud EDR Assessment Scenario

More Related