1 / 24

NPP Atmosphere PEATE Peer Review

NPP Atmosphere PEATE Peer Review. APSPS: Atmosphere PEATE Science Processing System Liam E. Gumley, Hank Revercomb, Scott Mindock, Steve Dutcher, Robert , Andy Heidinger, Mike Pavolonis, Richard Frey, Bryan Baum, Paolo Antonelli, Bruce Flynn, Rick Jensen.

akiko
Download Presentation

NPP Atmosphere PEATE Peer Review

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NPP Atmosphere PEATEPeer Review APSPS: Atmosphere PEATE Science Processing SystemLiam E. Gumley, Hank Revercomb, Scott Mindock, Steve Dutcher, Robert , Andy Heidinger, Mike Pavolonis, Richard Frey, Bryan Baum, Paolo Antonelli, Bruce Flynn, Rick Jensen Space Science and Engineering CenterUniversity of Wisconsin-MadisonAugust 2007

  2. What is the Atmosphere PEATE?

  3. The PEATE is Part of NASA NPOES SDS program. One of five. Ocean NASA Goddard Land NASA Goddard Ozone NASA Goddard Sounder NASA JPL Atmosphere UW-Madison SSEC

  4. NPP Sensor Payload Includes: Visible Infrared Imager Radiometer Suite (VIIRS) Cross-Track Infrared Sounder (CrIS) Advanced Technology Microwave Sounder (ATMS) Ozone Mapping & Profiling Sensor (OMPS) Launch is expected in 2010 The NPP Mission for NASA: Continue the Climate Record • NPP is intended to continue the climate record established by Terra and Aqua. • NPP Science Products are known as “Environmental Data Records” or EDRs. • NPP EDR Algorithms were developed by industry, not by the NPP Science Team. Land EDRs Surface Albedo Land Surface Temperature Snow Cover and Depth Surface Type Active Fires Ice Surface Temperature Vegetation Index Aerosol Optical Thickness Aerosol Particle Size Ocean EDRs Chlorophyll Sea Surface Temperature Ozone EDRs Ozone Total Column/Profile Atmosphere EDRs Cloud Mask Suspended Matter Cloud Cover/Layers Cloud Effective Particle Size Cloud Top Height Cloud Top Pressure Cloud Top Temperature Cloud Base Height Cloud Optical Thickness Sounder EDRs Vertical Moisture Profile Vertical Temperature Profile Vertical Pressure Profile

  5. Atmosphere PEATE Science Processing System (APSPS)

  6. Algorithm Lifecycle Algorithm can come from anywhere. Once qualified, the algorithm can be applied from ARM.

  7. Why and How is the PEATE different from other systems at SSEC? • Continuous, automated, testing of software systems. • Continuous, automated, science product generation. • Continuous, automated, global science product evaluation. • Rapid retrospective global product generation (100 > real-time). • (e.g., to create CDRs, or to evaluate alternative EDR algorithms). • Simple Algorithm Integration • Few Algorithm Implementation Restrictions. • No science to operations transformation required. • Rule based process triggers. • Truly scalable system • SOA - Service Orientated Architecture

  8. Science Processing System Trade Studies and Key Decisions • We examined NASA Ocean SDPS and MODIS Land Processing Systems. • Lessons learned: • Recipe-based approach to running science algorithms (system doesn’t care what the algorithm is, as long as it knows how to assemble the ingredients to make the recipe) • Cluster of compute resources (no need for a large shared memory computer) • Decouple the components of the processing system (store, compute, distribute) • Use commodity hardware/software (e.g., Rackmount Intel/AMD servers, Linux) • Key Design Decisions: • Create a system where individual components have loose dependencies on each other. • Leverage existing cluster processing hardware infrastructure and knowledge base. • Use established software technologies where possible to speed development time (e.g, Eclipse, SOAP) • Create a system which is scalable, efficient, and cost effective.

  9. Atmosphere PEATE Design Goals • Use Commodity Hardware - x86 • Use Open Source Software - Linux, Unix like(OSX) • Scaleable: • Laptops • Desktops • Clusters • Maintainability: System lifetime spans years. • Replace/Add hardware, software, people. • Manage Dependencies. • Automated Testing • Reusability: • Software Patterns. • Development processes. • Interfaces. (Public and Internal)

  10. What technologies are being used? • Subversion : • Revision controlled system code. • Revision controlled science application code. • Revision controlled qualified binaries. • Bugzilla : • Bug tracking of system and applications. • To do work scheduling and prioritization. • Issue tracking

  11. Nightly Builds Checks out code. Builds code. Tests code with direct interfaces. Tests code with service interfaces. Tests all packages. Runs system use case tests. If everything passes, checks into distribution repository.

  12. Implementation Technologies • Java: • Established high level language. • Strict type checking. • Good tool support. (Eclipse,Ant,junit,log4j) • Good integration with Web Technologies. (Web Services,Axis2) • UML: • Software Industry Standard. (Standard Diagram Examples) • Good tool support. (Eclipse plugins) • Provides standard methods of describing/designing architectures. • XML: • Established Cross platform Technology

  13. Development Environment • Eclipse : • UML, code framing, reverse engineering. • Web Tools, web service exploring • C++, contractor algorithms, LEOCAT • Fortran, contractor algorithms, LEOCAT • Subversion • Ant + junit • Development Builds • Night Builds, Automated Tests and Distribution • Ant works inside and out of Eclipse.

  14. PEATE Software Architecture • ING - Ingest System • Brings external data into system • DMS - Data Management System • Holds Data Files • Requires Unique Names • Preserves Unique Names • CRG - Computation Resource Grid • Implements simple scheduling • ARM - Algorithm Rule Manager • Loosely Coupled • Web Services • Maintainable • Scalable

  15. Sample Activity Diagram Like Flowchart Ovals = Activity, Rectangle = Data Action A and B Decoupled Dot = start, Circle = end Swim Lanes = system boundaries

  16. PEATE Service Activities

  17. ARM - Algorithm Rule Manager Allows rule based algorithm execution. Match rules to incoming data. Uses regular expressions. Provides rule status.

  18. DMS - Data Management System All files stored in files system with original names. Database compliments file system. Web Services provide distributed access. Data Migration for maintenance.

  19. CRG - Computation Resource Grid Provides job list of algorithms and data ready to be processed. Provides status of jobs Implements simple scheduling.

  20. ALG - Algorithm host • Consumes web services. • Jobs from CRG. • Uses DMS for inputs and outputs. • Runs on nodes or available CPUs. • Obtains Binaries from subversion.

  21. ALG - Algorithm host

  22. PEATE Deployment - 9/5/2007

  23. PEATE Roadmap • Short term: • Scaling test. • ING / DMS Deployment. • DMS Recovery Techniques. • DMS Status Server. • CRG/ARM database integration. • 1.5 VIIRS cloud mask deployment. • Long Term: • Multiple DMS Services - Service Chaining • GOES Archive • Discovery Services: Advertise Data and Computational Resources. • WSIL • UDDI

  24. APSPS Variations • Single Computer: • System can run on single computer with or without Web Services • Good for debugging • System is well suited for running binaries on multi-core systems. Work Group: • APSPS can be run on group of PC. • DMS can be used to share data. Cluster: • APSPS (Alghost) can run on cluster nodes providing extra CPUs. Web Service Variations: • Different implementations of web services can exist. • Specialized ARM for McIDAS V integration. • Specialized CRG for unique job scheduling.

More Related