1 / 16

BMGT in ECS Release 8.2

BMGT in ECS Release 8.2. ECHO Technical Interchange Meeting 2013 Timothy Goff timothy.d.goff @nasa.gov. Why Update BMGT?. Support multi format metadata for future missions such as SMAP . Leverage ECHO REST API More efficient – less lag time between ECS updates and ECHO updates.

giona
Download Presentation

BMGT in ECS Release 8.2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. BMGT in ECS Release 8.2 ECHO Technical Interchange Meeting 2013 Timothy Goff timothy.d.goff@nasa.gov Raytheon EED Program | ECHO Technical Interchange 2013

  2. Why Update BMGT? • Support multi format metadata for future missions such as SMAP. • Leverage ECHO REST API • More efficient – less lag time between ECS updates and ECHO updates. • Simplify BMGT system • Recent changes to ECS have rendered some of BMGT’s complexity unnecessary. • Sybase to Postgresql transition • Simplify Database interaction and avoid transitioning current highly complex stored procs to postgresql Raytheon EED Program | ECHO Technical Interchange 2013

  3. SMAP • ISO/REST and ECHO10/FTP cannot co-exist on same provider. • Two providers per DAAC – ASF approach • Move all ingest to REST – NSIDC (ECS) approach • Different processing track in BMGT for ECS and SMAP • ECS metadata translated to ECHO10 with additional metadata added • SMAP metadata left intact, with some XML subtrees injected Raytheon EED Program | ECHO Technical Interchange 2013

  4. What’s New? • REST API • Support for SMAP ISO metadata • Exports no longer packaged – more responsive synchronous upload • Short form verifications handled locally, rather than remotely. • DAAC Hosted Browse (See next slide) Raytheon EED Program | ECHO Technical Interchange 2013

  5. DAAC Hosted Browse • Browse Binary files no longer exported to ECHO • Browse Imagery provided via OnlineResource URLs • Hosted on DAAC Datapool • JPG format • Already populated for public granules • Hidden granules (e.g. AST_L1A) • Browse files will be published • Discontinue export of browse metadata • Browse exists only as a reference from science granule metadata Raytheon EED Program | ECHO Technical Interchange 2013

  6. New BMGT Architecture • Dispatcher server polls frequently to pick up export queue entries. • Entries are added by BMGT Drivers (one for each export type). • Dispatcher requests metadata from Generator, sends it to Exporter where it is HTTP PUT/DELETEd to ECHO. Response is processed by Response Handler • BMGT keeps a pool of HTTPS connections and ECHO tokens which are reused, reducing overhead. Raytheon EED Program | ECHO Technical Interchange 2013

  7. Architecture - Continued • Multiple queues dynamically allocate processing resources • Queues take resources when available, but do not drown out other queues which have work to do • E.g. manual or incremental exports will not drown out nominal automatic export • XSLT stylesheets allow augmentation of ECS and ISO conformant metadata with dynamic fields • E.g. URLs, update times, orbital Raytheon EED Program | ECHO Technical Interchange 2013

  8. BMGT Interface • Updated GUI provides monitoring capabilities • View each export, as well as errors • View batch statistics • Complex filtering mechanism • Command line interface for Manual/Verification. • Similar interface to legacy scripts • Can be called via cron. • Dispatcher and Event (Automatic) Driver run continuously as daemons with little user interaction. • Polling can be suspended/resumed via GUI. Raytheon EED Program | ECHO Technical Interchange 2013

  9. BMGT GUI Raytheon EED Program | ECHO Technical Interchange 2013

  10. BMGT GUI Raytheon EED Program | ECHO Technical Interchange 2013

  11. BMGT GUI Raytheon EED Program | ECHO Technical Interchange 2013

  12. BMGT GUI Raytheon EED Program | ECHO Technical Interchange 2013

  13. BMGT GUI Raytheon EED Program | ECHO Technical Interchange 2013

  14. BMGT Development Status • Development complete • Pre itegration testing to complete imminently • ISO/SMAP testing to commence shortly • Awaiting SMAP data in test modes • Currently working with ECHO on proper Xpaths for ingest of ISO SMAP • DAAC Hosted Browse • Publish utility updated to publish all browse • End to end testing to be performed shortly Raytheon EED Program | ECHO Technical Interchange 2013

  15. Transition -Before • NCR 80507882 (and 8050995) fix will allow DAAC to discontinue export of new browse files to ECHO prior to 8.2, if desired • Possible to start removing ECHO hosted browse links on 8.1 for public granules • 8.1 BMGT quiesced prior to 8.2 install. • Record end of 8.1 export so that 8.2 BMGT knows where to start looking for new events when it comes up. • Coordinate with ECHO to modify provider to REST/multiformat during 8.2 installation downtime. • All data exported in 8.1 BMGT will be maintained after cut-over. No need to re export. Raytheon EED Program | ECHO Technical Interchange 2013

  16. Transition - After • Use 8.2 Datapool Publish utility to publish all browse including hidden. • JPEGs will be generated • Requires 8.2 utility changes so must be performed after 8.2 install • Bring up BMGT 8.2, starting processing at end of last 8.1 cycle. • Browse Transition: • ECHO already has browse URLs for all public granules • Restart incremental verification • Trickle through entire holdings • Add browse URL for hidden granules • Remove ECHO hosted browse URL • Could start on 8.1 for public collections • Once all granules are updated, ECHO can remove hosted browse files Raytheon EED Program | ECHO Technical Interchange 2013

More Related