1 / 19

Tools and Services Vision FY12-FY14

Tools and Services Vision FY12-FY14. C. Thompson. SOTO Raster Processing Pipeline: FY12. gridding. n ew g ranule collection. tiling. Dataset A. unified data structure. reader/ conditioner A. config file. config file. Dataset C. reader/ conditioner B. KML pyramids. MRF files.

miles
Download Presentation

Tools and Services Vision FY12-FY14

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tools and Services VisionFY12-FY14 C. Thompson

  2. SOTO Raster Processing Pipeline: FY12 gridding new granule collection tiling Dataset A unified data structure reader/ conditioner A config file config file Dataset C reader/ conditioner B KML pyramids MRF files reader/ conditioner C full MRF generation global geoTIFFtile generation global PNG generation Dataset B KML generation floating tile repository

  3. SOTO Raster Processing Pipeline: FY13 DMAS (execute) gridding new granule collection unified data structure universal file format tiling Dataset A reader/ conditioner A config file config file universal reader Dataset C reader/ conditioner B trigger DMAS (ingest) MRF files reader/ conditioner C full MRF generation global geoTIFFtile generation Dataset B generate metadata floating tile repository

  4. SOTO Raster Processing Pipeline: FY14 DMAS (execute) gridding new granule collection tiling Dataset A unified data structure reader/ conditioner A PO.DAAC Image Generation Subsystem config file config file universal file format global & per granule hdf5-MRF generation universal reader floating MRF generation Dataset C reader/ conditioner B trigger DMAS (ingest) MRF files reader/ conditioner C full MRF generation global geoTIFFtile generation Dataset B generate metadata floating tile repository

  5. HITIDE Web Services: FY12 reader/tiler A tile DB crawlers XML ingestor reader/tiler B space time dataset(s) Dataset A reader/tiler C XML L2 OpenSearch WS L2 extract WS L2 WMS Dataset C dynamic extraction dynamic image generation imaging plugin C imaging plugin A imaging plugin B Dataset B extract plugin A extract plugin B extract plugin C

  6. HITIDE Web Services: FY13 space time dataset(s) data summaries DMAS (execute) reader A tiler A universal tiler reader/tiler A tile DB trigger reader B tiler B crawlers XML ingestor reader/tiler B space time dataset(s) Dataset A reader C tiler C reader/tiler C config file XML universal file format L2 OpenSearch WS L2 extract WS L2 WMS Dataset C dynamic extraction dynamic image generation universal writer universal file format imaging plugin C imaging plugin A imaging plugin B universal reader Dataset B extract plugin A extract plugin B extract plugin C

  7. HITIDE Web Services: FY14 DMAS (execute) space time dataset(s) data summaries reader A universal tiler tile DB reader B crawlers XML ingestor Dataset A reader C config file XML universal file format Tiled WMS L2 OpenSearch WS L2 extract WS L2 WMS Dataset C dynamic extraction dynamic image generation hdf5-MRF granule files universal writer universal file format imaging plugin C imaging plugin A imaging plugin B universal reader Dataset B extract plugin A

  8. Tiled WMS: FY12/FY13 (GWiNN) • Editable color palette (fixed bin size/data range) • WMTS compatible • Partial MRF updates • Dynamic tile resizing (“overzoom”) • Server architecture optimization • Re-organization of source code for easier deployment • Bolster documentation • Per granule MRF generation • Multiple projections • Sample clients • RESTful calls to individual tiles • Vector support • More flexible key-value pair ordering client-side solution completed high priority client-side solution high priority high priority high priority mid priority mid priority completed mid priority high priority mid priority

  9. SOTO Raster Processing Pipeline: FY14 DMAS (execute) gridding new granule collection tiling Dataset A unified data structure reader/ conditioner A PO.DAAC Image Generation Subsystem config file config file universal file format global & per granule hdf5-MRF generation universal reader floating MRF generation Dataset C reader/ conditioner B trigger DMAS (ingest) MRF files reader/ conditioner C full MRF generation global geoTIFFtile generation Dataset B generate metadata floating tile repository

  10. Tiled WMS: FY13/FY14 (PO.DAAC) • Design and development of hdf5-MRF format • Benchmark hdf-MRF against MRF access times • Build hdf5-MRF Apache plug-in colorized cylindrical or reprojected image (cached request) gridding tiling colorized reprojected image (new request) unified data structure universal file format global & per granule hdf5-MRF generation floating MRF generation colorized reprojected image (new request) universal reader colorized cylindrical image (new request) colorized cylindrical image (new request)

  11. Consolidated Web Services: FY12

  12. Consolidated Web Services: FY13

  13. Event Service: FY12

  14. Event Service: FY13

  15. FY13: SOTO • Enhance user experience • Reduce latency • Move to MRF • Modify raster image generation backend to produce daily MRF’s (Jeff, Charles) • Re-imagine vector processing backend (U/V components) utilizing MRF’s (Jeff, Charles, GWiNN) • Update GUI controls • Add flat Earth view • Integrate OpenLayers view (Christian, GWiNN) • Perform usability review • Identify use cases (Eric, Charles, Michelle) • Generate requirements and design document (Eric, Charles, Michelle, Christian) • Generate GUI mockup (Christian, GWiNN) • Implement new GUI design (Christian, GWiNN) • Improve maintainability • Streamline process execution • Port to python • Rewrite floating-tiles-to-image-tiles package (Jeff, Charles) • Rewrite dataset-to-floating-tiles package (Jeff, Charles) • Unbundle processing and GUI packages • Transition processing package into baseline imaging subsystem component • Identify data storage structures • Design and define input: universal PO.DAAC data format model (Andy, Eric, Charles, Thomas, Michelle, Ed) • Design and define output: MRF contents (Charles, Andy, Eric, Mike, Jeff) • Revise and update architecture • Generate functional requirements and design document (Eric, Charles, Jeff) • Integrate with PO.DAAC system • Connect to DMAS • Initiate processing via DMAS-Execute • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike) • Archive MRF’s via DMAS-Ingest • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)

  16. FY13: HITIDE • Stabilize system • Mitigate tile database connectivity issues • Re-architect existing tile database • Review current schema and tables (Richard, Mike) • Generate requirements and design document (Eric, Richard, Mike) • Identify revised or new schema and tables (Richard, Mike) • Build database (Richard, Mike) • Update all services connecting to database (Richard, Mike) • Improve maintainability • Upgrade installation and deployment procedures • Streamline ingest • Review existing procedures and documentation and update as necessary (Richard, Mike) • Update messaging • Add error, information, and warning notifications • Insert calls to SigEvent service throughout ingestion module (Richard, Mike) • Insert calls to SigEvent service regarding database “health” (Richard, Mike) • Enhance user experience • Add/update functionality • Revise extraction web service • Review existing extraction output: organization, format, precision (Richard, Mike) • Generate use cases, requirements, design (Eric, Richard, Mike, Ed, Andy, Charles) • Modify web service as necessary (Richard, Mike) • Revise search web service • Apply indexing scheme to increase speed (Richard, Mike) • Make 100% OpenSearch compatible by returning total count (Richard, Mike) • Revise imaging web service • Replace existing WMS with Tiled WMS granule MRF’s (Richard, Mike, Jeff, Charles) • Allow search by data summaries • Identify standard set of data summaries across datasets to add to tile database (Richard, Mike, Ed, Andy, Charles) • Review and revise process for adding unique data summaries per dataset to tile database (Richard, Mike, Ed) • Update GUI controls • Perform usability review • Identify use cases (Eric, Charles, Ed) • Generate requirements and design document (Eric, Richard, Charles, Ed, Sargis) • Generate GUI mockup (Sargis, Richard) • Implement new GUI design (Sargis, Richard) • Integrate with PO.DAAC system • Connect to imaging subsystem • Utilize encapsulated WMS • Update calls via Tiled WMS to standard WMS with optional keywords as necessary, e.g., scaling, colors, projection (Richard, Mike) • Connect to DMAS • Initiate tiling via DMAS-Execute • Design and define input: universal PO.DAAC data format model (Andy, Eric, Charles, Thomas, Michelle, Ed) • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)

  17. FY13: Tiled WMS • Stabilize system • Mitigate tile database connectivity issues • Re-architect existing tile database • Review current schema and tables (Richard, Mike) • Generate requirements and design document (Eric, Richard, Mike) • Identify revised or new schema and tables (Richard, Mike) • Build database (Richard, Mike) • Update all services connecting to database (Richard, Mike) • Improve maintainability • Upgrade installation and deployment procedures • Streamline ingest • Review existing procedures and documentation and update as necessary (Richard, Mike) • Update messaging • Add error, information, and warning notifications • Insert calls to SigEvent service throughout ingestion module (Richard, Mike) • Insert calls to SigEvent service regarding database “health” (Richard, Mike) • Expand functionality • Add/update functionality • Revise extraction web service • Review existing extraction output: organization, format, precision (Richard, Mike) • Generate use cases, requirements, design (Eric, Richard, Mike, Ed, Andy, Charles) • Modify web service as necessary (Richard, Mike) • Revise search web service • Apply indexing scheme to increase speed (Richard, Mike) • Make 100% OpenSearch compatible by returning total count (Richard, Mike) • Revise imaging web service • Replace existing WMS with Tiled WMS granule MRF’s (Richard, Mike, Jeff, Charles) • Allow search by data summaries • Identify standard set of data summaries across datasets to add to tile database (Richard, Mike, Ed, Andy, Charles) • Review and revise process for adding unique data summaries per dataset to tile database (Richard, Mike, Ed) • Update GUI controls • Perform usability review • Identify use cases (Eric, Charles, Ed) • Generate requirements and design document (Eric, Richard, Charles, Ed, Sargis) • Generate GUI mockup (Sargis, Richard) • Implement new GUI design (Sargis, Richard) • Integrate with PO.DAAC system • Connect to imaging subsystem • Utilize encapsulated WMS • Update calls via Tiled WMS to standard WMS with optional keywords as necessary, e.g., scaling, colors, projection (Richard, Mike) • Connect to DMAS • Initiate tiling via DMAS-Execute • Design and define input: universal PO.DAAC data format model (Andy, Eric, Charles, Thomas, Michelle, Ed) • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)

  18. FY13: Event Service • Stabilize system • Mitigate tile database connectivity issues • Re-architect existing tile database • Review current schema and tables (Richard, Mike) • Generate requirements and design document (Eric, Richard, Mike) • Identify revised or new schema and tables (Richard, Mike) • Build database (Richard, Mike) • Update all services connecting to database (Richard, Mike) • Improve maintainability • Upgrade installation and deployment procedures • Streamline ingest • Review existing procedures and documentation and update as necessary (Richard, Mike) • Update messaging • Add error, information, and warning notifications • Insert calls to SigEvent service throughout ingestion module (Richard, Mike) • Insert calls to SigEvent service regarding database “health” (Richard, Mike) • Enhance user experience • Add/update functionality • Revise extraction web service • Review existing extraction output: organization, format, precision (Richard, Mike) • Generate use cases, requirements, design (Eric, Richard, Mike, Ed, Andy, Charles) • Modify web service as necessary (Richard, Mike) • Revise search web service • Apply indexing scheme to increase speed (Richard, Mike) • Make 100% OpenSearch compatible by returning total count (Richard, Mike) • Revise imaging web service • Replace existing WMS with Tiled WMS granule MRF’s (Richard, Mike, Jeff, Charles) • Allow search by data summaries • Identify standard set of data summaries across datasets to add to tile database (Richard, Mike, Ed, Andy, Charles) • Review and revise process for adding unique data summaries per dataset to tile database (Richard, Mike, Ed) • Update GUI controls • Perform usability review • Identify use cases (Eric, Charles, Ed) • Generate requirements and design document (Eric, Richard, Charles, Ed, Sargis) • Generate GUI mockup (Sargis, Richard) • Implement new GUI design (Sargis, Richard) • Integrate with PO.DAAC system • Connect to imaging subsystem • Utilize encapsulated WMS • Update calls via Tiled WMS to standard WMS with optional keywords as necessary, e.g., scaling, colors, projection (Richard, Mike) • Connect to DMAS • Initiate tiling via DMAS-Execute • Design and define input: universal PO.DAAC data format model (Andy, Eric, Charles, Thomas, Michelle, Ed) • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)

  19. FY13: Consolidated Web Services • Stabilize system • Mitigate tile database connectivity issues • Re-architect existing tile database • Review current schema and tables (Richard, Mike) • Generate requirements and design document (Eric, Richard, Mike) • Identify revised or new schema and tables (Richard, Mike) • Build database (Richard, Mike) • Update all services connecting to database (Richard, Mike) • Improve maintainability • Upgrade installation and deployment procedures • Streamline ingest • Review existing procedures and documentation and update as necessary (Richard, Mike) • Update messaging • Add error, information, and warning notifications • Insert calls to SigEvent service throughout ingestion module (Richard, Mike) • Insert calls to SigEvent service regarding database “health” (Richard, Mike) • Enhance user experience • Add/update functionality • Revise extraction web service • Review existing extraction output: organization, format, precision (Richard, Mike) • Generate use cases, requirements, design (Eric, Richard, Mike, Ed, Andy, Charles) • Modify web service as necessary (Richard, Mike) • Revise search web service • Apply indexing scheme to increase speed (Richard, Mike) • Make 100% OpenSearch compatible by returning total count (Richard, Mike) • Revise imaging web service • Replace existing WMS with Tiled WMS granule MRF’s (Richard, Mike, Jeff, Charles) • Allow search by data summaries • Identify standard set of data summaries across datasets to add to tile database (Richard, Mike, Ed, Andy, Charles) • Review and revise process for adding unique data summaries per dataset to tile database (Richard, Mike, Ed) • Update GUI controls • Perform usability review • Identify use cases (Eric, Charles, Ed) • Generate requirements and design document (Eric, Richard, Charles, Ed, Sargis) • Generate GUI mockup (Sargis, Richard) • Implement new GUI design (Sargis, Richard) • Integrate with PO.DAAC system • Connect to imaging subsystem • Utilize encapsulated WMS • Update calls via Tiled WMS to standard WMS with optional keywords as necessary, e.g., scaling, colors, projection (Richard, Mike) • Connect to DMAS • Initiate tiling via DMAS-Execute • Design and define input: universal PO.DAAC data format model (Andy, Eric, Charles, Thomas, Michelle, Ed) • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)

More Related