1 / 19

LP DAAC OPeNDAP Services

LP.DAAC has implemented a tiered approach to data access using OPeNDAP. This allows users to access and extract specific subsets of data based on spatial, band/layers, and temporal parameters. The OPeNDAP services use HTTP for data delivery, reducing time and data downloaded. The OPeNDAP URLs are complex but can be easily generated using the API middleware.

bcathy
Download Presentation

LP DAAC OPeNDAP Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. LP DAAC OPeNDAP Services Jason Werpy LP DAAC Enterprise Architect SGT, Inc., Contractor to the USGS EROS Center *Work performed under USGS contract G15PD00766

  2. OPeNDAP – The base of LP DAAC Data access • Tiered Concept • LP DAAC created a tiered approach to data access

  3. A Quick Aside – HTTP access to everything • OPeNDAP utiilzes HTTP to access data, so LP DAAC decided to deploy all the components for data access as HTTP services as well. • Why HTTP services? • Simple • Standard based protocol • Securable • Dynamic interaction methods (get/post, etc.) • Stream based data delivery

  4. OPeNDAP – The base of LP DAAC Data access • Data Access methods • OPeNDAP - Open-source Project for a Network Data Access Protocol. Provides a direct way of subsetting LP DAAC data from our Data Pool.

  5. LP DAAC OPeNDAP Deployment • Hyrax was chosen for the LP DAAC deployment because of its capabilities for handling of MODIS tile data. • Due to the uniform tiled nature of the LP DAAC’s MODIS data, we chose to aggregate data based on date. • Due to the configuration of the ECS data pool we chose to programmatically create and update our aggregation files instead of using the scanning based aggregation capabilities. • Aggregation scripts are available at ECC, contact me for further details. • LP DAAC OPeNDAP Services • http://opendap.cr.usgs.gov/opendap/hyrax

  6. LP DAAC OPeNDAP Deployment • To handle large volumes of data access LP DAAC deployed a cluster of servers to provide OPeNDAP services.

  7. OPeNDAP Data Access UI API Middleware Data Services

  8. OPeNDAP Data Access UI API Middleware Data Services

  9. LP DAAC OPeNDAP Data Access UI API Middleware Data Services link

  10. OPeNDAP Data Access • Access and extract discrete subsets • Spatial • Band/Layers • Temporal (from aggregation) • Request data via URL http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml.ascii?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] • Returns data that the user intends to use! • Reduces time spent downloading and processing • Minimizes the amount of data downloaded • Converts the data format if desired UI API Middleware Data Services

  11. OPeNDAP Data Access OPeNDAP request: http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml.json?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] Return: { "name": "h11v04.ncml", "attributes": [ {"name": "HDF_GLOBAL","attributes": [ {"name": "HDFEOSVersion", \ "value": ["HDFEOS_V2.9"]} ] } ], "leaves": [ { "name": "sur_refl_b01", "type": "Float32", "attributes": [ {"name": "coordinates", "value": ["Latitude Longitude"]}, {"name": "long_name", "value": ["Surface_reflectance_for_band_1"]}, {"name": "units", "value": ["reflectance"]}, {"name": "scale_factor_err", "value": [0.0000000000000000]}, {"name": "add_offset_err", "value": [0.0000000000000000]}, {"name": "calibrated_nt", "value": [5]}, {"name": "orig_scale_factor", "value": [0.00010000000000000000]}, {"name": "orig_add_offset", "value": [0.0000000000000000]}, {"name": "_FillValue", "value": [-28672]}, {"name": "valid_min", "value": [-0.009999999776]}, {"name": "valid_max", "value": [1.599999905]}, {"name": "grid_mapping", "value": ["MOD_Grid_250m_Surface_Reflectance_eos_cf_projection"]} ], "shape": [1,11,11], "data": [[[0.5359, 0.5657, 0.5258, 0.4739, 0.4648, 0.5087, 0.5234, 0.5234, 0.6214, 0.5793, 0.5374], [0.5759, 0.5414, 0.5338, 0.4489, 0.432, 0.5036, 0.6051, 0.6214, 0.5793, 0.455, 0.446], [0.4594, 0.5409, 0.6224, 0.6289, 0.6289, 0.6051, 0.5005, 0.5005, 0.3325, 0.3092, 0.3826], [0.5831, 0.5831, 0.6126, 0.6307, 0.6307, 0.5277, 0.612, 0.3325, 0.3092, 0.332, 0.4036], [0.6329, 0.6126, 0.6061, 0.5691, 0.5691, 0.612, 0.5841, 0.5841, 0.4301, 0.4782, 0.6271], [0.6092, 0.6092, 0.6152, 0.5691, 0.6297, 0.6035, 0.5052, 0.5052, 0.4782, 0.6518, 0.6913], [0.5712, 0.6152, 0.6248, 0.5686, 0.5686, 0.3522, 0.5088, 0.5912, 0.6316, 0.4729, 0.5756], [0.5737, 0.5737, 0.524, 0.5368, 0.3522, 0.5691, 0.5944, 0.4085, 0.4729, 0.5375, 0.5375], [0.5606, 0.5211, 0.5368, 0.5874, 0.5874, 0.343, 0.26, 0.26, 0.3825, 0.5794, 0.5863], [0.5883, 0.5591, 0.5263, 0.4325, 0.343, 0.4485, 0.626, 0.626, 0.5462, 0.3682, 0.3463], [0.5435, 0.5263, 0.4325, 0.4887, 0.4887, 0.5175, 0.626, 0.5475, 0.4101, 0.4592, 0.4203]]] } ], "nodes": [] } UI API Middleware Data Services

  12. OPeNDAP Data Access • OPeNDAP URLs are complex • Need to know: • Syntax • Product name • Layer name • Time • Geolocation • Tile locations (H & V) – Data is aggregated over time by tile • Pixel location (Ydim & Xdim) http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml.json?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] UI API Middleware Data Services

  13. With the OPeNDAP in Place what can be done? • Using these building blocks the LP DAAC (or anyone who wants to really) can build new interfaces and systems like….

  14. AppEEARS • Application for Extracting and Exploring Analysis Ready Samples • Web application interface for accessing and extracting LP DAAC’s tiled MODIS and WELD data UI API Middleware Data Services

  15. AppEEARS location https://lpdaacsvc.cr.usgs.gov/appeears/ UI API Middleware Data Services

  16. Questions? Questions? UI API Middleware Data Services

  17. AppEEARS • Built on top of services and tools • Capabilities • Easy access and selection of MODIS and WELD land data • Extracts pixel values for sample point locations • Data exploration and interaction • Dynamic Graphs • Tables • Download Bundle • Analysis-ready data UI API Middleware Data Services

  18. For More Information about our services Subscribe to our listserv: https://lists.nasa.gov/mailman/listinfo/lpdaac

  19. BEDI – LP DAAC Response • Middleware Web Services • Tilemap – Coverts Latitude and Longitude values into tile and pixel values for datasets located on a common grid structure. • Fromjulian – Converts dates from a Julian date format to a mm-dd-yyyy format. • Tojulian – Converts dates from a mm-dd-yyyy format to a Julian date format of the number of days since 01-01-2000. • Product (coming soon!) – Service that provides information about the different products available from the LP DAAC. • Inventory – Service that returns information about the granules in the LP DAAC inventory. Powered by NASA Common Metadata Repository (CMR). UI API Middleware Data Services

More Related