1 / 57

LP DAAC Big Earth Data Initiative (BEDI) Developed Web Services

LP DAAC Big Earth Data Initiative (BEDI) Developed Web Services. Jason Werpy LP DAAC Enterprise Architect SGT, Inc., Contractor to the USGS EROS Center *Work performed under USGS contract G15PD00766. NASA ’ s LP DAAC. Land Processes (LP) Distributed Active Archive Center (DAAC)

montej
Download Presentation

LP DAAC Big Earth Data Initiative (BEDI) Developed Web Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. LP DAAC Big Earth Data Initiative (BEDI) Developed Web Services Jason Werpy LP DAACEnterprise Architect SGT, Inc., Contractor to the USGS EROS Center *Work performed under USGS contract G15PD00766

  2. NASA’s LP DAAC • Land Processes (LP) Distributed Active Archive Center (DAAC) • https://lpdaac.usgs.gov • Located in Sioux Falls, SD at the USGS Earth Resources Observation Science (EROS) Center

  3. LP DAAC Surface Reflectance, Land Cover, Vegetation Indices

  4. Big Earth Data Initiative • NASA’s BEDI efforts • Increase accessibility of Earth Science Data • Increase availability of Earth Science Data • Increase usability of Earth Science Data

  5. BEDI – LP DAAC Response • The LP DAAC decided that to address the BEDI goals, we would work on services designed and architected to give users improved data visibility, access, and tools.

  6. BEDI – LP DAAC Response • Tiered Concept • LP DAAC created a tiered approach to data access UI API Middleware Data Services

  7. A Quick Aside – Getting data • Why HTTP services? • Simple • Standard bases protocol • Securable • Dynamic interaction methods (get/post, etc.) • Stream based data delivery • It’s even…

  8. A Quick Aside – Getting data • Shown as a good way to get data in the ‘future’…

  9. BEDI – LP DAAC Response • Data Access methods • OPeNDAP - Open-source Project for a Network Data Access Protocol. Provides a direct way of subsetting LP DAAC data from our Data Pool. UI API Middleware Data Services

  10. BEDI – LP DAAC Response • Middleware Web Services • Tilemap – Coverts Latitude and Longitude values into tile and pixel values for datasets located on a common grid structure. • Fromjulian – Converts dates from a Julian date format to a mm-dd-yyyy format. • Tojulian – Converts dates from a mm-dd-yyyy format to a Julian date format of the number of days since 01-01-2000. • Product (coming soon!) – Service that provides information about the different products available from the LP DAAC. • Inventory – Service that returns information about the granules in the LP DAAC inventory. Powered by NASA Common Metadata Repository (CMR). UI API Middleware Data Services

  11. BEDI – LP DAAC Response • Interface • AppEEARS – Interface for selecting point based data from the LP DAAC archive, analyzing that data, and downloading the specific values desired. UI API Middleware Data Services

  12. OPeNDAP Data Access • OPeNDAP was a key component of NASA’s BEDI efforts • OPeNDAP access to the LP DAAC’s Data Pool provides information about singular data granules. • Because MODIS tiles (and WELD tiles) are stored in a uniform grid of tiled data, the LP DAAC was able to create Aggregations by date. This allows an extra parameter that selects data from the date desired. • LP DAAC OPeNDAP Services • http://opendap.cr.usgs.gov/opendap/hyrax UI API Middleware Data Services

  13. OPeNDAP Data Access UI API Middleware Data Services

  14. OPeNDAP Data Access UI API Middleware Data Services

  15. OPeNDAP Data Access UI API Middleware Data Services

  16. OPeNDAP Data Access • Access and extract discrete subsets • Spatial • Band/Layers • Temporal (from aggregation) • Request data via URL http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml.ascii?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] • Returns data that the user intends to use! • Reduces time spent downloading and processing • Minimizes the amount of data downloaded UI API Middleware Data Services

  17. OPeNDAP Data Access OPeNDAP request: http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml.ascii?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] Return: Dataset: h11v04.ncml sur_refl_b01[0][0], 0.5359, 0.5657, 0.5258, 0.4739, 0.4648, 0.5087, 0.5234, 0.5234, 0.6214, 0.5793, 0.5374 sur_refl_b01[0][1], 0.5759, 0.5414, 0.5338, 0.4489, 0.432, 0.5036, 0.6051, 0.6214, 0.5793, 0.455, 0.446 sur_refl_b01[0][2], 0.4594, 0.5409, 0.6224, 0.6289, 0.6289, 0.6051, 0.5005, 0.5005, 0.3325, 0.3092, 0.3826 sur_refl_b01[0][3], 0.5831, 0.5831, 0.6126, 0.6307, 0.6307, 0.5277, 0.612, 0.3325, 0.3092, 0.332, 0.4036 sur_refl_b01[0][4], 0.6329, 0.6126, 0.6061, 0.5691, 0.5691, 0.612, 0.5841, 0.5841, 0.4301, 0.4782, 0.6271 sur_refl_b01[0][5], 0.6092, 0.6092, 0.6152, 0.5691, 0.6297, 0.6035, 0.5052, 0.5052, 0.4782, 0.6518, 0.6913 sur_refl_b01[0][6], 0.5712, 0.6152, 0.6248, 0.5686, 0.5686, 0.3522, 0.5088, 0.5912, 0.6316, 0.4729, 0.5756 sur_refl_b01[0][7], 0.5737, 0.5737, 0.524, 0.5368, 0.3522, 0.5691, 0.5944, 0.4085, 0.4729, 0.5375, 0.5375 sur_refl_b01[0][8], 0.5606, 0.5211, 0.5368, 0.5874, 0.5874, 0.343, 0.26, 0.26, 0.3825, 0.5794, 0.5863 sur_refl_b01[0][9], 0.5883, 0.5591, 0.5263, 0.4325, 0.343, 0.4485, 0.626, 0.626, 0.5462, 0.3682, 0.3463 sur_refl_b01[0][10], 0.5435, 0.5263, 0.4325, 0.4887, 0.4887, 0.5175, 0.626, 0.5475, 0.4101, 0.4592, 0.4203 UI API Middleware Data Services

  18. OPeNDAP Data Access • OPeNDAP URLs are complex • Need to know: • Syntax • Product name • Layer name • Time • Geolocation • Tile locations (H & V) – Data is aggregated over time by tile • Pixel location (Ydim & Xdim) http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml.ascii?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] UI API Middleware Data Services

  19. Utility / Metadata Services • These Services can help fill in the “complicated” parts of the OPeNDAP URLs • Geolocation • The Tilemap service was created to convert Latitude and Longitude information into tile H and V coordinates and x and y pixel locations within the tile. • Time Information • Time data in OPeNDAP is returned as “number of days since 01-01-2000 which is hard to convert “in your head” UI API Middleware Data Services

  20. Utility / Metadata Services • Product Service • The LP DAAC’s (coming soon!) Product Service can be used to get information about LP DAAC products and the bands contained within those products. • Inventory Service • The inventory service provides a view of data granules in the LP DAAC archive, it presents an alternate way to OPeNDAP for data access if a user wants to find entire granules of data. UI API Middleware Data Services

  21. Utility / Metadata Services and Data Services • Demo • Tilemap (also - how do I use this; or – Help!) • An OPeNDAP request (through the LP DAAC aggregated view) • Fromjulian • Tojulian • Inventory Service UI API Middleware Data Services

  22. Demo Service Call Wait…. we don’t know how this service works! But if you don’t know how to call it the service will tell you…. https://lpdaacsvc.cr.usgs.gov/services/tilemap UI API Middleware Data Services

  23. Demo Service Help Response UI API Middleware Data Services

  24. Demo Service Call https://lpdaacsvc.cr.usgs.gov/services/tilemap?product=MOD09A1&latitude=44.306&longitude=-96.7881 Service Response <tilemap> <horizontal>11</horizontal> <vertical>04</vertical> <line_YDim>1366.06</line_YDim> <sample_XDim>176.27</sample_XDim> </tilemap> UI API Middleware Data Services

  25. Demo Located! UI API Middleware Data Services

  26. Demo • Location is now known • Create OPeNDAP call for that location http://opendap.cr.usgs.gov/opendap/hyrax//MOD09A1.005/h11v04.ncml.ascii?Latitude[1366:1:1366][176:1:176],Longitude[1366:1:1366][176:1:176],sur_refl_b01[730:1:733][1366:1:1366][176:1:176],time[730:1:733] • Output Dataset: h11v04.ncml Latitude[0], 44.3062499960216 Longitude[0], -96.7900810678634 sur_refl_b01[0][0], 0.5054 sur_refl_b01[1][0], 0.3101 sur_refl_b01[2][0], 0.3967 sur_refl_b01[3][0], 0.4866 time, 5860, 5868, 5876, 5884 UI API Middleware Data Services

  27. Demo • Output Dataset: h11v04.ncml Latitude[0], 44.3062499960216 Longitude[0], -96.7900810678634 sur_refl_b01[0][0], 0.5054 sur_refl_b01[1][0], 0.3101 sur_refl_b01[2][0], 0.3967 sur_refl_b01[3][0], 0.4866 time, 5860, 5868, 5876, 5884 • The [730:1:733] isgetting the last four acquisition dates, but what are they? Fromjuliancanconvert the values to dates. https://lpdaacsvc.cr.usgs.gov/services/fromjulian?day=5860, 5868, 5876, 5884 Jan 17, 2016;Jan 25, 2016;Feb 2, 2016;Feb 10, 2016 • Tojulian can convert calendar dates to the from 01-01-2000 dates to calculate these values in reverse. UI API Middleware Data Services

  28. Demo • Other LP DAAC Middleware services provide higher level metadata interactions • Inventory Service – Lists LP DAAC granules and select granule information (powered by NASA CMR) • Lets take a look at this service…. UI API Middleware Data Services

  29. Demo • Inventory Service • The LP DAAC Inventory service returns select information about granules in the LP DAAC inventory. It utilizes the information in the NASA Common Metadata Repository (CMR) • URL for the Service • https://lpdaacsvc.cr.usgs.gov/servcies/inventory UI API Middleware Data Services

  30. Demo Service Help Response UI API Middleware Data Services

  31. Demo Service Form Response UI API Middleware Data Services

  32. Demo Service Form Response UI API Middleware Data Services

  33. Demo Service REST GET Call https://lpdaacsvc.cr.usgs.gov/services/inventory? product=MOD11A1.006& latitude=43.6& longitude=-96.7& date=2010-01-01,2016-03-01 UI API Middleware Data Services

  34. Demo Service Data Response <inventory> <url> http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2010.01.01/MOD11A1.A2010001.h10v04.006.2016025150337.hdf </url> <url> http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2010.01.02/MOD11A1.A2010002.h10v04.006.2016025151521.hdf </url> <url> http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2010.01.03/MOD11A1.A2010003.h10v04.006.2016025163542.hdf </url> <url> http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2010.01.04/MOD11A1.A2010004.h10v04.006.2016025165101.hdf </url> . . . <url> http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.06.13/MOD11A1.A2011164.h10v04.006.2016057141329.hdf </url> <url> http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.06.14/MOD11A1.A2011165.h10v04.006.2016057152744.hdf </url> </inventory> UI API Middleware Data Services

  35. Demo Service REST GET Call https://lpdaacsvc.cr.usgs.gov/services/inventory? product=MOD11A1.006& bbox=-98.7,42.8,-96.7,43.6& julianrange=90-132& years=2006-2011& response=metadataurl&output=html UI API Middleware Data Services

  36. Demo Service Data Response http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.01/MOD11A1.A2006091.h10v04.006.2015283114357.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.02/MOD11A1.A2006092.h10v04.006.2015283133023.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.03/MOD11A1.A2006093.h10v04.006.2015283140035.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.04/MOD11A1.A2006094.h10v04.006.2015283153952.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.05/MOD11A1.A2006095.h10v04.006.2015283162912.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.06/MOD11A1.A2006096.h10v04.006.2015283174328.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.07/MOD11A1.A2006097.h10v04.006.2015283184609.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.08/MOD11A1.A2006098.h10v04.006.2015283200712.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.09/MOD11A1.A2006099.h10v04.006.2015283212242.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.10/MOD11A1.A2006100.h10v04.006.2015287200907.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.11/MOD11A1.A2006101.h10v04.006.2015287213340.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2006.04.12/MOD11A1.A2006102.h10v04.006.2015287233224.hdf.xml . . . http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.05.10/MOD11A1.A2011130.h10v04.006.2016055053025.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.05.11/MOD11A1.A2011131.h10v04.006.2016055055057.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.05.12/MOD11A1.A2011132.h10v04.006.2016055074200.hdf.xml http://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2011.05.13/MOD11A1.A2011133.h10v04.006.2016055080155.hdf.xml UI API Middleware Data Services

  37. Demo Click on the URL, get the metadata file <GranuleMetaDataFile> <DTDVersion>1.0</DTDVersion> <DataCenterId>EDC</DataCenterId> <GranuleURMetaData> <GranuleUR>SC:MOD11A1.006:2195949529</GranuleUR> <DbID>2195949529</DbID> <InsertTime>2015-10-10 07:03:24.858</InsertTime> <LastUpdate>2015-10-10 07:03:24.858</LastUpdate> <CollectionMetaData> <ShortName>MOD11A1</ShortName> <VersionID>6</VersionID> </CollectionMetaData> <DataFiles> <DataFileContainer> <DistributedFileName>MOD11A1.A2006091.h10v04.006.2015283114357.hdf</DistributedFileName> <FileSize>4309570</FileSize> <ChecksumType>CKSUM</ChecksumType> <Checksum>4191800068</Checksum> <ChecksumOrigin>DPLIngst</ChecksumOrigin> </DataFileContainer> </DataFiles> <ECSDataGranule> . . . . UI API Middleware Data Services Hint: This would work the same for the browse URL or the data granule URL!

  38. With the Building Blocks in Place What Next? • Using these building blocks the LP DAAC (or anyone who wants to really) can build new interfaces and systems like….

  39. AppEEARS • Application for Extracting and Exploring Analysis Ready Samples • Web application interface for accessing and extracting LP DAAC’s tiled MODIS and WELD data UI API Middleware Data Services

  40. AppEEARS • Built on top of services and tools • Capabilities • Easy access and selection of MODIS and WELD land data • Extracts pixel values for sample point locations • Data exploration and interaction • Dynamic Graphs • Tables • Download Bundle • Analysis-ready data UI API Middleware Data Services

  41. DEMO https://lpdaacsvc.cr.usgs.gov/appeears/ UI API Middleware Data Services

  42. Future Activities • The LP DAAC is planning on expanding its service offerings in the future • AppEEARS Area Services API and UI • Quality Values Services • Data Analysis Services • Addition of more tiled datasets into the API and UI UI API Middleware Data Services

  43. Future Activities • Area Services will allow for the selection of data over more than just the pixel space. • Data harmonization • Align different datasets • Sample datasets into comparable samples. UI API Middleware Data Services

  44. For More Information about our services Subscribe to our listserv: https://lists.nasa.gov/mailman/listinfo/lpdaac

  45. AppEEARS has an API too! • Even AppEEARS is built on top of public services, the AppEEARS API is directly usable, too (its not just an interface!) • AppEEARS API • Product Service – list of products available through AppEEARS (note the same help context) • Quality Service – list the quality layers for products • Sample Extraction Service – extracts a sample of data from OPeNDAP • DEMO UI API Middleware Data Services

  46. Future Activities UI API Middleware Data Services

  47. Demo Service Help Response - json UI API Middleware Data Services

  48. Demo Service Help Response - xml UI API Middleware Data Services

  49. Demo • If we know the dates we want are 10/08/2015, 10/16/2015, and 10/24/2015, we can find the number of days since 01-01-2000 for those dates https://lpdaacsvc.cr.usgs.gov/services/tojulian?day=10-08-2015,10-16-2015,10-24,2015 &output=f2k Output: 5759,5767,5775 Match those dates with the output form the time parameter to get that data. http://opendap.cr.usgs.gov/opendap/hyrax//MOD09A1.005/h11v04.ncml.ascii?time[717:1:719] Dataset: h11v04.ncml time, 5759, 5767, 5775 UI API Middleware Data Services

  50. Demo • Now we can get that data http://opendap.cr.usgs.gov/opendap/hyrax/GroupedAggregates/MOLT/MOD09A1.005/h11v04.ncml.ascii?Latitude[1365:1:1365][176:1:176],Longitude[1365:1:1365][176:1:176],sur_refl_b01[717:1:719][1365:1:1365][176:1:176],time[717:1:719] • Output Dataset: h11v04.ncml Latitude[0], 44.3062499960216 Longitude[0], -96.7900810678634 sur_refl_b01[0][0], 0.0656 sur_refl_b01[1][0], 0.0772 sur_refl_b01[2][0], 0.0654 time, 5759, 5767, 5775 UI API Middleware Data Services

More Related