1 / 21

Improving Data Catalogs with Free and Open Source Software

Improving Data Catalogs with Free and Open Source Software. Kevin O’Brien University of Washington Joint Institute for the Study of the Atmosphere and Ocean. Steven C Hankin – NOAA/PMEL Roland Schweitzer – Weathertop Consulting. AGU Fall Meeting 2013. The Unified Access Framework (UAF).

naif
Download Presentation

Improving Data Catalogs with Free and Open Source Software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Improving Data Catalogs with Free and Open Source Software Kevin O’Brien University of Washington Joint Institute for the Study of the Atmosphere and Ocean Steven C Hankin – NOAA/PMEL Roland Schweitzer – Weathertop Consulting AGU Fall Meeting 2013

  2. The Unified Access Framework (UAF) • A Global Earth Observation Integrated Data Environment (GEO-IDE) project • An attempt to improve scientific data management and access • Focus on successes

  3. Lots of data already available

  4. Projects: (too many to name) Dataformats: netCDF GRIB HDF … Servicestack: What “success” did UAF chose to copy? Year 1 focused on gridded datasets. netCDF-CF-DAP-THREDDS-WMS Applications: Matlab ArcGIS Ferret GrADS IDV Google Earth LAS … ERDDAP Users: (too many to name)

  5. Developing the UAF Catalog Cleaner(a ‘web crawler’) ‘RAW’ UAF ‘RAW’ catalog UAF ‘CLEAN’ catalog NOAA NOAA NOAA Affiliated NOAA Affiliated IOOS Regional Partners IOOS Regional Partners OAR OAR NMFS NMFS NWS NWS NESDIS NESDIS IOOS National Partners IOOS National Partners ESRL ESRL OCO OCO PFEG PFEG GFDL GFDL PMEL PMEL NDBC NDBC AOML AOML NGDC NGDC NODC NODC NAVO NAVO AOOS AOOS NOMADS NOMADS ‘CLEAN’ GCOOS GCOOS SCCOOS SCCOOS Coastwatch Coastwatch PACIOOS PACIOOS SECOORA SECOORA NERACOOS NERACOOS GLOS GLOS NANOOS NANOOS CENCOOS CENCOOS CARICOOS CARICOOS MACOORA MACOORA

  6. Tree Crawl Dataset Crawl Cleaner CatalogRef and Dataset URL’s Raw catalog XML

  7. Tree Crawl Dataset Crawl Cleaner url="http://cwcgom.aoml.noaa.gov/thredds/dodsC/OCEAN_GEOSTROPHIC_CURRENTS/CURRENTS.nc" url="http://cwcgom.aoml.noaa.gov/thredds/dodsC/GLOBAL_MONTHLY_CARBON_FLUXES/FLUXES.nc" url="http://cwcgom.aoml.noaa.gov/thredds/dodsC/GLOBAL_SEASON_CARBON_FLUXES/FLUXES.nc" url="http://cwcgom.aoml.noaa.gov/thredds/dodsC/ROMSMETEO/kk1.nc" url="http://cwcgom.aoml.noaa.gov/thredds/dodsC/MCI_GULF/kk1.nc" url="http://cwcgom.aoml.noaa.gov/thredds/dodsC/MSGSST/SST.nc" url="http://cwcgom.aoml.noaa.gov/thredds/dodsC/TERRA_K490_GULF/terrak490.nc" url="http://cwcgom.aoml.noaa.gov/thredds/dodsC/TERRA_K490_GULF_3D/terrak490.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.199910.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.199911.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.199912.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.200001.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.200002.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.200003.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.200004.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.200005.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.200006.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.200007.nc" url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/NARR.dailyavgs/subsurface/soill.200008.nc" . CatalogRef and Dataset URL’s

  8. Tree Crawl Dataset Crawl Cleaner UAF Clean Catalog

  9. UAF Clean Catalog

  10. How to provide feedback to data providers? • Remember the “Building on Success” theme • ncISO metadata assessment tool is very successful

  11. How to provide feedback to data providers? • Remember the “Building on Success” theme • ncISO metadata assessment tool is very successful How about a catalog quality assessment tool?

  12. Statistics for current catalog and all it’s children Links to rubric reports for child catalogs

  13. Missing services Data issues

  14. url url url url url url url url

  15. Original Catalog Data issues

  16. Moving Forward…. • Welcome feedback on rubric and Catalog Cleaner tool • Change wording in rubric • UAF master catalog to go beyond gridded files • Use ERDDAP to including In Situ featureTypes • Continue community outreach to improve catalogs

  17. Thank you! UAF: geo-ide.noaa.gov Catalog Cleaner code and documentation: http://ferret.pmel.noaa.gov/LAS/documentation/the-uaf-catalog-cleaner/ THREDDS: www.unidata.ucar.edu/projects/THREDDS netCDF: www.unidata.ucar.edu/netcdf OPeNDAP: www.opendap.org CF: cf-pcmdi.llnl.gov Kevin.M.O’Brien@noaa.gov AGU Fall Meeting 2013

More Related