ESDIS Metadata Architecture Study Discussion

1. ESDIS Metadata Architecture Study Discussion Presented by Kevin Murphy and the Metadata Architecture Study Team Members 2013 ESIP Summer Meeting | Chapel Hill, NC http://bit.ly/13D7r5A

2. 2013 ESIP Summer Meeting | Chapel Hill, NC Outline • Background • MAS Phase I • MAS Phase II • Community Feedback • Open Discussion

3. 2013 ESIP Summer Meeting | Chapel Hill, NC Background

4. 2013 ESIP Summer Meeting | Chapel Hill, NC EOSDIS Mission The Earth Science Data and Information System (ESDIS) project provides access to data through the development and operation of the science systems of the Earth Observing System (EOS) Data and Information System (EOSDIS).

5. 2013 ESIP Summer Meeting | Chapel Hill, NC Metadata Architecture Study (MAS) ESDIS initiated a study of EOSDIS metadata capabilities/processes in the spring of 2012 to evaluate opportunities for improved data access and use, reduce efforts by data providers, and improve metadata integrity. Goals: • Unify several disparate metadata formats (DIF, ECHO, EMS). • Provide consistent results, performance, and access to users regardless of the mechanism through which they access EOSDIS metadata. • Improve the consistency and quality of EOSDIS metadata holdings. • Provide a platform capable of expanding EOSDIS metadata systems to incorporate services, tools, documents, etc.

6. 2013 ESIP Summer Meeting | Chapel Hill, NC Ground Rules Metadata Harmony Study • EOSDIS metadata are currently a widely used resource which must remain available with at least the same level of service during system definition, development and implementation. • Enhancements to metadata systems must focus on improving user interactions with the system (End Users, International Users, Data Providers), sustaining engineering of the system, and the integrity of metadata.

7. 2013 ESIP Summer Meeting | Chapel Hill, NC Metadata architecture study (phase I)

8. 2013 ESIP Summer Meeting | Chapel Hill, NC MAS I Research Study • Objective: • Identify opportunities for improved data access and use, reduce efforts by data providers, and improve metadata integrity. Of interest were the overlapping roles and interfaces between the GCMD and ECHO. The five-member study team included representatives from ESDIS, GCMD, ECHO, EMS, and the DAACs. • Results - Eight recommendations in four categories:

9. 2013 ESIP Summer Meeting | Chapel Hill, NC MAS I Recommendations • Move to a single, shared NASA Earth Science Metadata repository for submission of collection level metadata. • ESDIS should define a consistent dataset identifier policy to be used across DAACs and central data systems (ECHO, GCMD, EMS).   • Consider a regular communication forum to address and clarify ESDIS requirements.   • Schedule periodic technical telecons between GCMD and the DAACs to provide insight into new or planned capabilities. • GCMD should implement a mechanism to notify DAACs of DIF and SERF structure modifications. • Both ECHO and GCMD should develop a DAAC-accessible mechanism for tracking implementation plans and status for requested changes. • GCMD should roll out the new Keyword Management System and its RESTful APIs to the DAACs as soon as practical. • ESDIS should continue to monitor emerging metadata needs and related technologies, and work with the community to define conventions for these new metadata.

10. 2013 ESIP Summer Meeting | Chapel Hill, NC Metadata architecture study (Phase II)

11. 2013 ESIP Summer Meeting | Chapel Hill, NC MAS II Architecture Development • Objectives: • Develop a Unified Metadata Model (UMM) that unifies the EOSDIS metadata assets into one cohesive model.   • Good position for future ISO compatibility. • Create a Concept of Operations document that provides an overview of how the Unified Metadata Model (UMM) and Common Metadata Repository (CMR) will be used.

12. 2013 ESIP Summer Meeting | Chapel Hill, NC Unified Metadata Model (UMM)

13. 2013 ESIP Summer Meeting | Chapel Hill, NC Overview • Unification Process • What was in scope, what was not in scope • Vocabulary • UMM Core Vocabularies • Extendable Vocabularies • Knowledge Base System • Documentation

14. 2013 ESIP Summer Meeting | Chapel Hill, NC Unification Process • Initial focus is to unify the following models: • Current conversions are “lossy”. • By unifying these models ESDIS can latter focus on creating one mapping to ISO utilizing the upcoming NASA “flavor”. • The ISO model was mapped where possible, and will be in scope for a future phase. Metadata

15. 2013 ESIP Summer Meeting | Chapel Hill, NC Unifying the Models ECHO DIF • CitationForExternalPublication • ProcessingLevelId • Price • SpatialKeywords • TemporalKeywords • Temporal • Contact • ScienceKeywords • Platform • Instrument • Sensor • Campaigns • TwoDCoordinateSystem • OnlineAccessURL • OnlineResource • AssociatedDIFs • Spatial • ArchiveCenter • AdditionalAttributes • AssociatedBrowseImages • SpatialInfo • CollectionProgress • RestrictionComments • Data Set Citation • Personnel • Instrument • Platform • Temporal Coverage • Paleo-Temporal Coverage • Spatial Coverage • Location • Data Resolution • Project • Quality • Access Constraints • Use Constraints • Distribution • Data Set Language • Data Set Progress • Related URL • DIF Revision History EMS Not Shown

16. 2013 ESIP Summer Meeting | Chapel Hill, NC Unified Fields

17. 2013 ESIP Summer Meeting | Chapel Hill, NC Vocabularies • UMM Core Vocabularies • Extendable Vocabularies

18. 2013 ESIP Summer Meeting | Chapel Hill, NC Benefits of Core Vocabularies • Provides a classification system for metadata. • Prevents zero-hit queries. • Improves metadata accuracy and completeness. • Improves the usability of earth science metadata.

19. 2013 ESIP Summer Meeting | Chapel Hill, NC Core Vocabularies - Unique Keyword Identifiers • Controlled vocabularies are referenced in the UMM using keyword identifiers (UUID’s) to SKOS concepts in the GCMD Keyword Management System Service. • Updates to keywords referenced in metadata happen automatically since keywords are only “referenced” using these identifiers. • Additions and/or changes to the vocabularies can be made through the existing governance process.

20. 2013 ESIP Summer Meeting | Chapel Hill, NC UMM Core Vocabulary Types • Science Keywords • Platforms • Instruments • Sensors • Projects • Organizations • Organization Types • Locations • Temporal Resolution Range Keywords • Horizontal Resolution Range Keywords • Vertical Resolution Range Keywords • Chronostratigraphic Units • Dataset Languages • Related URL Types and Protocols • ISO Topic Categories • DataSet Progress • DataSetLanguage • Temporal Coverage Types • Processing Level Identifiers • Units for Resolutions • Units for Geographic Coordinates

21. 2013 ESIP Summer Meeting | Chapel Hill, NC Extendable Vocabularies AncillaryKeyword Class Provides a way to reference controlled vocabularies external to the CMR. This UMM class has a resource attribute associated with it, allowing the metadata author to reference an external RDF concept with its own namespace identified through a URI. Example: <AncillaryKeyword resource="http://mmisw.org/ont/cf/parameter/wind_speed.rdf">Wind Speed</AncillaryKeyword> The example above links to a CF standard_name keyword provided by the Marine Metadata Interoperability Project (MMI).

22. 2013 ESIP Summer Meeting | Chapel Hill, NC Knowledge Based System (Many Interconnections)

23. 2013 ESIP Summer Meeting | Chapel Hill, NC UMM Documents (Finalized by Early August) • UMM Overview Document • High level UML diagrams of key parts of the model and descriptions of how metadata can be represented in the model. • UMM Crosswalk Study Document • Analysis of each model element, similarities, differences, and provides recommendations. • UMM Crosswalk Spreadsheet • Detailed mapping between DIF, ECHO, EMS. • Maps UMM to ISO (where possible). • UMM Schema created from the model • UMM is not intended to be a format, but can be useful to describe in this way. • Online Javadoc style navigation of the UMM Schema

24. 2013 ESIP Summer Meeting | Chapel Hill, NC Common Metadata Repository (CMR)

25. 2013 ESIP Summer Meeting | Chapel Hill, NC URS GIBS EMS EOSDIS Discovery Tools DAACs Near Real Time (LANCE) ??? CMR

26. 2013 ESIP Summer Meeting | Chapel Hill, NC

27. 2013 ESIP Summer Meeting | Chapel Hill, NC

28. 2013 ESIP Summer Meeting | Chapel Hill, NC Metadata Concept Support

29. 2013 ESIP Summer Meeting | Chapel Hill, NC Extensibility

30. 2013 ESIP Summer Meeting | Chapel Hill, NC 1 2 UMM ISO19115 Ingest Adapter 3

31. 2013 ESIP Summer Meeting | Chapel Hill, NC Improving Metadata Quality with Assessment and Scoring

32. 2013 ESIP Summer Meeting | Chapel Hill, NC

33. 2013 ESIP Summer Meeting | Chapel Hill, NC

34. 2013 ESIP Summer Meeting | Chapel Hill, NC How Will This Affect Me? http://store.metmuseum.org/content/ebiz/themetstore/invt/80010981/80010981_01_l.jpg

35. 2013 ESIP Summer Meeting | Chapel Hill, NC Metadata Providers • You won’t have to reconcile multiple systems • Most systems will continue to work • ECHO FTP ingest going away • Current formats will continue to function • You will need to be engaged in the adaptation process • You will need to support controlled vocabularies • You will have a voice through the Technical Committee Streamlined Processes, More Engagement

36. 2013 ESIP Summer Meeting | Chapel Hill, NC Client Developers More Metadata, Better Quality Richer, higher quality metadata will be available Existing APIs will be supported Standards-compliant APIs will be available Metadata will be available in an ISO19115 compliant format

37. 2013 ESIP Summer Meeting | Chapel Hill, NC End Users More Metadata, Better Quality Richer, higher quality metadata will be available Unified view of EOSDIS holdings Easier access through a variety of clients, data casting, etc Metadata will be available in an ISO19115 compliant format

38. 2013 ESIP Summer Meeting | Chapel Hill, NC Community Feedback

39. 2013 ESIP Summer Meeting | Chapel Hill, NC Review Groups June – July 2013 April 2013 May – June 2013

40. 2013 ESIP Summer Meeting | Chapel Hill, NC Community Feedback Process • Reviewers submitted feedback and received comments using the EOSDIS Issue Tracking Software – JIRA • 334 “Issues” created: 206 Resolved/Closed, 68 for Future Study

41. 2013 ESIP Summer Meeting | Chapel Hill, NC Summary of Feedback Received • Provide consistent product identifiers (i.e. UUID’s) • Address ISO and NASA flavor of ISO • Include more required fields • Provide common values for project, platform, instrument • Offer more controlled vocabulary…such as processing level • Contain more structured fields...such as resolution units • Provide types and descriptions for any URL • Include provenance tracking

42. 2013 ESIP Summer Meeting | Chapel Hill, NC Next Steps

43. 2013 ESIP Summer Meeting | Chapel Hill, NC MAS II Team • Lead: Kevin Murphy (Kevin.J.Murphy@nasa.gov) • ECHO Representatives • Katie Baynes (Kathleen.Baynes@nasa.gov) • John Farley (John.D.Farley@nasa.gov) • Dan Pilone (Daniel.J.Pilone@nasa.gov) • EMS Representative • Lalit Wanchoo (Lalit.Wanchoo-1@nasa.gov) • GCMD Representatives • Thomas Cherry (Thomas.A.Cherry@nasa.gov) • Tom Northcutt (Robert.T.Northcutt@nasa.gov) • Scott Ritz (Scott.A.Ritz@nasa.gov)

44. 2013 ESIP Summer Meeting | Chapel Hill, NC Open Discussion