https://public/rehabermann/ToolsAndSamples.zip

1. Workshop Resources https://public.me.com/rehabermann/ToolsAndSamples.zip

2. Transitioning to ISOTed Habermann, NOAA/NESDIS/NGDC i checked my 2002 email archives, and here is what i found out: it appears that the current 3rd generation algorithm was implemented into operations around Oct-Nov 2002 time frame. cannot say more precisely, as all email correspondence i am looking at, talks about this indirectly. (maybe it's what's referred to as the Phase II algorithm.) At the same time, we had implemented quite a few other changes fixing data bugs and formats: view angle problem, increased digitization in all channel's reflectances and AODs, etc. The jump is deemed due to introducing 3rd generation algorithm, which replaced the 2nd generation. The new numbers (~0.08) look more realistic than the previous ones (~0.05 or so). The changes seen in the data is close to the expected effect of this change. The 3rd gen alg takes into account the exact spectral response, whereas the 2nd gen is generic ("one size fits all"). hopefully this settles the issue.. Why? 50% change in global average Documentation: It’s not just discovery...

3. Why Documentation?

4. Discovery Portal User User Metadata Types and Sharing Community Metadata Collections Discovery Use / Mashup Understanding More documentation is required for understanding data than discovering or using it.

5. Designated Communities - Users Data preservation is communicating with the future

6. Extended Content Basic Content FGDC Mandatory Identification Metadata Custom Documentation FGDC Discovery Distributor FGDC to ISO 19115 International Standard Unified Modeling Language eXtensible Markup Language ISO 19115 Code Lists Many Identifiers OnLine Resources Responsible Parties Data Quality Reports Spatial/Temporal Extents Metadata Levels / Scope ISO 19119 Services ISO 19110 Feature Types ISO 19115-2 Quality Coverage Usability Platform Instrument Processing Algorithm Evolution: Documentation Standards FGDC Complete Entity and Attribute Data Quality Lineage Spatial Organization Spatial Reference FGDC RSE Identifier Documentation Mission Platform Instrument Aggregation Algorithm Processing FGDC BIO Taxonomy Geographic Extent

7. FGDC to ISO 19115 International Standard Unified Modeling Language eXtensible Markup Language Evolution: Documentation Standards FGDC Complete Entity and Attribute Data Quality Lineage Spatial Organization Spatial Reference A small step in content but a huge step in tools 1. UML - Unified Modeling Language 2. XML - eXtensible Markup Language 3. Wiki - Community

8. Technology Adoption Technological Disruption Selection The rate of progress increases because community energy is focused. Developments are supportive and cumulative Era of Ferment Tipping Point TIME

9. What Does a Tipping Point Look Like? AMS Metadata Policy Working Group Recommendations: The AMS should adopt ISO 19115-2 and SensorML as the metadata standards for the Nationwide Network of Networks. In coordination with other organizations within the broader atmospheric and oceanic sciences community, the AMS should provide opportunities for its members to become better educated on the nomenclature, structure, and implementation of metadata and the standards recommended below. The AMS should establish a permanent committee to serve as a body of experts … to provide advice to organizations working to implement a Nationwide Network of Networks and to coordinate with the Federal Committee for Integrated Observing Systems (CIOS). Federal Coordinator for Meteorology Metadata Joint Action Committee The Joint Action Group recommends that the Federal departments and agencies adopt ISO 19115-2 and SensorML as weather observation metadata standards and participate in the process of implementing those standards across the national weather enterprise. Chapter VI - Other Post Award Requirements and Considerations 4. Dissemination and Sharing of Research Results b. Investigators are expected to share with other researchers, at no more than incremental cost and within a reasonable time, the primary data, samples, physical collections and other supporting materials created or gathered in the course of work under NSF grants. Grantees are expected to encourage and facilitate such sharing. GOES-R: NOAA and NASA

10. Shared Needs NSF NASA DOE Do they each head off in their own direction… Or come together to communicate and collaborate? The groups that created these documents are leading the transition to the ISO Standards and recognizing the need for guidance and tools. USGS

11. The ISO Standards are broad

12. ISO Standards

13. ISO Metadata Objects MD_SpatialRepresentation MD_MetadataExtensionInformation MD_ReferenceSystem MD_DataQuality MD_ApplicationSchemaInformation MD_ or MI_Metadata MD_Distribution MD_PortrayalCatalogueReference MD_ContentInformation MD_MaintenanceInformation MI_AcquisitionInformation MD_DataIdentification MD_BrowseGraphic MD_Constraints MD_Usage MD_AggregateInformation EX_Extent MD_Format MD_Keywords

14. Addresses many important needs Do you need different documentation for different parts of your data? Do you need different documentation for different temporal and spatial subsets? Do you have datasets with multiple sources? Do you need to reference On-Line Resources? Do you need to describe a series of related granules? Do you need to describe many kinds of aggregations? Does data quality vary within the dataset? Do you need to track processing for multiple data sources? Do you need to track compliance with standards? Do you need to use spatial features to describe quality, like grids of quality flags? Do you need to explain why you did things to the data? Do you have datasets in multiple geographic locations? Do you need to describe instruments used to make observations? Do you need to unambiguously identify things using your own namespace? Do you want to manage metadata using a relational or XML database? Do you want to serve metadata using a REST web service? Do you need to identify people in different roles? Do you need to keep track of user problems? Do you need to explain why you did things to the data? Do you need to track requirements and plans? Do you need to share data with international and national partners? Do you need to describe data formats and structures? Do you need to track data transformations and processing? What might the entity and attributes look like for station level metadata?

15. Multiple Dialects and Connections ISO DIF, FGDC, Data.Gov Documentation Repository ISO 19115, 19115-2, 19119 and extensions netCDF SensorML THREDDS WCS, WMS, WFS, SOS KML

16. The ISO Standards are broad What Goes Where?

17. Spiral Development / Training Metadata Content Independent of standard Spiral 1: Initial Content Check Back With Data Collectors/ Providers Check Back With Users Standard Guidance / Implementation Spiral 2-N: Scientific Questions New Requirements New Use Cases

18. Potential Spirals Discovery Understanding Identification Id Title Abstract Resource Date Topic Category Theme Keyword Metadata Contact Science Contact Extent Geospatial Bounding Box Temporal Start/End Vertical Min/Max Place Keywords Text Searches Purpose Extent Description Lineage Statement Project Keywords Content Information Attribute Type Attribute Names Attribute Definitions Attribute Units Connection OnlineResource: Linkage (URL) Name Description Function Distribution Distributor Contact Online Resource Distribution Format Data Center Keywords Browse Graphic Acquisition Information Instrument Platform Instrument Keywords Platform Keywords Quality/Lineage Sources Process Steps Quality Reports / Coverages

19. Spirals and Rubrics Metadata Evaluation for Dataset X Date: 2010/10/24

20. Initial Rubric for FGDC > ISO Translation

21. Identifying Bounding Extents

22. Connections are Critical

23. Rubric Reflects the Improvements

24. Migrating Standards Upstream Disciplinary Experts # Users R E A L T I M E S T A N D A RDS Science Question Data Collection Processing Distribution Discovery Archive Analysis Non-Experts

25. Multiple Dialects and Connections ISO DIF, FGDC, Data.Gov Documentation Repository ISO 19115, 19115-2, 19119 and extensions netCDF SensorML THREDDS WCS, WMS, WFS, SOS KML

26. Multiple Dialects: THREDDS Data Server OPeNDAP WMS WCS Extract Data THREDDS Catalog THREDDS Catalog THREDDS Catalog file2.nc file3.nc file4.nc file1.nc file2.nc file1.nc file.nc file.nc file.nc file.nc

27. Multiple Dialects: Web Accessible Folders

28. Documentation in Three Dialects Documentation Repository NcML Spiral Tracker

29. Documentation Dialects in THREDDS https://geo-ide.noaa.gov/wiki/index.php?title=NcISO

30. Persistence vs. Transport - OPeNDAP File System File System DAP DAP Client

31. Persistence vs. Transport - Documentation Relational Databases File System Rich Inventory XML XML Client

32. Database Objects and References Citation ID Title Date Friend_ID Location_ID Person ID Name EMail OnlineResource ID Name URL

33. XML Attributes: Objects and References ISO XML consists of tags, elements (with or without content), and attributes. An attribute is a name/value pair that exists within a start-tag or empty-element tag. Attributes provide additional information about an element which is not part of the data. Attribute values must contain either single or double quotes. This example shows a step element with one attribute, number with a value of “3”: <step number="3">Connect A to B.</step> Many of the XML attributes used in the ISO Standards fall into two groups: identifiers and references: Identifiers: id and uuid References: uuidref and xlink:href Objects that start with upper case letters have identifiers (id and uuid) Roles that start with lower case letters have references (uuidref and xlink:href) role: friend xlink:href=#JohnDoe role: friend xlink:href=#JaneDoe object: CI_ResponsibleParty id="JaneDoe" object: CI_ResponsibleParty id="JohnDoe"

34. Database and XML Keys XML <person id=JaneDoe> <friend xlink:href="#JohnDoe"/> </person> Citation ID Title Date Friend_ID Location_ID Person ID Name EMail … … … … <person id=JohnDoe> <friend xlink:href="#JaneDoe"/> </person> OnlineResource ID Name URL

35. Leadership Model: Positive Deviance Positive deviance says that if you want to create change, you must scale it down to the lowest level of granularity and look for people within the social system who are already manifesting the desired future state. Take only the arrows that are already pointing toward the way you want to go, and ignore the others. Identify and differentiate those people who are headed in the right direction. Give them visibility and resources. Bring them together. Aggregate them. Barbara Waugh

36. Links GEO-IDE Wiki: Home page: https://www.nosc.noaa.gov/dmc/swg/wiki/index.php ISO Pages: https://nosc.ngdc.noaa.gov/dmc/swg/wiki/index.php?title=Category:ISO_19115 ISO WAF with Rubric Links: http://www.ngdc.noaa.gov/metadata/published/19115/isoMetadataHome.html Spirals: https://www.nosc.noaa.gov/dmc/swg/wiki/index.php?title=Creating_Good_Documentation https://nosc.ngdc.noaa.gov/dmc/swg/wiki/index.php?title=Documentation_Spirals Workbooks: ftp://ftp.ncddc.noaa.gov/pub/Metadata/wkbk ISO Standards: http://www.isotc211.org/Outreach/ISO_TC_211_Standards_Guide.pdf

37. Questions? Questions / Comments / Suggestions: ted.habermann@noaa.gov, jacqueline.mize@noaa.gov, anna.milan@noaa.gov