Development of a NOAA Policy on the use of External Data Sources

1. Data Archive and Access Requirements Working Group (DAARWG) Meeting Development of a NOAA Policy on the use of External Data Sources

2. Overview • Motivation for Data Sharing Partnerships • Proposed Climate Service • Interagency Climate Portal • National Climate Assessment • Data Management Challenges of External Data • Example: CoCoRaHS • Example: EarthNetworks GHG Network • Existing NOAA Policy • Use Case: Argo Ocean Floats

3. Proposed Climate Service Partnership a Key Tenet for Success “The Climate Service Line Office at NOAA … would work with the broader climate service enterprise … to provide businesses, communities, and resource managers with services and information for decision-making. The proposed Climate Service Line Office at NOAA would improve NOAA’s organization, such that the agency can be a more accessible, transparent, and collaborative partner… NOAA’s reorganization would also support economic innovation and entrepreneurship. This includes supporting development of the private sector climate services industry.” — Written Statement to the House Committee on Science, Space and TechnologyUnder Secretary Jane Lubchenco, June 22, 2011

4. Federal Response to the nation’s climate challenges Proposed Climate Service Security Vision and Strategic Framework Forestry Water Health Partnerships & Collaboration NOAA’s Assets Infrastructure Information Delivery and Decision Support NOAA uses its national and regional infrastructure to deliver climate services today Assessments of Climate Change and Impacts NOAA is a leader in national and regional climate impact assessments Over 70% of Federal IPCC AR4 WG1 authors were from NOAA Land Management Climate Change Research and Modeling International award winning models of the global climate Oceans Climate Observations and Monitoring NOAA operates over 90 observation and monitoring systems NOAA is mandated to monitor and provide access to climate data and information Energy

5. Interagency Climate Portal Administrations Call for Action • National Climate Adaptation Summit, called for by John Holdren, Director of the White House Office of Science and Technology Policy, issued “seven priorities for near-term action” in 2010, including “Creating a federal climate information portal. This would provide single-point access to data from all relevant federal agencies and programs and would evolve over time into a more ‘national’ portal with information about relevant non-federal efforts.” • Also, Executive Office of the President, Committee on Environment, Natural Resources, and Sustainability (CENRS) Roundtable on Climate Information Services called for an interagency climate information portal plan, focusing on National Climate Assessment

6. Interagency Climate Portal Where to begin development? • A National Climate Portal as envisioned is an immense task • Use the National Climate Assessment (NCA) to focus the initial scope • The NCA can be… • A starting point, extensible to interagency data • A driver of requirements for Data Management Planning: transparency, access, usability, data quality, and business rules • A test case for connecting robust and federated data structures with appropriate user experiences

7. National Climate Assessment A Change in Direction from Past Reports • A product AND a sustained process • More stakeholder engagement and contributions • More transparency, traceability, accessibility and usability • More data, more authors, more users • A suite of Indicators (discrete and composite, physical and societal) • Next report due in 2013, draft due in 2012, Federal Advisory Committee just formed • Snapshots plus frequent updates • More frequent interim reports, updated indicators and fresh web-content are expected… BUT • An assessment is necessarily a snapshot in time and should be reproducible

8. National Climate Assessment Data Management Planning Consideration • External Data • Clear need to move beyond government data and the peer reviewed scientific literature as information source, but who guarantees access and a continuous archive? • Quality • Information Quality Act requirements (highest level) • Peer review requirements (highest level)

9. National Climate Assessment Data Management Requirements • NCA will likely not ‘host’ basic data, but will provide central access to federated system (critical questions remain for external data) • Highly-processed, derived data used in the report may be centrally hosted • Must accommodate huge data sets • Too big to move, too big to host all data in one location • Implies distributed data architecture, federated storage • Metadata must adhere to common standards • Uncertainty must be documented • Quality and original source of data must be clear and peer reviewed • Data system must be interoperable • Use of data.gov must have clear guidelines • Narrative assessment and graphics must link to source data and information

10. External Data Management Challenges • Data Documentation and Quality • Are the metadata sufficient and standardized? • Does the data come from a trusted source (NASA vs. Joe’s Data Shack) and what is the process to certify data sources as “trusted”? • How robust are the science algorithms and quality control procedures? • Data Access and Usage • Are the data accessible via standardized Web services? • Is the data source operationally reliable? • Are there data usage or cost restrictions? • Data Retention • What are the long-term archive prospects for the data? • Must NOAA ensure the data are archived? • Can NOAA depend on the data whenever necessary?

11. Examples of External Data Sources Community Collaborative Rain, Hail & Snow Network (CoCoRaHS) • Unique, non-profit, community-based network of volunteers working together to measure and map precipitation (rain, hail and snow) • Using low-cost measurement tools, stressing training and education, and utilizing an interactive Web-site, aim is to provide the highest quality data for natural resource, education and research applications • NWS routinely uses CoCoRaHS data and NCDC includes in snow archives • Non-standard procedures along with data quality and continuity issues have affected the climate record • FEMA relies on NOAA-provided snow totals to assess natural disasters • Snow Workshop, May 25-27, 2011organized to address these and other issues

12. Examples of External Data Sources Earth Networks GHG Observing Network • Over the next five years, Earth Networks willinvest $25 million to build and deploy a state-of-the-art greenhouse gas emissions observation and measurement network • By the middle of 2012, Earth Networks will deploy 100 greenhouse gas measuring instruments • 50 in the United States • 25 throughout Europe • 25 elsewhere around globe • Collaborating with Scripps Institution of Oceanography • Potential contributor to NOAAcarbon observing network?

13. Existing NOAA Policies NOAA Information and Quality Act (IQA) Guidelines • Much of data NOAA receives from external sources is in the form of raw observations, or “original data” • IQA requires that original data be managed using documented processes with routine QC/QA • http://www.cio.noaa.gov/Policy_Programs/info_quality.html

14. Existing NOAA Policies Policy on Partnerships in the Provision of Env Info • Section 3.04. …NOAA will take advantage of existing capabilities and services of commercial and academic sectors to support efficient performance of NOAA's mission and avoid duplication and competition in areas not related to the NOAA mission… • …NOAA will give due consideration to these abilities and consider the effects of its decisions on the activities of these entities, in accordance with its responsibilities as an agency of the U.S. Government, to serve the public interest and advance the nation's environmental information enterprise as a whole. • http://www.corporateservices.noaa.gov/~ames/NAOs/Chap_216/216-112.pdf

15. Existing NOAA Policies Web Mapping Applicaton Policy Implementation Guide • Primarily directs NOAA’s provision of data to the public. • Data layers provided by sources other than federal agencies…and displayed on NOAA Web pages must be used judiciously, because these situations present special challenges with respect to maintaining agency Information Quality (IQ) Act standards. Such data layers must meet NOAA IQ requirements for NOAA use of third-party information… In addition, any external data from third-party providers must meet the following criteria: • The data must be must be necessary for, and material to, the presentation of agency information or the delivery of agency services, and the map must credit the contributing source of the data or provide a direct link back to the third-party source data provider. • The data must be relevant and timely, and complete steps must have been taken to ensure that data layers are actively updated to achieve the highest level of quality possible. • http://www.cio.noaa.gov/Policy_Programs/NOAA_web_mapping_applications_policy.html

16. Use Case: Argo Ocean Floats • Provider: International - Multiple Nations • Trusted Source(s): Yes (Global Ocean Observing System) • Purpose: Long-term climate monitoring of sea surface temperature and salinity • Science Quality: Excellent (e.g., Scripps Institution of Oceanography) • Metadata: Excellent • Data Access: Excellent with multiple access points • Restrictions on the use of the data: None • Long-term Archive Dependability: Excellent

17. Summary • NOAA is committed to partnering with other data producers, even more so in dealing with the challenges of the National Climate Assessment • NOAA has been using external data sources for a long time for a number of applications, and this will continue • An overall NOAA policy documenting this usage (taking into account existing policies across the agency) is needed to inform acquisition decisions • NOAA has an opportunity to lead the way in defining data management practices more broadly • The DAARWG is well positioned to advice NOAA on the content of such a policy

18. Conclusion • Task is to develop a NOAA policy that defines what data should be acquired through non-NOAA sources and what standards should be applied to such acquisitions • Agenda • National Weather Service Existing Policy (Ed Johnson) • IOOS and Data Provider Certification (Zdenka Willis) • National Climate Assessment Consideration (Anne Waple)

19. Thank you!