Unlocking the Mystery of the Data Quality Act: What it Means for NOAA Research

1. Unlocking the Mystery of the Data Quality Act: What it Means for NOAA Research Jamie Krauk, Office of Scientific Support August 6, 2002

2. Overview • Section 515 – Data Quality Act • Directives from OMB • What is “Information”? • The Three Elements of Quality: Utility, Integrity, and Objectivity • NOAA’s Information Standards • Compliance Procedures • Administrative Mechanism • Implications for NOAA Research • Discussion

3. Section 515 – Data Quality Act • Section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554): • Directs the Office of Management and Budget (OMB) to issue government wide guidelines that “provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information disseminated by Federal agencies”

4. Directives from OMB • OMB issued guidelines directing all Federal agencies to: • Create and issue agency-specific standards to ensure Information Quality and develop a pre-dissemination review process • Establish administrative mechanisms allowing affected persons to seek and obtain correction of information • Report periodically to OMB on the number and nature of complaints received

5. What is “Information”? • Any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, cartographic, narrative, or audiovisual forms (includes web dissemination but not hyperlinks to information that others disseminate; does not include clearly stated opinions)

6. The Three Elements of Quality • Utility • The usefulness of the information to its intended users, including the public • Integrity • The protection and security of information from unauthorized access or revision to prevent corruption or falsification • Objectivity • The presentation of accurate, reliable, unbiased information in an accurate, clear, complete and unbiased manner within the proper context

7. NOAA’s Information Standards

8. Ensuring Utility • NOAA strives to continually improve the usefulness of its data and information products • NOAA interacts with broad range of customers through workshops, surveys, product reviews to assess and improve product utility and accessibility • NOAA disseminates products in a manner that allows them to be accessible and understandable to a broad range of users via print, diskettes, CD-ROM, internet, and broadcast media using consistent format and attribution

9. Ensuring Integrity • Information is safeguarded from improper access prior to dissemination to a degree commensurate with the risk associated with the information if it was compromised • All electronic disseminated information complies with “Security of Automated Information Resources”, Computer Security Act, and Government Information Systems Reform Act • Confidentiality of information is protected under the Privacy Act • Additional protections are provided as appropriate

10. Ensuring Objectivity • Original Data • Synthesized Products • Interpreted Products • Hydro-meteorological, Hazardous Chemical Spill, and Space Weather Warnings, Forecasts, and Alerts (Time Sensitive Products) • Experimental Products • Natural Resource Plans • Corporate and General Information

11. Objectivity – Original Data • Data are collected according to documented procedures or in a manner that reflects standard practices accepted by the relevant scientific and technical communities • Original data undergo quality control prior to being used by the agency, including: • Gross error checks, comparisons with other independent sources of measurement, examination of statistical summaries, visual inspection • NOAA strives for transparency regarding data collection procedures, level of quality, and limitations • NOAA’s quality control process is ongoing

12. Objectivity – Synthesized Products • Data and information sources are identified or made available upon request • All data is of known quality or from sources acceptable to the relevant community • Products are created using methods in standard manuals, documented in an accessible format, or generally accepted by the relevant community • All synthesized products, or the procedures used to create them, are reviewed • All methods are included with product or made available upon request

13. Objectivity – Interpreted Products • Data and information sources are properly referenced or made available upon request • Products are created using methods in standard manuals, documented in an accessible format, or generally accepted by the relevant community • All products are put into context by additional information that demonstrates quality and limitations of products • All products are reviewed ranging from internal peer review to external, independent review • All methods are included with product or made available upon request

14. Objectivity – Time Sensitive Products • Data of known quality is used to the extent possible under tight time constraints • Data and information sources are identified or made available upon request • To the extent possible, information is produced using methods and techniques that are documented in accessible formats or are generally accepted • Accuracy and skill score of warnings, forecasts, and advisories is tracked as a mechanism for evaluation

15. Objectivity – Experimental Products • Products are either disseminated for experimental use, evaluation or feedback, or used in urgent, time-critical cases when qualified scientists believe risk will be lessened if the product is used • The best science and supporting studies available are used in accordance with sound and objective practices, evaluated in relevant communities, and peer reviewed if feasible • Provisional documentation of theory and methods are prepared, including assumptions • When products are used, they are accompanied by explicit limitations and used in conjunction with non-experimental methods and tools if possible

16. Objectivity – Natural Resource Plans • Plans are developed according to published standards and are often used as basis for policy • Plans are a composite of several types of scientific information from both internal and external sources and are based on the best information available • All supporting materials and data are properly referenced • Plans are reviewed by technically competent individuals ranging from internal to external independent review commensurate with importance

17. Objectivity – Corporate Information • This is non-scientific, non-financial, non-statistical information • Sources of materials are noted as appropriate and information is reliable and accurate to an acceptable degree of error • Product review is incorporated into the normal product formulation process, including: • Personal or supervisory review, process design and monitoring, use of quality check lists, use of management controls

18. Compliance Procedures • Read and understand all standards carefully and ensure compliance by October 1, 2002 • Identify what research products and information fall into each category, and the corresponding standard for each • Prepare a current and updateable inventory of all scientific procedures and limitations for each type of information and have available upon request, including: • Methods, materials, assumptions, techniques • Process by which information was last reviewed • Date of last review, names of reviewers, actions taken

19. Administrative Mechanism • Any affected person may request timely correction of disseminated information • Requests must be directed to NOAA Section 515 Officer (NOAA Executive Secretariat) for recording and forwarding to Lab/Program • Requests must contain all relevant information or the request is denied or returned to be amended and resubmitted • If the request is complete the Lab/Program director determines whether the request states a claim • Either way, a response is required within 60 days from original receipt to NOAA Section 515 Officer

20. Administrative Mechanism • A claim is stated if the information was based on misapplication or non-application of the appropriate standard • All supporting documents pertaining to the information in question must be gathered and filed as potential future administrative record • Lab/Program decides whether the information complies with the applicable standard • If yes, an explanatory response is drafted within 60 days and requestor has 30 days to file an appeal to DAA • If no, Lab/Program decides whether the information can be corrected, how, and when and then Lab/Program drafts a response within 60 days as above

21. Implications for NOAA Research • Paradigm shift in NOAA’s research enterprise? • Opens door for court cases and limits flexibility and scientific freedom • Required to keep impeccable notes • Web community needs to be aware and well-versed in standards in order to be the eyes and ears for NOAA Research

22. Questions?