Barry Bozeman, Arizona State University Jan Youtie, Georgia Institute of Technology USA

1. Credibility and Use of Scientific and Technical Information in Policy Making: An Analysis ofthe Information Bases of the National Research Council’s Committee Reports Barry Bozeman, Arizona State University Jan Youtie, Georgia Institute of Technology USA

2. Motivation • Contribution: little empirical understanding of use of STI • Much literature on use of formal information in decision-making • No literature on use of STI in science, technology and innovation (S&T) policy • Research questions for study • Does the perception of the limited use of formal scientific and technical information (STI) accord with empirical reality? • What types of information “compete” with STI for inclusion in science policy-making (where one might expect greatest receptivity)? • How does the choice of various types of information relate to the use and impacts of science policy reports and recommendations?

3. Definition of STI • Open scientific and technical literature appearing in peer-reviewed academic journals or proceedings. • STI used in a narrow sense v. typical in the literature (McClure, 1988; Walker and Hurt, 1990)

4. National Research Council (NRC) • Performs research work for the production of reports on science and technology issues within the National Academies • National Academy of Science (NAS) • National Academy of Engineering (NAE) • Institute of Medicine • National Academies serves as advisor about science and technology intensive policy issues to Congress • Little research on the NRC • Ellefson (2000): single case on non-forest federal land management • Policansky (1999): an ecologist and staff member at the NRC, argues that well-constructed committees with high level of trust given a precisely constructed policy question are most successful • Shapiro and Guston (2006): bureaucracies will shirk their duties, relying on the peer review process to for correction • Fein (2011): NRC plays an increasingly important role in regulatory peer review • Parascandola (2007): conflict of interest policy history in the NAS and NRC • Martin and Irvine (1989): lack of priority setting in NRC reports

5. National Research Council Process National Academies (2006) Our Study Process: Ensuring Independent, Objective Advice.

6. Group-based Credibility Model

7. Sample • All National Academies reports published 2005-2012 • Exclusion of workshop, narrow/very particular studies: Transportation Bureau (NAE), Health and Safety (IOM), repeat Congressionally authorized standing studies • Focus on board appointed/empaneled single shot studies (mostly NAS) • Results=589 reports

8. Method • Report variables: publication year, policy area, National Research Council department, Congressional authorization, # pages, # committee members, # reviewers by sector (business, government, academia, etc.) and geographic location (domestic, foreign), # cited references, # congressional briefings • Committee variables: chair versus member, terminal degree and concentration, industry affiliation by three-digit NAICS, sectoral affiliation (business, government, academia, etc.), title (e.g., president, secretary of agency, gov’t laboratory director) • Committee publications: # publications in the Web of Science and Scopus, # citations, primary Web of Science category, self-citation in reference list • Report outcomes: # Web of Science and Scopus papers, newspapers, industry trade press, newsletters, magazines, news transcripts, hearing transcripts, Congressional record daily issues, bills, House and Senate reports, Congressional Research Service reports. • Reference variables: STI (journal article, published proceeding) v. government report/document, book or book chapter, working paper, industry and nonprofit organization document, interview/testimony, website, expert opinion

9. Database Linkages • NRC annual reports • Web of Science • Scopus • LexisNexis (news, web) • Proquest, UNT Digital Library (hearings, legislation, bills, Congressional reports)

10. Preliminary Findings: Reports • 14% Congressionally authorized • Natural Resource and Defense areas most common and most likely Congressionally authorized (~ 30% of studies in these areas)

11. Average Number of Pages Declining • Average number of pages=188

12. Policy Areas Vary in Use of References in NRC reports (2005,2006)

13. Preliminary Findings: Outcomes • NRC reports have greater impact than expected • 37% self-reported Congressional communications • Informal briefings (30%) more common than formal testimony (15%) • 20% reports have independent presence in Congressional documents • Not necessarily overlapping with self-reports (only 28% of reports with any Congressional impact are independent + self reports) • Taken together 45% of NRC reports have either (or both) of the above impacts

14. Preliminary Findings: Reports and Outcomes • ~ over time (except for independent outcomes being much higher, and self-reported outcomes lower, in 2005) • Congressionally authorized studies are associated with self-reported briefings (29% versus 6% for nonauthorized) and independent outcomes (26% versus 12% for nonauthorized) • Longer NRC studies tend to have self-reported briefings and independent outcomes (briefings=216 mean # pages versus 171 for no briefings; outcomes=213 mean versus 181 for no outcomes)

15. Preliminary Findings: References and Outcomes • Inclusion of references makes little difference to outcomes

16. Plans and some issues • Planned analysis Impact f(STI, active researchers on committee, other committee information, report characteristics) • Limitations • Not every NRC report has references; reference formats differ • We are not analyzing report content • We would like to do some interviews

17. Acknowledgements • This work was supported by the US National Science Foundation, Science of Science and Innovation Policy, Award #1262251. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors. • For more information http://stip.gatech.edu/credibility-and-use-of-scientific-and-technical-information-in-science-policy-making/