Where are We: “The Science of Science Policy”

1. Where are We:“The Science of Science Policy” Bhavya Lal, Core Staff Member Metrics, Performance Measurement, and Evaluation Science and Technology Policy Institute (STPI) 1899 Pennsylvania Avenue NW Washington DC 20006 blal@ida.org American Evaluation Association Portland Oregon November 4, 2006 This presentation reflects the author’s views and not those of the Science and Technology Policy Institute nor any other group or organization with whom the author works

2. Overview • What are we trying to do? • Why is it important? • Why is it hard? • What do we know already? • What needs to be done next?

3. What are we trying to do?

4. Background • Science policy discussions are often dominated by advocates for particular scientific fields or missions • Policy decisions are frequently based upon past practice, beliefs or data trends that may be out of date or have limited relevance to the current situation • We do not have the capacity to predict how best to make and manage future investments so as to exploit the most promising and important opportunities

5. An Intellectual Call to Arms “Are we funding all the R&D we need to defend ourselves, improve and sustain our quality of life, and compete with other nations in a globalized high-technology economy?... How much should a nation spend on science? What kind of science? How much from private versus public sectors? Does demand for funding by potential science performers imply a shortage of funding or a surfeit of performers?... …We need econometric models that encompass enough variables in a sufficient number of countries to produce reasonable simulations of the effect of specific policy choices.” John Marburger, Director Office of Science and Technology Policy Executive Office of the PresidentApril – May 2005

6. Why is it important?

7. Not Just Rhetorical Questions • Justify public R&D investments – not taken for granted • S&T investments are proposed as the solution to all national problems – from Sputnik to Chinindia • Help create frameworks for discussion and prioritization for future R&D investments in times of scarce resources and increasing global competition • Fear that other countries are doing betterthan we are • Overall, help understand linkages between inputs, outputs and enabling conditions • Identify levers other than more funding for interventions • But what, apart from the roads, the sewers, the medicine, the Forum, the theater, education, public order, irrigation, the fresh-water system and public baths... what have the Romans ever done for us?(and the wine, don’t forget the wine…) • Monty Python film “Life of Brian”

8. Why is it hard?

9. Lack of Theory and Frameworks • Modeling the innovation system is a challenge • Federal investment in research only partial component of and not always the first step in a non-sequential, complex, tightly interconnected innovation system • few metrics to characterize the innovation system • it might underestimate the impact of publicly funded research • Unless other conditions in place and measured appropriately, cannot attribute outcomes to R&D funding alone • role of regulatory and legal systems (e.g. R&D tax credits, Bayh-Dole) needs to be better understood • International dimension – especially for modeling workforce - needs to be much more explicitly considered • Insights from evaluation, but piecemeal

10. Data Limitations • No reliable way to define and incorporate R&D that contributes to growth • Studies tend to use only one source of R&D, but likely some fraction of other R&D (e.g. industry, national laboratories, venture capital) may also contribute • No reliable mechanism for disaggregating funding of research by field • Current Federal databases are inadequate • Agencies’ own funding databases are not complete, or cross-comparable • No reliable mechanism for attributing ideas (whether publications or papers) to funding source or agency • Limitations of attribution have been identified

11. Loss of Voice of the Science Policy Analyst • Tenuous link between policy makers and policy analysts • Insights, even when they exist, may not find their way into the decision-making process

12. What do we know (or have we done) already?

13. Summary of Efforts To-Date • Various versions and pieces of the questions have been addressed since the 1950s primarily through • (not discussed – retrospective evaluations) • Econometric/large scale studies • Case studies

14. Efforts To-Date: Econometric Studies • Both cross-sectional and time-series and at levels of aggregation ranging from the firm to the nation • Focus on calculating returns to investment in publicly funded R&D • Methodological and data constraints limit the applicability of current econometric techniques • times-series models don’t mesh with cross-sectional models • spillover benefits typically not captured • most measure the contribution of private sector R&D and especially in the manufacturing rather than the service sector • all assume a sequential, primarily uni-directional flows from research to commercialization • more recent studies do not give returns but rather marginal returns, or stop short of a numerical value

15. Example: Generalized Econometric Model Credit: Brian Zuckerman, STPI

16. Efforts To-Date: Case Studies • Studies range from large-scale agency-wide analyses to project-level cost-benefit studies • Methodologies limit the ability of the case studies to address larger questions • disparate goals and methods make them difficult to generalize or aggregate up to a national level (what is needed)

17. Example: Nanobank Study

18. Summary – What We Know • No methodological winners or losers yet - each method brings its own strengths and weaknesses • Some simplify the structure and feedback loops and rely more on data (econometric models) • Others focus more on structures and relationships (system dynamics) • Some focus on telling a coherent story (case studies) • All reveal need for better data and many lead to theoretical and methodological insights

19. Summary – What We Know Parts are studied and understood better than the whole! http://www.cs.unibo.it/schools/AC2005/docs/Bertinoro.ppt#266,11,The Blind Men and the Elephant

20. What needs to be done next?

21. Eleven Questions About the Universe A panel of US physicists and astronomers has identified a list of eleven fundamental questions about the nature of the universe that will require the combined skills of particle physicists and astrophysicists to answer. • What is dark matter? • What are the masses of the neutrinos, and how have they shaped the evolution of the universe? • Are there additional spacetime dimensions? • What is the nature of the dark energy? • Are protons unstable? • How did the Universe begin? • Did Einstein have the last word on gravity? • How do cosmic accelerators work and what are they accelerating? • Are there new states of matter at exceedingly high density and temperature? • Is a new theory of matter and light needed at the highest energies? • How were the elements from iron to uranium made? "From quarks to the cosmos", the first report from the committee on the physics of the universe set up by the National Academy of Sciences, 2001

22. 1. Identify a Set of Questions • Dr. Marburger identified some • “Are we funding all the R&D we need to defend ourselves, improve and sustain our quality of life, and compete with other nations in a globalized high-technology economy?” (Talk about a Big Hairy Audacious Question) • Are there others? What are they? Can/should the S&T policy community agree on a set? How? • A Grand Challenge meeting?

23. Prospective • Making future investments • in what • how much • why • Retrospective • Role of past investments • impact • how 2. Develop Framework • No broadly accepted frameworks exist

24. Retrospective Assessments • The Good • A thriving evaluation community • New methods being incorporated continually • Issues around data availability, timeliness, quality etc. being recognized and addressed • International cooperation (especially around data issues) • The Bad • Focus at the program level • Less emphasis on portfolio, agency or system levels • Data collection is based on an outdated innovation system • The Ugly • Efforts uneven • Long way before there is enough data of the right kind that is of high enough quality that is linked causally • Unclear how retrospective assessments are being used (or if)

25. Prospective Assessments • The Good • Need has been articulated • Tools exist in many other disciples and communities of practice – possible to borrow? • The Bad • Predictive tools are challenging to develop • Need system level understanding – difficult to model • The Ugly • Somewhat discredited • (quantitative tools) Remember systems analysis and the whiz kids? • (predictions) NSF predictions about a shortfall of scientists and engineers in the 1990s • Decisionmaking may continue to be (has always been) driven by equity and ideological criteria rather than scientific criteria • Little structured dialogue between policymakers and science policy experts • A role for evaluators to address the conceptual, methodological and computational challenges

26. 3. Review State-of-the-Art • Best practices in frameworks, tools and techniques from other communities and domains • Private sector • R&D investment decisionmaking in firms • Venture capital community • Supply chain management • Math and operations research • Algorithms and models • Hard sciences • Modeling tools and techniques • In-SPIRE

27. Emerging Frameworks: System Dynamics Modeling • Tom Fiddaman’s work with the Office of Science • Work at DOE labs to model the aging workforce in sensitive fields of research

28. Emerging Tools: Text Mining to Visualize Clusters of Research University of Cape TownStellenbosch UniversityUniversity of Witwatersrand Data Source: PubMed publications affiliated with institutions in South Africa, 2003-2005

29. Emerging Tools: Visualization of the Growth of Research Networks • Katy Borner’s work in visualization • Caroline Wagner’s work in network analyses • Gretchen Jordan - frameworks Source: http://iv.slis.indiana.edu/ref/iv04contest/Ke-Borner-Viswanath.gif

30. 4. Promote Policy Effort • Government-wide effort • Data issues • volume, quality, connectivity • NSF SRS working with the domestic and international community • Theories, frameworks and tools for modeling and analysis • NSF SBE launching new program to fund research • Best practices from other domains • Coordination • Interagency task group established to develop roadmap

31. Discussion • What are the issues/questions in the space called “science of science policy”? • How can these questions be explored (framework(s))? • What tools are techniques can be applied to address questions? How can best practices – for frameworks, models, and tools – from other domains be adapted to this space? • Other questions/comments/concerns But Mousie, thou art no thy-lane In proving foresight may be vain; The best-laid schemes o' mice an' men Gang aft agley, An lea'e us nought but grief an' pain, For promis'd joy! Robert Burns, "To a Mouse”

32. Thank You For Staying Feedback Bhavya Lal 617 331 0567 blal@ida.org “Some bargains are Faustian, and some horses are Trojan. Dance carefully with the porcupine, and know in advance the price of intimacy” Source: http://nanoandsociety.com/ourlibrary/documents/bsts-nano.pdf