1 / 18

Theory-Based Roadmap for Prospective Evaluation and Innovation Policy

This presentation explores the concept of prospective evaluation in the context of innovation policy. It discusses national interest and models of innovation, theories, and provides an example. The goal is to provide policymakers and researchers with a scientific basis for assessing the impacts of science and engineering enterprise and improving understanding of its dynamics.

solano
Download Presentation

Theory-Based Roadmap for Prospective Evaluation and Innovation Policy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What Might a Theory-Based Roadmap for Prospective Evaluation and Developing Innovation Policy Look Like? Presented at American Evaluation Association Conference November 2009 Gretchen Jordan, Sandia National Laboratories gbjorda@sandia.gov Portions of the work presented here were completed for the U.S. DOE Office of Science by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the author. SAND Number: 2009-7359C

  2. Outline • Prospective evaluation in context • National interest (SoSP, SciSIP) • Models of what is known about innovation, what we need to know • Theories and an example • Conclusions G. Jordan AEA November 2009

  3. Evaluation in the Policy Cycle Technology Assessment Wolfgang Polt 30-10-2007 Foresight Technology Roadmapping G. Jordan AEA November 2009

  4. National Interest: SoSP and SciSIP • The science of science policy (SoSP) is an emerging field of interdisciplinary research, the goal of which is to provide a scientifically rigorous, quantitative basis from which policy makers and researchers can assess the impacts of the Nation’s scientific and engineering enterprise, improve their understanding of its dynamics, and assess the likely outcomes. John Marburger April 2005 • A National Science and Technology Council (NSTC) Interagency Task Group (ITG) • The Science of Science & Innovation Policy (SciSIP) program was established at NSF in 2005. G. Jordan AEA November 2009

  5. SoSP Workshop in December 2008 Primary Conclusion of SoSP Roadmap: “Expert judgment” remains the best available decision support tool for science policy makers, but a nascent community of practice is emerging in the science policy arena that holds enormous potential to provide rigorous and quantitative decision support tools in the near future. ” The White House SoSP Interagency Task Group should take the lead to set the Federal agency research agenda. G. Jordan AEA November 2009

  6. White House S&T Priorities for the FY 2011 Budget Agencies should describe in their budget submission how they are • prioritizing activities toward four challenges and strengthening four cross-cutting areas (which include productivity of research institutions) • Expecting outcomes of research in above areas, providing quantitative metrics where possible • Building capacity to rigorously evaluate programs, and how assessments have been used to eliminate or reduce programs • Operating in the open innovation model and supporting long term high-risk, high payoff research Agencies will: • Develop outcome oriented goals for S&T, target investment toward high performers, develop ‘science of science policy” tools that can improve management and assessment of impact -Peter Orszag, John Holdren, August 4, 2009 G. Jordan AEA November 2009

  7. The SoSP Roadmap 10 Science Questions The National Imperative Theme 1: Understanding Science and Innovation Theme 2: Investing in Science and Innovation Theme 3: Using the Science of Science Policy to Address National Priorities Science Questions 1. What Are The Behavioral Foundations Of Innovation? 2. What Explains Technology Development, Adoption And Diffusion? 3. How And Why Do Communities Of Science And Innovation Form And Evolve? 4. What Is The Value Of The Nation’s Public Investment In Science? 5. Is It Possible To “Predict Discovery”? 6. Is It Possible To Describe The Impact Of Discovery On Innovation? 7. What Are The Determinants Of Investment Effectiveness? 8. What Impact Does Science Have On Innovation And Competitiveness? 9. How Competitive Is The U.S. Scientific Workforce? 10. What Is The Relative Importance Of Different Policy Instruments In Science Policy? Findings Recommendations Source: J. Lane, April 2009 G. Jordan AEA November 2009

  8. Parts are studied and understood better than the whole! Source: Bhavya Lal, STPI, at AEA 2006 http://www.cs.unibo.it/schools/AC2005/docs/Bertinoro.ppt#266,11,The Blind Men and the Elephant G. Jordan AEA November 2009

  9. A Science of Science and Innovation Policy must build a theory that connects levels Research Team Research Organization The Sector’s Idea Innovation Network The Sector’s National and Global Context micro meso macro G. Jordan AEA November 2009

  10. SoSP Roadmap Questions Rearranged into a Three Level Logic Model Investment, incentives, Use 7 Determinants of investment effectiveness 10 Relative importance of policy instruments 8 Impacts on competitiveness, etc. Macro Meso 5 4 6 2 Understand technology development & diffusion Anticipate effects of Scientific discovery Anticipate effects of science on R&D The (non-linear) S&T, R&D process Assess real time value of new knowledge Micro 1 3 9 People & organizational inputs & incentives Understand behavioral foundations Understand network behaviors Science Workforce competitiveness Understanding a multi-level eco-system Draft by G. Jordan 12/12/2008 G. Jordan AEA November 2009

  11. A National Innovation System Model A National Innovation System Model Framework conditions Framework conditions Demand Demand Financial environment; taxation and Financial environment; taxation and Consumers (final demand) Consumers (final demand) incentives; propensity to innovation incentives; propensity to innovation Producers (intermediate demand) Producers (intermediate demand) and entrepreneurship; mobility and entrepreneurship; mobility Education and Education and Political system Political system Industrial system Industrial system research system research system Professional education Professional education Large companies Large companies Government Government Intermediaries Intermediaries and training and training Research Research institutes institutes Brokers Brokers Higher education Higher education Mature small/ medium Mature small/ medium Governance Governance and research and research enterprises ( enterprises ( SMEs SMEs ) ) New, technology New, technology - - Public sector Public sector The potential reach The potential reach RTD Policies RTD Policies based firms based firms research research of public policies of public policies … … Infrastructure Infrastructure Banking, Banking, IPR and IPR and Innovation and Innovation and Standards Standards venture capital venture capital information information business support business support and norms and norms Source: Arnold and Kuhlman, 2001 The science must explain relationships among institutions G. Jordan AEA November 2009

  12. Connectivity and Throughput Production, Refinement Micro, meso, macro impacts 6 Diffusion and use 9 10 8 7 Marketing R&D, Quality R&D Engineering & manufacturing R&D The science must explain connections among arenas of research and development Source: G. Jordan, 2007. Modified from R. Cooper/ Exxon’s Stage Gate, Hage & Hollingsworth’s Idea Innovation Network G. Jordan AEA November 2009

  13. Socio-cultural/market environment • Market structure • Market segments • Prior practice • Culture and norms • Innovativeness Communication field • Broadcast • Contagion Feedback Awareness Implementation Confirmation Persuasion Decision Continued adoption Later adoption Adoption Characteristics of the decision-making unit Product Characteristics Discontinuance Continued rejection Rejection • Adopter type • Personality type • Communication behavior • Socio-economic status • Relative advantage • Compatibility • Complexity • Trialability • Observability The science must understand Diffusion and relate it to R&D Source: Everett Rogers 1994 as modified by Innovologie, LLC. 2005

  14. Socio Socio - - cultural/market cultural/market environment environment • • Market structure Market structure • • Market segments Market segments Communication field Communication field • • Prior practice Prior practice • • Broadcast Broadcast • • Culture and norms Culture and norms • • Contagion Contagion • • Innovativeness Innovativeness Feedback Feedback Awareness Awareness Implementation Implementation Confirmation Confirmation Persuasion Persuasion Decision Decision Continued adoption Continued adoption Adoption Adoption Characteristics of the Characteristics of the Product Product Later adoption Later adoption decision decision - - making unit making unit Characteristics Characteristics Discontinuance Discontinuance • • Adopter type Adopter type • • Relative advantage Relative advantage Rejection Rejection Continued rejection Continued rejection • • Personality type Personality type • • Compatibility Compatibility • • Communication Communication • • Complexity Complexity behavior behavior • • Trialability Trialability • • Socio Socio - - economic economic • • Observability Observability status status All this information is useful to predict where and how policy makers can intervene to achieve desired goals Interventions at micro, meso, and/or macro levels? G. Jordan AEA November 2009

  15. Theories that could be integrated to understand how we can drive innovation Research Team Management of innovation literature, learning theory Research Organization Organizational innovation theories Research Profiles theory Science/technological Sector Idea Innovation Network on S&T/R&D process Network theories Diffusion theory Sector economic models National and global context Modes of coordination theories Institutional and institutional change theory Policy decision making –theories of G. Jordan AEA November 2009

  16. High risk capital – available where Capabilities – Level, mix, availability Modes of coordination – effective? Socio economic outcomes Technical progress Network connectedness Organizational profiles – do attributes match the profile? Portfolios - need more/ less radical, large scope? RTD arenas – are there sufficient funds One possible decision tool to identify bottlenecks, policy objectives & effectiveness • Policy Objectives • Structural • Technical Macro- Institutional Rules as they affect the sector Meso - Performance by Tech sector and arena Commercialization research Quality research Basic research INNOVATION Manufacturing research Applied research Development research Micro - funds allocation by arena and profile Source: Jordan, Hage, and Mote, 2006, 2007, 2008 G. Jordan AEA November 2009

  17. Conclusion • Innovation occurs within a multi-level, complex, dynamic eco-system • Prospective evaluation predicts • Prediction requires understanding, characterization, theory • There are theories that can be used now • Synthesis of existing theories and building new theories are needed going forward. G. Jordan AEA November 2009

  18. Selected References Arnold, E. (2004). Evaluating research and innovation policy: A systems world needs systems evaluations. Research Evaluation, 13(1), 3-17 Hage, Jerry, G.B. Jordan and J. Mote (2007). A Theories-Based Innovation Systems Framework for Evaluating Diverse Portfolios of Research: Part Two - Macro Indicators and Policy Interventions. Science and Public Policy, 34(10): 731-741. Jordan, G. B., Hage, J., & Mote, J. 2008. A theories-based systemic framework for evaluating diverse portfolios of scientific work, part 1: Micro and meso indicators. In C.L.S. Coryn & Michael Scriven (Eds.), Reforming the evaluation of research. New Directions for Evaluation, 118, 7–24. Jordan, G.B. 2006. Factors Influencing Advances in Basic and Applied Research: Variation Due to Diversity in Research Profiles. In Innovation, Science, and Institutional Change: A Handbook of Research, J. Hage and M. Meeus (eds). Oxford University Press: Oxford, 173-195. Mote, J., Y. Whitestone, G. Jordan and J. Hage. 2008. Innovation, Networks and the Research Environment: Examining the Linkages. International Journal of Foresight and Innovation Policy 4(3): 246-264. Reed, John H, G. Jordan, Using Systems Theory and Logic Models to Define Integrated Outcomes and Performance Measures in Multi-program Settings, in Research Evaluation, Volume 16 Number 3 September 2007. G. Jordan AEA November 2009

More Related