Measuring the Economic Impacts of Technology Transfer from Federal Labs

1. Measuring the Economic Impacts of Technology Transfer from Federal Labs Strategy and Development Branch Frederick (Rick) Kijek June 12, 2007

2. Introduction • Presently undertaking a study to measure economic impacts of NRC • Demonstrate the return on investment from NRC’s R&D programs/activities • Retrospective - 10 years • Develop baseline to measure impact of Renewal Strategy • Benchmark against other R&D performers • Align with S&T Strategies • Technology Transfer is one of the main channels included in the analysis • Several proposed approaches • Presently organizing consensus on methodology • No answers or conclusions yet “Greater Sophistication in Measuring the Impacts of Our Science and Technology Investments” Mobilizing Science & Technology to Canada’s Advantage, Page 88

3. Basic Design Tenets • Emphasis on quantitative impacts • Capture impacts in a consistent, quantitative and credible manner • Identify and conceptualize qualitative impacts • Contextualize with emphasis on relative (vs absolute) impacts • Objectivity • Adopt known and accepted analytical methods, measures and metrics • Adopt existing guidelines (TBS) for impact analysis • Involve advisorypanel of international experts, including representatives from central agencies (Treasury Board, Finance and Industry Canada) in the review of the methodology • develop “multiple lines of evidence” • Transparency & Repeatability • Understandable to NRC decision makers and central agencies • Ability to trace and repeat results • Methods and assumptions reviewed within NRC “Canada’s Federal Government will increase its accountability to Canadians by improving its ability to measure and report on the impact of S&T expenditures.” Mobilizing Science & Technology to Canada’s Advantage, Page 88

4. Proposed Analytical Methods • Adopt a number of analytical methods • Benefit Cost Analysis • Risk (Data-Variability) Analysis • Input-Output Analysis • Econometric Analysis • Comparative Analysis • Use of multiple sources and techniques • Multiple lines of evidence - validation of findings • Interweaving of impact story - results of one feed into the next • Spreading risk of data availability problems - depending on data captured “The government will improve its understanding of Canadian S&T developments and the impact of federally performed S&T....”Mobilizing Science & Technology to Canada’s Advantage, Page 88

5. Methodological Challenges • Accessing quality and reliable data • Establishing an appropriate analytic framework acceptable to stakeholders at Finance, TBS, Industry Canada • Determining the incrementality and attribution of impacts • Assigning values to difficult-to-quantify benefits (i.e. social, environmental, health benefits) and indirect (spill-over) effects of R&D activities • Limiting burden on NRC staff and stakeholders • Lack of international performance benchmarking information “… and will work with the OECD and other countries to develop metrics that will enable comparisons against international benchmarks of success.”Mobilizing Science & Technology to Canada’s Advantage, Page 88

6. Measuring Impacts from Technology Transfer • There are few quantitative studies on government technology transfer in Canada • Lack of reliable quantitative measures of utilization and impacts of technology transfer • Most of the studies in this area are based on qualitative measures • There are a few quantitative studies in the United States. • To measure Technology Transfer Impacts • Estimate how successful government labs are in terms of transferring the technology they produce • Estimate the impact on the technology recipients – firms, spin-off, other labs • Measure their collective economic impacts – multiple methods

7. Previous studies • Two qualitative and focus group studies for Canada: Warda (1999) and Zieminski and Warda (1999) • Quantitative study for Canada: CANMET Study (Partial Cost-Benefit Analysis): Belinko (1994) • Some of the qualitative studies for the United States: O'Brien and Franks (1981), Winebrake (1992), Bozeman (1992), Rogers and Bozeman (1997), Sally (2000), Link and Scott (2005) • Some of the quantitative studies in the United States: Bozeman and Coker (1991), Bozeman (1994), Bozeman (2000), Bozeman and Pandey (1994), and Adams, Chiang, and Jensen’s (2003)

8. Approach/Conclusions from Other Studies Focus Group Approach • The Conference Board of Canada(Warda, 1999): private sector points out six advantages for government laboratories versus universities. Econometric Approach • Measuring both the success of the transfer “out-the-door”& impact on technology recipient “market impact” • Bozeman and Coker (1991): Industry is increasingly viewing the federal laboratories as a potential technology development partner. • Bozeman (2000): Main comparative advantage of federal laboratories is interdisciplinary team research - difficult at universities. Expensive/unique scientific equipment/facilities of laboratories Benefit-Cost Approach • CANMET Study (1994): Positive B/C Ratio of 8/1

9. Approach/Conclusions from Other Studies (Contn’d) Partial Cost-Benefit Analysis • The CANMET Impact Study (1994): • methodology was reviewed with officials from Department of Finance and Treasury Board • The lab selected a sample of 121 research projects over a 5-year period  • All projects involved industrial partners and resulted in successful technology transfer with economic return • The lab invested $100 million in the R&D for these projects, while industry invested $264 million – a leverage of 2.6 • Industry generated $1.4 billion in actual economic wealth with additional anticipated return of $1.8 billion for a total of $3.2 billion • Based on a total R&D investment for the lab of $364 million over the 5 year period (total budget), this represented an 8 fold return on investment • The average attribution was estimated to be 56%

10. Measuring Successful Transfer Econometric Approach • "Out-the-Door" model • Once the technical good/process has been adopted by another organization, then an instance of successful technology transfer has been accomplished • A measure of technology transfer will be regressed on variables such as lab budget, the number of researcher, administrative overhead, diversity of mission • Limitation: the model not concerned if the transferred technology will have any benefit – commercial or non-commercial – for its recipient or the economy at large • Data to be collected mainly from lab directors & program administrators

11. Measuring Impact on Recipient Econometric Approach • "Market Impact" model • Identifies technology transfer success in terms of commercial contributions to the technology recipient firm and its level of activity. • Model can be based on a number of metrics for commercial impact, including # of licenses issued by the gov. lab. , recipient sales, value-added, labour productivity • A measure commercial impact will be regressed on variables such as lab budget, the number of researcher, administrative overhead, diversity of mission • Data to be collected from technology recipients

12. Measuring Impact on Canadian Economy Benefit Cost Analysis • Decide on Partial vs Full BCA • Benefits include, license fees, sales and cost savings to technology recipeients • Risk (data variability) analysis on all data Input-Output (Impact) Analysis • Use combination of data collected from technology recipients and Statistics Canada Input-Output Tables • Impact on Jobs, GDP, Value-Added Econometric Analysis • Impact on Productivity, Value-added Comparative Analysis • Benchmarking

13. Extra Slides ANALYSIS METHODS

14. Benefit-Cost Analysis • To ultimately measure if benefits of program outweigh cost to taxpayers and by how much • Present value of benefit and cost streams • Methodology based on TB guidelines • Benefits to include revenues generated, cost savings, value of services • Attributed increased commercial revenues • Cost savings/productivity improvements • Costs will be broken down by grants, services, overhead • To address issues of wealth creation

15. Econometric Analysis • To assess relationships between the data • Estimate elasticities • a change in the explanatory variable (i.e. increased R&D in that sector) will cause what change in the dependent variable (i.e. increased economic wealth or productivity growth) • Estimate complimentarity and incrementality • Address issues of • relevance – program satisfies economic needs • social return of R&D greater than private return • stimulating increased private sector R&D • wealth creation using productivity growth as a proxy

16. Comparative Analysis • To compare relative impacts of non-NRC affected firms or similar programs (nationally or internationally) • Benchmarking NRC Clients, spin-offs & technology transfers • Compare to impacts from other R&D performers, Universities, other countries • Statistics Canada Surveys (Community Innovation Survey, Manufacturers Survey, LEAP, GIFI Data) • Identify established parameters on attribution, incrementality • Deal with issues of program reach, improved technical R&D capability, improved mngmt, mrktg, finance capabilities, supporting of new knowledge creation, enhancing of commercialization, client growth

17. Input-Output Analysis • To measure national & regional impacts from expenditures • Based on national account multipliers • Estimate impacts on • Job Creation (direct and indirect), • GDP, • Imports, Exports • Compare estimates with data collected from survey of companies • Provide regional/provincial breakdown • Address issue of wealth creation

18. Risk Analysis • To address inherent uncertainty/variability in data • Primarily to be used in Benefit Cost Analysis • Establishes ranges of data and probability distribution • Uses Monte-Carlo Simulation to identify to what extent data variability impact range and distribution of benefit & cost streams • Provides impact numbers as a distributions around a mean as opposed to a point estimate