The Dutch innovation voucher and evaluation issues

1. The Dutch innovation voucher and evaluation issues Marc Van der Steeg CPB Netherlands Bureau for Economic Policy Analysis The Hague, The Netherlands m.w.van.der.steeg@cpb.nl

2. The evaluation problem • A two-way relation • causal: policy leads to more innovation • correlation: innovative firms make more use of innovation policy instruments • How to disentangle these two relations? • add covariates to the regression equation • do highbrow econometrics (e.g. matching) • or... • Controlled experiment • experimental group and control group • random allocation • difference is causal impact

3. The Dutch innovation voucherpilot 2004 • Goal: “Lead them to water and pay them to drink” (Angrist et al., 2006) • introduce SMEs to public research institutes • [market-oriented incentives for research institutes] • Characteristics • credit note, worth max EUR 7500.- • for SME’s only • no own contribution required • application-oriented research questions • placed with a defined group of institutes • valid for half a year • no restrictions on level of question or technology • 100 vouchers available • LOTTERY if demand exceeds supply

4. Example of voucher project • “Biodiesel from Africa” • Seeds of tropical plant “Jatropha” can be used for production of biodiesel • However, current oilpresses less efficient than for production of biodiesel from coleseed: 40 % of oil lost • Diligent Energy Systems used an innovation voucher and asked Technical University of Eindhoven to improve efficiency of process • PHD-student carried out experiments • Plans to do further research on adaptation of coleseed press for “Jatropha”

5. Research questions What is the effect of the innovation voucher on: • the commissioning of assignments to public knowledge institutes • Direct effect (input additionality) • Persistence effect (behavioural additionality) • Innovation (output additionality) • Product innovations • Process innovations

6. Data (1) • 1,044 applications on September 17th, 2004 • Lottery: 100 winners, 944 losers • Telephone interviews May, 2005 • 100 winners • 500 randomly selected losers • questions about actual behaviour • questions about counterfactual behaviour • Response rate • 71 winners (71%) • 242 losers (48%) • Second round of telephone interviews in September 2006

7. Data (2) • No significant differences between winners and losers in background characteristics: • firm size (e.g. 19 versus 16 employees) • region • sector • We can confidently attribute any differences in outcomes to the voucher policy instrument • however, we still add all available covariates to regression equation

8. Analysis (1): Direct effect • Data; during voucher period: • 62 out of 71 (= 87%) winners commissioned a project • 20 out of 242 (= 8%) losers commissioned a project • Effect estimates • 13% of the vouchers not used (= (71-62)/71) • 8% crowding out (= 20/242) • 79% additional assignments (= 62/71 - 20/242) • standard errors are small • Counterfactual behaviour • 76% winners say: without voucher, fewer projects • 86% losers say: with voucher, more projects • Conclusion: 8 out of 10 vouchers additional

9. Analysis (2): Persistence and innovation output effect • Persistence: no effect • Voucher winners do not carry out more assignments than voucher losers in one and a half year after voucher period. • Innovation output • indications for small effect on process improvements • no significant effects on realization of new or better products and new processes

10. Crucial issues for design and evaluation of voucher scheme • Identify measurable goals of voucher • Randomization is crucial for evaluation! • Sufficient vouchers and size of control group • Collect pre-treatment characteristics / behaviour of firms • via application form; administrative datasets • Cooperation to evaluation obligatory • also for control group of losers in lottery! • Avoid abuse • e.g. print assignment to copy shop of university • which market failure does the voucher solve for?

11. The Dutch voucher instrument after 2004 pilot • 2 new larger pilots in 2005: 1000 vouchers • Definitive instrument since 2006: • around 6000 vouchers annually • Split in small and large vouchers • 2010: introduction of private vouchers

12. Other evaluation activitities (1) Dutch cohesion policy (1) • 2007 Dutch government introduced plans to improve 83 disadvantaged neighborhoods • Substantial budget of 350 million euro per year • Neighborhood action plans with goals to improve housing, jobs, education, integration and safety.

13. Other evaluation activitities (1) Dutch cohesion policy (2) • Ranking of neighborhoods on various socio-eoconomic indicators: worst 83 have been selected • Evaluation design: • Exploit ranking with cut-off at neighborhood 83 • Exploit availability of before and after information on wide range of relevant outcome indicators • Compare development in 1-83 with that in 84-183 • Local diff-in-diff design: 50-83 versus 84-117

14. Dutch cohesion policy (3)Effect on primary school test scores

15. Other evaluation plans:Innovation and science policy • Innovation loans • Selection of proposals on basis of certain criteria / judgements • Compare performance of just selected versus just not selected companies • Scholarships for talented researchers “Veni & Vidi grants” • Selection of researchers on basis of ranking of proposals by comittee • Compare scientific output and careers of just selected versus just not selected applicants

16. Contact • Report on innovation vouchers: http://www.cpb.nl/eng/pub/cpbreeksen/discussie/58/ • Contact: m.w.van.der.steeg@cpb.nl