Polish Agency for Enterprise Development

1. Polish Agency for Enterprise Development Evaluation of the Innovation Voucher Programme in Poland: Results, Methods and Research Opportunities Jacek Pokorski Aleksandra Jaskólska Brussels, April 14th 2011

2. Presentation • 1. Innovation Voucher Programme • 2. Aboutthe evaluation • 3. Beneficiaries’ characteristics • 4. Methodology • 5. Keyfindings • 6. The most importantrecommendations • 7. Lessonlearned - researchopportunities

3. Innovation Voucher Programme AbouttheProgramme • Pilot project 2008-2010, will be continuedin 2011 • Financed from domesticfunds • Themainobjective: initiation of contacts between micro and small entrepreneurs and science institutions • For micro and small enterprises • For research & development servicesprovided by scientificinstitutions • Voucher’sworthabout 3750 euro(15 000zł)

4. Innovation Voucher Programme Programme’sscheme PAED APPLICATION FUNDS AGREEMENT REPORT ON THE SERVICE CHOICE SCIENTIFIC INSTITUTION (serviceprovider) BENEFICIARY SERVICE REPORT ON THE SERVICE

5. Aboutthe evaluation • The evaluation focused on the first two editions of the Programme (2008, 2009) • Researchduration: september – november 2010 • The primary evaluation goal was to assess the effectiveness, utility and sustainability of the programme results • We wanted to knowmoreabout: • Characteristics of applicants, beneficiaries and projects carried out under the 1st and 2nd edition, • Programme’s influenceon startingbusiness cooperation between enterprises and scientific institutions, • Identification of theeffects, • Looking for proposals for theProgramme’smodifications

6. Methodology DESK RESEARCH PAED REPRESENTATIVES INDIVIDUAL IN -DEPTH INTERVIEWS BENEFICIARIES INDIVIDUAL IN-DEPTH INTERVIEWS Inception report N=2 N=3 APPLICANTS WHO WERE GRANTED THE SUPPORT BUT EVENTUALLY REFUSED IT INDIVIDUAL IN-DEPTHTELEPHONE INTERVIEWS UNSUCCESSFUL APPLICANTS INDIVIDUAL IN-DEPTHTELEPHONE INTERVIEWS SCIENTIFIC INSTITUTIONS INDIVIDUAL IN-DEPTHTELEPHONE INTERVIEWS BENEFICIARIES COMPUTER ASSISTED TELEPHONE INTERVIEW N=256 N=10 N=10 N=13 EXPERT REVIEWINDIVIDUAL IN -DEPTH INTERVIEWS N=2 Final report

7. Beneficiaries’ characteristics • The avarage enterprise age was 9.6 years • 66% enterprises were functioning longer than 5 years • 55.1% enterprises’ income was less than 250.000 euro, 16.8 % - from 250.000 to 1 mln euro • Only 6% Beneficiaries decided to invest their own funds in their projects (projects above 4 575 euro [18 300 zł]) BENEFICIARIES’S SECTORS INDUSTRY 29,2% PROFESSIONAL, SCIENTIFIC AND TECHNICAL ACTIVITY 22,9% TRADE AND REPAIRS 15,9% INFORMATION, COMMUNICATION 12,5% OTHER 19,8% N=256

8. Keyfindings Project’s objective Beneficiarieswereimplementingprojectswhichobjectiveswere… % N=256

9. Keyfindings Theuse of projects’ results • 79.9% Beneficiaries declared that they use project’s results (N = 167) • 75,9% enterprises currently produce products introduced or developedwithin the Programme • 78,5% use technology introduced or developedwithin the Programme • 94,7% sell product certified within the Programme • 90% exploit analysis developed within the Programme • 83,3% apply solutions supporting enterprises’ management implemented within the Programme

10. Keyfindings Effects of theProgramme • Almost 58% of Beneficiariesdeclared positive influencearising out of their participation in the Programme and c.a. 35% underlined that positive results should occur in the future. • According to the Beneficiaries’ opinions, the value of the Programme is related with: • improved quality of the offered products (almost 68% of opinions about the very high or high impact of such factor), • positive impact of participation in the Programmeon the Beneficiary’s image (approx. 61%), • competitiveness and market position (57.2%), • better management (55.5%). • The Programme had lower influence on enterprises’ income, nevertheless such influence was regarded as considerable (32.5% of respondents declared high or very high importance of the supported project’s role in this aspect).

11. Keyfindings Cooperationwiththe scientific institutions • 96.1% Beneficiaries were satisfied or very satisfied with cooperation with scientific institutions (services’ providers), 91.4% declared that no problems appeared during cooperation. • A major group of the Beneficiaries (approx. 41%) continued cooperation with scientific institutions upon completion of the service financed under the Innovation VoucherProgramme. • 46% declaredvarious plans concerning cooperation established with scientific institutions in the future in relation to new research & development projects. • Among themalmost half (c.a. 54%) intends to launch such cooperation within the next 6 months. N=256 N=118

12. Keyfindings and the most importantrecommendations The evaluation’s general conclusion isthatInnovation Voucher has positive influence on direct contacts and cooperation between micro and small enterprises and scientific institutions. Important • Recommendation 1 • The period for implementing the project should be at least 4-6 months, which shall facilitate smooth performance and limit the number of beneficiaries withdrawing from the Programme. • Recommendation 2 • Works should be commenced to launch a sub-programme offering grants of a higher value – approx. 7500 – 12500 euro (PLN 30000 – 50000 ) . If such sub-programme is launched, it shall be absolutely crucial to allow for a longer period of project implementation, approx. 6 months. • In case of such sub-programme: • A requirement should be introduced for the entrepreneur’s own contribution in the project This shall help minimize the risk of projects of questionable usefulness. The beneficiary’s own contribution cannot be excessive and should range from 10 to 20%. • Recommendation 3 • As the higher amount of grant shall stir greater interest in the sub-programme, consideration should be given to qualitative evaluation criteria included in the evaluation of applications so as to avoid the necessity of closing the enrollment process too quickly. Unfortunately, the introduction of such criteria may extend the evaluation process; hence, such criteria must be as simple as possible. Medium Medium

13. Lessonlearned Whatcould be done to improvemeasurement and conclusions ? • Better reconstruction of the Programme's theory • Focusing on projects’ quality and their sustainability • Estimating impact in a more rigorous way WHY? Programme has many underlying assummptions, which are not explicitly shown in the Programme’s documentation. There is a risk, that the methods we used didn't revealed all the Programme's causal chain failures. WHAT? Programme's assumptions (not explicitly made) HOW? Mapping out the programme’s theory (reconstruction), In-depth desk research In-depth qualitative methods An expert panel (involving experts which were implementing similar projects in other MSs) Better reconstruction of the programme's theory

14. Researchopportunities Focusing on projects’ quality and their sustainability WHY? We know that projects are useful for beneficiaries but we know little about their quality.The Programme’s requirements for service providers (scientific institutions) were not very strict, that is whyprojects’ quality verification process was limited. We also have problems with assessing how works the long-term sustainability of the Programme. WHAT? Projects’s quality and their long–term influence on enterprises HOW? Case studies (selected projects) Focus Group Interviews Benchmarking (involving similar programmes implemented in other MSs)

15. Researchopportunities • Estimating impact in a more rigorous way WHY? We do not know to whatextenttheProgrammeitselfstimulatescooperationbetween micro and small enterprises and scientific units. We rely on informationabouttheProgramme’seffectsdeclaredby thebeneficiaries, thatiswhy we needmorerigorousmeasurementrelatingreliablecounterfactualsituation. WHAT? programme’seffects HOW? Control group (problems: limited capability – for examplethose applicants, whowere granted the support but eventually refused itorother scientific institutions’ clients)

16. Other currentPAED’s evaluations of pro-innovative programmes • “The Innovation Barometer” – on going evaluation of 13 Measures of The Innovative Economy OP, 2007-2013, addressed to polish companies and the Business Support Org. • a systematic approach to tracking the situation of beneficiaries afterreceiving the support (from programme’s theory, outcomes indicators, tools for research to multi-evaluation scheme implementation- with net effect measurement in certain Measures IE OP) • Long-term impact evaluation of pro-innovative investment grant scheme, implemented during the previous financial perspective 2004-2006 • the first results are available on www.parp.gov.pl(a year after The SOP Improvement of the competitiveness of enterprises completion) • Evaluation of the quality of pro-innovative services for enterprises, delivered by the Business Support Org. under The Innovation Economy OP, 2007-2013. • the wide range of the mystery shopping techniques have been applied (evidence-based feedback from potential beneficiaries)

17. Thankyou for yourattention Polish Agency for Enterprise Development 81/83 Pańska Street 00-834 Warsaw, Poland www.parp.gov.pl For moreinformation on theevaluationinthe PAED, pleasecontactus: j Jacek_Pokorski@parp.gov.pl Aleksandra_Jaskolska@parp.gov.pl