Impact Analysis and Counterfactuals in Practice

1. Impact analysis and counterfactuals in practise:the case of Structural Funds support for enterprise Gerhard Untiedt GEFRA-Münster,Germany Conference: Improving Evaluation Methods6th Conference of Evaluation of Cohesion Policy Warsaw, Poland 30.11.2009 – 01.12.2009

2. Content • Overview • Direct Aid to Enterprises • Evaluation process • Counterfactual Analysis • Data requirements • R&D support to firms in Thuringia • Conclusions

3. 2. Direct Aid to enterprise • Direct grants or subsidies to reduce the (user) cost of capital • to build up and modernise the enterprise capital stock • to induce more R&D investment at the enterprise level • Target variables: • higher investment, higher R&D • more employment • higher productivity • better long-term development • Economic Rationale: Theory of the firm, R&D and public goods

4. 2. Direct Aid to enterprise Cohesion Policy 2000-2006, including n+3: 163 billion euro Approximately 33 billion euro as direct aid to firms Distribution of Structural Funds by economic categories, in % Source: DG Regional Policy (2009), own calculations

5. 3. Evaluation Process • An ideal evaluation process can be looked at as a series of three steps (Fay,1996) • Microeconometric evaluation • The impacts of the measures on the individual firm should be estimated • Macroeconomic evaluation • It should be examined if the impacts are large enough to yield net social gains if all spillover effects and side-effects are taken into account • Effectiveness- or cost-benefit analysis • It should be examined if this is the best outcome that could have been achieved for the money spent

6. 4. Counterfactual Analysis • Potential-Outcome Approach • Measuring the causal effect of direct aid to enterprise on some outcome variable • Notation: • Y1i : outcome if enterprise i received grants • Y0i : outcome if enterprise idid not get support • Di : indicator (0,1) signals enterprisei did (1) or did not (0) receive direct aid • Yi : Y0i + Di(Y1i – Y0i) • X : set of firm characteristics (branch, number of employees, turnover, etc.) • Causal effect for enterprise i:

7. 4. Counterfactual Analysis Fundamental problem of evaluation: Y1 is observed for those who received direct aid Y0 is given for those who did not receive direct aid Y1 and Y0 can never be observed for one firm at the same time

8. 4. Counterfactual Analysis Treatment effects and selection bias Population Average treatment effect (ATE) Average treatment effect of the treated (ATT) Policy relevant effect is given by (5) But second term in (5) is not observable Main task: Identification of an unbiased estimator

9. 4. Counterfactual Analysis Only if the outcome is not effected whether a firm received direct aid or not the population average of the non-receiving firms can be used as an unbiased estimator This assumption is normally not fulfilled if non-experimental data is used. Firms are not randomly applying for subsidies and / or are not chosen randomly by the authorities • Selection: • Observables: Branch, Age, Profitability etc. • Unobservables: motivation of the entrepreneur, administration decision rules etc.

10. 5. Data requirement • Data availability is crucial to perform an impact analysis by counterfactual methods • Cross-sectional data (one time survey) • Panel data (the same firms are surveyed several times) • Panel data allow more advanced methods to be used • In most EU Member States detailed databases at the firm level that include a treatment variable (direct aid) are not at hand • Need to generate data by specific surveys • Costly and time consuming

11. 5. Data requirement • Firm-specific variables for the statistical analysis: • Outcome variables: investment, R&D spending, employment, productivity, • Exogenous variables: Branch, Size, Age, share of intermediate inputs, capital stock, legal form • Treatment variable (policy impact): EU-public funding or total sum of overall public funding in the establishment

12. 6. R&D Support in Thuringia • One time survey of Thuringian firms concerning their R&D behaviour and investment support • Performed in 2004, covering the period from 2001-3 • Cross-section data for 1484 establishments • 284 firms in the dataset received R&D subsidies (either regional, national or supranational EU-wide) • Information concerning firm characteristics and performance variables (employment, investment, innovation performance etc.) • different measures for enterprise support including R&D-support

13. 6. R&D Support in Thuringia

14. 6. R&D Support in Thuringia • Propensity Score Matchting • Steps involved in the analysis: • Estimating the propensity score for each unit in the sample(estimate the individual probability that a firm received support, based on observable firm characteristics) • Matching the units using the estimated propensity scores • Assess the quality of the matching • Estimate the impact and its standard errors

15. 4. Counterfactual Analysis • Target variables: • General innovativeness(binary dummy for patent application between 2001-2003) • Total number of patents, Number of R&D employees • R&D Expenditures and Intensity (=R&D expenditure as share of total turnover) • Observable firm Characteristics: • Age, Export and Import quota, legal status, Branch, Ownership, Number of employees etc.

16. 4. Counterfactual Analysis Means of control variables differ significantly between treated and untreated firmsbefore the matching procedure is applied. Afterwards matched firms are “identical” in their observables!

17. 4. Counterfactual Analysis Matching Results with ATT = Average Treatment Effect on the Treated:

18. 4. Conclusions • Significant share of the Structural Fund is spend on direct aid to firms • Little is known at the micro (firm) level about the impact • Counterfactual analysis could act as a standard instrument • But: data requirements are high! And better data could help! • Significant gains from these impact analysis can be expected