Practical Issues in Applying CIE

1. Practical Issuesin Applying CIE Daniele Bondonio University of Piemonte Orientale

2. How to incorporate policy relevant issues in CIE • Estimating average treatment effect for all the treated units is often non-informative….”black box evaluation” • Relevant policy issues can be incorporated in CIE by estimating different impacts for different types of treated units or different types of interventions • E.g. [enterprise support] different impacts can be estimated for: • -soft loans, grants, technical assistance • -different economic values of the subsidies • -different types of assisted firms (small, large) • (textile, chemicals, …services,…..)

3. E.g. job training: • -different ages of participants • -different education levels • -previous work experience • Urban revitalization: • -different degrees of initial distress • Tourism promotion programmes: • -cultural events • -infrastructure improvements • -urban renovations

4. When different impacts are estimated for different groups of treated units &/or different variations of how the treatment is provided CIE can offer insights on how a programme works

5. Single-programme vs multiple- programme evaluations • E.g. for enterprise support, local economic development ……multiple different programs are frequently available and affecting a same outcome variable of interest • (firm sales, productivity, employment, investments…. indicators of quality of life from survey-data……) • Caution with single-programmeevaluations using outcome variables that can be affected by other programmes not included in the analysis

6. Solution: try to include all programmes in your analysis (e.g. enterprise support taking into account multiple programs) • e.g. enterprise support: estimate the impact of different economic intensity of support, different types of incentives • II) Solution: try to focus on intermediate outcomes that are affected solely by the programme • Y=number of tourism visits instead of more distant outcomes like local employment (for tourism promotion programmes) • Y= innovation outcomes instead of sales growth, productivity…(for R&D support to enterprises)

7. Measuring the timing of the intervention • E.g. enterprise support. Data on program incentives • -dates on when the application was approved • -dates on incentive payments (1 installment, 2 installment…..) • Which dates are to be used in measuring the beginning of the treatment? • Overlooked issue with very strong consequences on impact estimates • -when should we locate T? Approval 1st payment 2nd payment Time

8. If the programme support is wrongly placed in a time earlier than the time in which the outcome of interest could be potentially affected, the outcomes of such time would be erroneously considered as exposed to the treatment • By contrast, if a programme intervention is wrongly placed in a time later than the time in which the outcome of interest could be potentially affected, the outcomes of such period would be erroneously considered as not-exposed to the treatment

9. How to measure changes in the outcome variable? • Percentage change is often used: • D%= • D% adequate for many CIE (e.g. unit of observations: individuals in need of assistance, geographic areas) • D% to be used with caution for enterprise support: the outcomes produced by the program (against the estimated counterfactual) has a social utility which may be independent from the initial dimension of the assisted firms [1_example] (Ypost- Ypre) (Ypre)

10. How to choose the control variables • With pre- post- intervention data, what really matters is separating changes due to the program from changes due to other factors • What characteristics may be different between treated and non-treated units that put non-treated units at risk of being exposed to different external factors generating changes in the outcome at the same time of the intervention • Even with discontinuity check with control variables that • units around the threshold are indeed similar: this is done even with pure randomized experiments (with small numbers sometimes randomization is unlucky….)

11. “Attrition bias” in outcome data • “Attrition bias” is generated when units with particularly bad (or good results) drop out from the data used to construct the outcome variable • Overlooked issue with strong consequences on impact estimates • E.g. • enterprise support (balance sheet data contains only corporations: treated firms with bad results may close or may loose their corporate status and drop out from the data) • Intuitively: “attrition bias” = programme impact is estimated only for the treated firms with no bad results)

12. E.g. • R&D and innovation support: treated firms (can be start-ups, small firms) doing particularly well may be lost in the data because of changes in their company names or corporate status occurring in the growing process • Enterprise support, Job training and social programmes: treated & non-treated units with particularly good (or bad) results may not be anymore available to answer the questionnaires or interviews used to build the outcome variable • “attrition” bias” can severely compromise CIE • check for possible reasons why units may • drop out from the data or check if characteristics of respondents are similar to non-respondents

13. Long-term program effects & CIE • For many programmes (e.g. local economic development, enterprise support, R&D-innovation support) extreme caution is needed in attempting to estimate long-term effects • In the medium and long-run it is likely that a positive programme impulse spreads (with positive or negative spillovers) also to the non-treated units • Outcomes from the non-treated units become affected by the intervention and cannot be anymore used to estimate the counterfactual

14. Regional macro-effects & CIE • In principle, every type of policy may produce long-run impacts on macro-economic outcomes measured at regional/country level • Estimating such regional impacts “spill-over effects” is to be avoided when the economic importance of the activities produced by the policy is negligible • compared to size of the regional economy and to the importance of the large number of socio-economic events (unrelated to the program intervention) that do affect the macro regional/country outcomes

15. When the economic importance of the activities produced by the programme is not negligible compared to size of the regional economy proceed as follows: • I) Use rigorous CIE the proximate impacts of the program intervention on firm-level (e.g. how much of the subsidized investment would have been done anyway?) • II) Include the results of CIE in macro-economic regional simulation models yielding multiplying effect for the regional economy • Keep in mind: in the absence of CIE the multipliers used by regional simulations would be applied directly to measures of programme activity (over-estimating the macro-regional impact)

16. Continuing programmes • With similar programmes available at pre-intervention times, pre-intervention data may have been affected by early treatments • Caution in using pre-intervention characteristics as control variables if you do not include in the analysis the early rounds of the treatment • Make efforts in gathering data to detect which units were treated also in pre-intervention times

17. Practical advantages of PSM • Post-intervention outcome data are not always readily available • (delays in releasing recent data by statistical offices or need of collecting data through questionnaires, interviews, direct observation….) • Estimation of PS requires exclusively pre-intervention data on observable characteristics • PSM reduces the number of non-treated unit in which post-intervention outcome data has to be collected

18. How to choose the outcome variable of the evaluation • Designing CIE starts from understanding the rationale of the programme intervention • (….. what is the market imperfection/negative externality that the policy wants to correct) • E.g. Enterprise support policies: • A) Aimed at producing firm-level investment, that would never take place in the absence of the program intervention (correction of credit market imperfections)

19. B) Aimed at boosting business activities in distressed areas • Rationale: market forces produce non-optimal allocation of economic development with negative externalities (e.g. urban sprawl, traffic congestion, pollution, abandoned areas that may be conducive to crime…..) • Goal: modifying the geographic allocation of firm-level investment (e.g. new investments wanted in Region A, not in Region B) • C) Aimed at boosting business activities in times with economic crises (Countercyclical policies) • Goal: modifying the temporal allocation of firm-level investments (firms activities wanted in time I, not in time II)

20. Example I Time FIRM Y (Non-treated) Region A (disadvantaged) FIRM X (treated) Region B Region B Region A (disadvantaged) +1 Million € investment +1 Million € investment

21. For policies A) [aimed at correcting credit market imperfections] the outcome variable has to keep tracks of all firm-activities (e.g. investments) recorded anywhere: • Example I)= zero impact • For policies B) [targeting distressed areas] the outcome variable has to keep track of firm-activities recorded solely in region A (disadvantaged) • Example I)= positive impact • For policies C) [countercyclical interventions] the outcome variable has to keep track of firm-activities recorded solely in time I • Example I= uncertain( impact depends on • the timing of the investments, not location)

22. Example II Time FIRM X (Assisted) FIRM Y (Non-Assisted) Period II +1 Million € investment +1Million € investment Period I

23. For policies A) [aimed at correcting credit market imperfections] the outcome variable has to keep tracks of all firm-activities recorded at any time: • Example II= Zero impact • For policies B) [targeting distressed areas] the outcome variable has to keep track of firm-activities recorded solely in region A (disadvantaged) • Example II)= uncertain (impact depends on • location of the investment not time) • For policies C) [countercyclical interventions] the outcome variable has to keep track of firm-activities recorded solely in time I • Example II= positive impact

24. CIE with survey outcome data • If surveys are planned to cover both treated and non treated units, all CIE methods can be applied on survey outcome data • PSM can be used to generate surveys outcome data: • Surveys are to be run on: • all treated units + the matched non-treated units • Radius matching procedures are preferable to ensure to have a representative sample of comparable non-treated units