Long term evaluation – summarizing “How-to-do-it” CIVITAS Thematic Groups, Interactive training

1. Long term evaluation – summarizing “How-to-do-it” CIVITAS Thematic Groups, Interactive training 7 May 2014, ECOMM 2014, Florence, Italy Anders Wretstrand, Lund University

2. « How-to-do » long term evaluation • Each measure • has one or more objectives • may be a combination, a bundle, of measures • Objectives may be like • Modal shift • Vivid city centre • Coordinated urban freight • These have to be operationalized into indicators that are • Linked to objectives • Quantifiable • Measurable • Each objective relate to several measures • Each measure should be linked to several indicators

3. Impact and process evaluation • Impact evaluation • is carried out in several steps • starting with a baseline study (before) • after implementation (after - short term, 1st or 2nd year) • after “a while” (after - long term, 5th or 10th year) • what matters is what impacts the measure has on the particular area that the measure aims at, revealed by monitoring indicators • Process evaluation • a means to facilitate dissemination and transferability of the results • tells the story of • planning and design • implementation and operation • reveals the “road to success” (or failure)

4. Short and long term • Short term evaluation • generally covers one year or year or duration of project funding duration • uses before and after data • Long term evaluation • can include projecting impacts into future • Forecasting, scenario-building • can involve time series data for programs/measures over years • running, ongoing surveys • planned, repeated long-term effect surveys • Different results… • no impact evidence on short term – but positive on long term • impact on short term – but no “up-scaling” on long term • in both cases: the process (the story) is important to capture!

5. About effects – impact assessment Effectofmeasure Beware! Impact Measure Time After - Short term Long term Before

6. About effects – impact assessment Effectofmeasure Impact Measure Effectofotherfactors ”Business as usual” Time After - Short term Long term Before

7. Evaluation design options The ideal situation withcontrol site – nicebutquite rare. To strive for! Baseline Impact I Impact II Measure site Measure Before After I After II Changes? Control site Before After I After II Changes?

8. Evaluation design options Control site withoutbaseline Baseline Impact I Impact II Measure site Measure Before After I After II Changes? Control site After I After II Changes?

9. Evaluation design options No baseline data Impact I Impact II Measure site Measure After I After II Changes? Control site After I After II Changes?

10. Evaluation design options No control site to comparewith Baseline Impact I Impact II Measure site Measure Before After I After II Changes?

11. Evaluation design options No control site or baseline data to comparewith Impact I Impact II Measure site Measure After I After II Changes? Common situation whenasked to evaluate!

12. Evaluation design options No control site, previousevaluation or baseline data to comparewith Impact Measure site Measure After Changes? Common situation whenasked to do long-term evaluation!

13. Examples – measures from DYN@MO • Cases • planning: SUMP • health and accessibility: healthy walking • public transport: more efficient trolley buses • mobilit management in residential areas

14. Examples – SUMP Oneof the objectives and indicators Verysuitable for long term evaluation!

15. Examples – Planning for healthy walking Oneof the objectives and indicators Verysuitable for long term evaluation! But still……difficult to assess long term effects, sincepeople age!!!

16. Examples – More efficient trolley buses Oneof the objectives and indicators Easy to do short term evaluation as well! Long term evaluationrevealsif system e.g. requiresmoremaintenance. Baseline is less of a problem whendealingwithtechnical data!

17. Examples – Electromobile living Oneof the objectives and indicators THEORETICALLY easy to do short term evaluation, butmightonlydetectattitudes and awareness! Long term revealsifpeoplesubstantiallyhavechangedtheirbehaviour, e.g. got rid of 2nd cars

18. Finally • The « How-to-guide on long term evaluation» from CiViTAS CAPITAL • will be available for all participants after approval as a final document • through us or Mobiel21 • Now, as a hand-out • Checklist: recommendations for long term evaluation of MM activities • If you or anyone else have done a previous evaluation of the measure, or • If no previous evaluation exists

19. Checklist: based on existing short-term evaluation

20. Checklist: no short-term evaluation available

21. Thank you! Anders Wretstrand Tom Rye Contact Details Lund University P.O. Box 118, SE-22100 Lund, SWEDEN anders.wretstrand@tft.lth.se tom.rye@tft.lth.se http://www.civitas.eu