MOBILITY MANGEMENT EVALUATION: The Long and Short of It

1. MOBILITY MANGEMENT EVALUATION:The Long and Short of It CIVITAS Training 7 May 2014 Florence, Italy Eric N. Schreffler Transport Consultancy ESTC San Diego, CA USA

2. OUTLINE My experience Why evaluate? and for whom? Forecasting vs. measurement Principles of good evaluation Overall framework for MM evaluation U.S. examples – California and Washington D.C. Transitioning from short-term to long-term evaluation The politics of MM evaluation Common mistakes Top ten tips for effective evaluation

3. Quick Bio • MM/TDM evaluation for over 30 years • Written/contributed to several seminar guidebooks • Involved in several EC-funded projects • MOST – MOST MET • MAX – MaxEVA • CIVITAS GUARD • CIVITAS II –final brochure • CIVITAS PLUS training • Currently leading evaluation of TDM component of 6 U.S. Urban Partnership Agreement projects

4. Why Evaluate? And for Whom? • Good management practice! • Measure progress against objectives • Output or outcome objectives • Program or public policy objectives • Integral to performance-based planning

5. Why Evaluate? And for Whom? • Benchmark program against peers • Compare cost effectiveness to other solutions • Marginal cost to accommodate a car vs. cost to reduce a car trip • e.g., LA Metro evaluation performance-based planning • Satisfy funding entities • Satisfy policy or oversight boards

6. Know your evaluation, Love your evaluation, One day your evaluation may save your (life) program. Don’t be afraid of evaluation!!!

7. Forecasting vs. Results • Evaluation can be defined as: • A priori estimation of expected impacts (forecasting) • Ex post measurement of outcomes (results) • Forecasting • Predicting what could/should occur • Using comparative case studies, models, sketch planning, SWAG • Measurement • Quantifying what has actually occurred • Using counts, surveys, etc.

8. PROST! Principles for Sound Evaluation Practical Rigorous Objective Standardized Timely

9. FRAMEWORK FOR EVALUATION • Important to plan for evaluation • Have plan to frame approach and tasks • Should include: • Purpose and objectives • Data collection methods • Analysis methods • Budget • Schedule • Reporting • Use conceptual framework - MaxEva

10. MaxEva Assessment Levels MaxEva – The EPOMM Evaluation Tool – www.epomm.eu/maxeva

11. European Example – SwedenEmployee Public Transit “Test Rider” Pass Pilot MaxEva – The EPOMM Evaluation Tool – www.epomm.eu/maxeva

12. U.S. Examples TDM and Highway Reconstruction US101 Cuesta Grade CA • Reconstruction mitigation • $730,000/year for TDM • Three elements: • More commuter express buses • Vanpool promotion/subsidies • Carpool fuel incentive • Surveyed all participants • Removed 310 cars daily • Carpool incentive most cost effective ($3.36/VTR/day)

13. U.S. Examples Regional TDM Program Commute Connections Washington D.C. • Triennial evaluation since 1997 • Consistent approach (MaxEva) • Evaluate total impacts = travel, emissions, energy (e.g., 118K fewer car trips per day) • Evaluate separate impacts • Ridematching • Employer outreach • Mass marketing • Guaranteed ride home • Bike to work • Carshare • Telework • Incentives

14. SHORT-TERM TO LONG-TERM Short-run generally covers one year or year or duration of project funding duration. Uses before and after data Long-term can include projecting impacts into future (BCA, lifecycle) Or can involve time series data for program over years Key is planning and consistency

15. RESEARCH There may be pressure to: - subvert findings - document desired outcomes - spin the results The Politics of Evaluation

16. Common Evaluation Mistakes Putting off evaluation Underfunding evaluation Assuming “projected” = “actual” results Incorrectly comparing projected to actual Projecting target group findings to entire population Ignoring causality and externalities

17. Common Evaluation Mistakes (con’t) Confusing outputs with outcomes Changing methods/assumption mid-evaluation Assuming all mode shift from drive alone Ignoring access mode to new travel options Ignoring non-response in surveys Giving into pressure to change findings

18. Top Ten Evaluation Tips • Get help, at least the first time • Plan, plan, plan • Plan for evaluation • Performance-based plan • Stick to the plan • Budget for evaluation

19. Top Ten Evaluation Tips • Keep findings simple! • Don’t be afraid of what you may find! • Be confident! No whining!

20. Top Ten Evaluation Tips • Seek local default factors (e.g. trip length) • Use standardized methods/tools/guidance • Learn to talk like an engineer

21. Top Ten Evaluation Tips • JUST DO IT!

22. Key Resources MaxEva – The EPOMM Evaluation Tool www.epomm.eu/maxeva Evaluation Matters: A Practitioners’ Guide to Sound Evaluation for Urban Mobility Measures www.civitas.eu/content/evaluation-matters Integrating Demand Management into the Transportation Planning Process: A Desk Reference (Chapter 9) www.ops.fhwa.dot.gov/publications/fhwahop12035/index.htm Canadian TDM Measurement Toolbox www.tc.gc.ca/media/documents/programs/tdm-toolbox.pdf

23. Eric N. Schreffler Transportation Consultant ESTC San Diego, CA, USA estc@san.rr.com Grazie per la vostraattenzione!