The Data Never Lie

1. The Data Never Lie But, Do We? Los datosnuncamienten pero, ¿y nosotros? Eric N. Schreffler, ESTC 13 May 2009 ECOMM 2009, San Sebastian, Spain

2. The Data Never Lie But, Do We? The Politics and Policy Implications of Mobility Management Evaluation

3. Overview • Why Evaluate? • Why Do I Care? • How Can Evaluation be Manipulated? • How Can We Avoid the “Dark Side?” • A Challenge

4. A Disclaimer… • This is my opinion • I am being a bit harsh to make a point • Most people here do very good evaluations • We will learn a lot more this week • I am an American… we know it all

5. Why Evaluate? • Satisfy funders? • Satisfy policy-makers? • Sound management practice? • Measure progress against objectives? • Refine program or project? • Don’t you really want to know?

6. Why are We Scared to Evaluate? • It costs money • It takes a lot of time • Need to plan before project starts • Behavior change takes a long time • We are not researchers or academics

7. Why Really Are We Scared to Evaluate? • What if the results are not favorable? • Will it make me look bad? • What if I FAIL?

8. Why Do I Care? • I have been evaluating Mobility and Demand Management programs for almost 30 years • I have seen the good, the bad, and the ugly • I know MM gets marginalized • I believe in the overall effectiveness and cost effectiveness of MM/TDM

9. Manipulating Results Can Evaluations be Manipulated? How Can Evaluations be Manipulated? How Are Evaluations Manipulated?

10. Manipulating Evaluation • Focus only on “before” forecasts, not “after” results; assuming forecasts = results

11. Manipulating Evaluation • Focus only on “before” forecasts, not “after” results • Use “rules of thumb” or expected results

12. Manipulating Evaluation • Focus only on “before” forecasts, not “after” results • Use “rules of thumb” or expected results • Focus data collection only on the “converted”

13. Manipulating Evaluation • Focus only on “before” forecasts, not “after” results • Use “rules of thumb” or expected results • Focus data collection only on the “converted” • Use anecdotal stories; qualitative findings

14. Manipulating Evaluation • Focus only on “before” forecasts, not “after” results • Use “rules of thumb” or expected results • Focus only on the “converted” • Use anecdotal stories; qualitative findings • Spin results

15. Manipulating Evaluation • Focus only on “before” forecasts, not “after” results • Use “rules of thumb” or expected results • Focus data collection only on the “converted” • Use anecdotal stories; qualitative findings • Spin results • Omit results

16. Avoiding the Dark Side Traffic Engineer Mobility Manager Funding Source

17. Avoiding the Dark Side • Build evaluation into funding process • Establish credibility through scientific rigor • Pool resources and results • Use guidance offered • Use local academics

18. Advice: know your weapon Know your evaluation, love your evaluation, for one day, your evaluation just might save your life

19. The Results May Surprise You According to Congressionally-mandated study of principle funding source used in US, TDM and MM are among the most cost effective strategies for reducing emissions TRB Special Report 264

20. Just Do It!!!

21. contact Eric N. Schreffer Transport Consultant San DiegoCalifornia 001.858.538.9430 estc@san.rr.com muchas gracias parasuatenciōn