Evaluation

1. Principles & Skills in Evaluating Interventions Course Tutor: Tanya Fosdick Evaluation

2. Evaluation Experiences • Please turn to the person next you and find out: • Their name • What experience they have of evaluation • How much evaluation they do in their current role • If they have any concerns about the course they wish to share • What they would like to get from the course • You will be asked to present this information

3. Aims of the session • Aims: • To increase the understanding of evaluation in road safety • To equip road safety professionals with the skills and confidence to undertake their own evaluations

4. Objectives of the course • Objectives: • Understand the basic terms and definitions used in evaluation • Consider basic principles of evaluation: setting clear objectives; developing logic models for evaluation planning; and choosing an evaluation design • Understand some different evaluation methodologies • Recognise the importance of sharing evaluation findings

5. Activity 1: Why evaluate? • In groups, discuss all the possible reasons for evaluating road safety schemes. • Produce a list of all the different reasons • Feedback to all

6. Why evaluate? • Demonstrate success (or not) • Inform policy decisions • Improve the delivery of an intervention • To share best practice • Show value for money • Be sure of “doing no harm”

7. Activity 2: Terms & Definitions • Match Terms to their Definitions • Which Term belongs to which Definition?

8. Terms & Definitions • Aim – Overall goal • Objective – What you want to achieve (specific) • Input – Resources • Output – What you are delivering/doing • Outcome – Intended changes • Monitoring – Checking progress • Evaluation – Understanding worth of the intervention • Effectiveness – Were aims & objectives achieved? • Efficiency – Ratio of input to outcomes

9. Intervention Planning

10. Aims • Aims – Can be to raise awareness; increase knowledge or skills; or change behaviour • Aims SHOULDN’T be to reduce casualties, even if this is our overall goal, as it is extremely difficult to measure the effect of campaigns on road casualties because: • Interventions are often short-term, one off and delivered to small numbers • There are other influences on road casualties, including changes in traffic, speed, roads, modes of transport • People change as they grow older and have different experience, changing the way they behave

11. Objectives • A specific outcome that the intervention is intended to achieve. An intervention may have several objectives, all of which should be S.M.A.R.T.

12. S.M.A.R.T. • Specific – clearly identify who will be affected by what is done and how they will be affected • Measurable – there are ways of measuring the achievement of the objective • Achievable – the objective can be achieved • Realistic – it is realistic, given the available resources • Time-bound – the objective can be achieved within a defined timeframe

13. S.M.A.R.T. • Specific – To increaseseatbelt wearing amongst local drivers by 10% in the 3 months after the campaign • Measurable – To increase seatbelt wearing amongst local drivers by 10% in the 3 months after the campaign • Achievable – To increase seatbelt wearing amongst local drivers by 10% in the 3 months after the campaign • Realistic – To increase seatbelt wearing amongst local drivers by 10% in the 3 months after the campaign • Time-bound – To increase seatbelt wearing amongst local drivers by 10% in the 3 months after the campaign

14. Activity 3: Objectives • Read through the activity hand-out (it provides an example of an aim and objective) • Read your group case study (A or B) • Use the group case study to try and answer the questions on aims and objectives on your own • As a group, agree aim(s) and objectives for your case study to present back

15. Activity 4: Objectives • Read the handout with the real road safety objectives on • Are they all SMART? Tick which elements of SMART each one is. • Add your objectives from Activity 3 – are they all SMART? • How could we make them more SMART?

16. www.roadsafetyevaluation.com • Is a complete guide to evaluation for ETP • Includes a toolkit to help you: • plan and • carry out your evaluation • AND • report your evaluation results • Has a glossary and topic guide to help you understand what evaluation is and help you with e.g. ethical issues

17. Evaluation of what? • To improve the intervention delivery (formative or process evaluation) • To show the effectiveness of the intervention (summative or outcome evaluation) • Both

18. What to measure? • Baseline • Qualitative • Quantitative • Experimental (most robust) • Quasi-experimental • Non-experimental (least robust)

19. Some Evaluation Designs • Pre and post intervention (with or without a control or comparison group) • Post intervention only (with or without a control or comparison group) • Post then pre intervention • Randomised Controlled Trial • Case Study

20. Post Only

21. Post then Pre Reflective

22. Pre then Post

23. Pre then Post with Comparison

24. Research Methods

25. Activity 5: Picking designs and methods • 5 interventions as puzzle pieces: • Intervention • Aim • Evaluation Design • Research Methods • In groups, match the 4 parts of the puzzle • Warning: some designs and methods are appropriate for more than 1 intervention

26. Logic Model • What is a logic model? • A causation chain of: • Inputs Outputs Outcomes • Aims, Objectives and Assumptions

28. Activity 6: Logic Models • Aims, Inputs, Outputs and Outcomes In small groups, devise a logic model for the example intervention. Use the aims, inputs, outputs, and outcomes cards as puzzle pieces and design a logic model

29. Sample Sizes • Qualitative data: ‘theoretical saturation’ • Quantitative data: sampling www.raosoft.com/samplesize.html

30. Sampling techniques – Simple Random

31. Stratified Random

32. Cluster Sample

33. Purposive Sampling

34. Snowball

35. Convenience

36. Quota

37. Activity 7: Sampling • The handout has 6 target audiences for an evaluation • Select which sampling techniques could be used for each Feedback: • Which ones could be used for each group? • Which would be the most robust for each?

38. Question Writing - Basics • Questions should be neither too broad or too narrow • Questions must relate to the intervention’s objectives • Basis for questionnaires: • Preliminary interviews/focus groups • Previous work • Literature reviews

39. Question Writing – Golden Rules Since your training are you more confident now than 12 months ago? How would you rate your cycling? Would you choose a new instructor or are you satisfied with the current one? How would you rate our ‘triple bottom line’ approach to sustainability when working with our organisation? Did the instructor follow the planned route? Were the staff friendly and helpful? Have you heard of RSA and its research? How good is your driving now that you have completed the course? Did the course meet your expectations?

40. Question Types

41. Scales • Likert scale • Subscales • On a scale of 1-10, how confident do you feel about the following aspects of driving (where 1 is not at all confident and 10 is extremely confident)? • Reverse scoring • On a scale of 1-10 how much do you enjoy the following while driving?

42. Activity 8: Question Writing • Use Speed Awareness Course Case Study to create some questions • Read the hand-out. Any questions about the evaluation or intervention? • In your group, write 10 questions for the evaluation which will be asked before and after the intervention • Report back

43. SUMMING UP CAMPAIGNS CHANNEL SELECTION EVALUATION FRAMEWORK NEEDS ANALYSIS BEHAVIOUR CHANGE TOTAL PROCESS PLANNING MODEL

44. SUMMING UP • Any questions

45. THANK YOU!