Process Evaluation

1. Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT

2. Session Goal To provide Participants with the information to design a process evaluation for an injury prevention project.

3. Session Objectives • Define Process Evaluation • Describe why and when to use process evaluation • Recognize the type of data collected during Process Evaluation • Describe Process Evaluation methods • Design a Process Evaluation for an injury prevention project

4. The 4 Stages of EvaluationReview Baseline Data Collection Project Implementation Changes in the Target Audience Formative Evaluation Process Evaluation Impact Evaluation Short-term: Knowledge, Attitudes & Practice Timely Feedback Outcome Evaluation Long-term: Pre-Testing Monitoring Health Outcomes

5. Process EvaluationDefined The type of evaluation used to determine if your project is: • Being implemented as planned, and • Reaching its target audience.

6. Process EvaluationDefined “Did I do what I set out to do, and did it make a difference?” Process Evaluation Impact/ Outcome Evaluation

7. Process EvaluationDefined Process Evaluation asks: • What was actually done? • Where and when was it done? • How often was it done? • Who did it, and who did they do it for?

8. Process EvaluationWhen its Used • Process evaluation planning takes place during project planning and before formative evaluation starts. • Begins immediately after your project is implemented. • Continues throughout the life of your project.

9. Process EvaluationWhy it’s Used • Allows you to make adjustments in a timely manner. • Needs of target population might change, and project may need to adapt. • Identifies any problems that occur in reaching the target population.

10. Process EvaluationWhy it’s Used • Can be used to show funding agencies the project’s level of activity. • Tells you how well your project is being implemented. • Tells other interested programs, the “how” and “why” your program works.

11. Process EvaluationLimitations • Is necessary but not sufficient in evaluating a program’s effectiveness. • Depends upon accurate record keeping and effective communication.

12. Process EvaluationMeasures Process Evaluation Implementation Target Audience Measures & evaluates project implementation Measures if your project is reaching its target audience

13. Process EvaluationImplementation Implementation activities: • Developing project’s goals and objectives • Creation of an implementation protocol • Monitoring daily operations • Data collection • Coalition building • Ensuring staffing is at proper level to meet program needs • Ensuring that staff are sufficiently trained

14. Process EvaluationImplementation Questions • What was actually done? • When and where was it done? • Who did it? • Who did they do it for? • Were any materials distributed? • What barriers or challenges were discovered? • What was the cost?

15. Process EvaluationTarget Audience Reaching the Target Audience activities: • Ensuring the target pop. is being reached • Measuring program participation by target pop. • Making materials and resources available and understood by target pop. • Determining if program is relevant to target pop. • Tracking the distribution of materials

16. Process EvaluationTarget Audience Questions • Is the target population being reached? • What was the nature of this contact? • How often and for how long was the target population involved? • Are the project’s messages and materials appropriate for the target population?

17. Process EvaluationData Collection Techniques • Project Exposure • Progress Review • Internal Audit • Target Population Survey • Project Site Survey

18. Process EvaluationData Collection Techniques • Project Exposure • Monitors all project contacts (telephone, email, classes, etc.) and materials distributed (brochures, products, etc.) • Monitors who utilizes project information (e.g., sign-in sheets)

19. Process EvaluationData Collection Techniques • Progress Review • Reviews program activities to determine if goals and objectives are being met (e.g., meetings with coalition, tribal members, courses provided) • Conducts interviews with staff and coalition members • Conducted by project staff and interested parties

20. Process EvaluationData Collection Techniques • Internal Audit • Compares implementation plan to actual activities • Documents staff efforts, resources, amount of time devoted to each task, and date of task completion • E.g., Sleep Safe Coordinator’s quarterly reports

21. Process EvaluationData Collection Techniques • Target Population Survey • Measures whether program is reaching target audience • Describes target population’s awareness of project, level of interest in the project, number who utilize project

22. Process EvaluationData Collection Techniques • Project Site Survey • Determines if materials are being distributed effectively • Monitors the number of materials used over a certain period of time • E.g., Sleep Safe smoke alarm follow-up data collection form

23. Process EvaluationInterpreting the Data The 5 process evaluation methods are now producing results… Results of Internal Audit are used to inform project staff No problems discovered: status-quo Problems discovered: timely revisions to project May require Formative evaluation to determine the problem’s cause

24. Process EvaluationInterpreting the Data Baseline Data Collection Project Implementation Changes in the Target Audience Formative Evaluation Process Evaluation Impact Evaluation Short-term: Knowledge, Attitudes & Practice Timely Feedback Outcome Evaluation Long-term: Pre-Testing Monitoring Health Outcomes

25. Process EvaluationConclusion • Process evaluation is a management tool that is used to make sure that your project is implemented as planned and on schedule. • Information collected during process evaluation can be used to make adjustments to your project. • Tells you how well your project is being implemented.

26. Process EvaluationConclusion “The only difference between stumbling blocks and stepping stones is how you use them.” (And your ability to recognize them.)

27. Process EvaluationExercise • Using the provided goal & objectives and the Process Evaluation worksheet, your group should: • Select an objective. • Describe one way you will measure how well your project is implemented and how it will reach the target audience. • Describe how you will measure the objective using one of the process evaluation data collection techniques. Time allowed: 15 minutes

28. Child Passenger Safety • Goal: ..to increase the use of child passenger safety seats & correct use of seats • Objectives: • Coalition to increase number of CPST and Instructors by end of this year. • Media advertisement of clinics, checkpoints, and importance of car seats by March 2007. • Conduct a clinic & checkpoint per community yearly, starting next year.

29. The 4 Stages of Evaluation Baseline Data Collection Project Implementation Changes in the Target Audience Formative Evaluation Process Evaluation Impact Evaluation Short-term: Knowledge, Attitudes & Practice Timely Feedback Outcome Evaluation Long-term: Pre-Testing Monitoring Health Outcomes

30. Impact Evaluation • Collects baseline information of people’s knowledge, attitudes, beliefs or behaviors. • Long-term data collection (3 to 5 years) to measure and reduce morbidity mortality.

31. Impact Evaluation • Changes in elders use (observed, self-reported) of walkers. • Changes in community members behavior in using occupant restraints (seat belt wearing for adults or car seats for children.

32. Impact Evaluation • Children bicycle helmet use (observed, self-reported) before/after a bicycle campaign. • Changes in elders use (observed, self-reported) of walkers. • Changes in number of operable smoke detectors installed/maintained in homes after a comprehensive fire prevention campaign.

33. Impact Evaluation Example: Alaska PFD Promotion Project • Drowning rate in Alaska’s YK Delta more than 3 times greater than the State average • PFD promotion project began in YK Delta • Observational surveys of PFD use were conducted after baseline data was collected • Timely changes made to increase sales and PFD use

34. Impact Evaluation Example: Alaska PFD Promotion Project Usage Rates % # Sales

35. The 4 Stages of Evaluation Baseline Data Collection Project Implementation Changes in the Target Audience Formative Evaluation Process Evaluation Impact Evaluation Short-term: Knowledge, Attitudes & Practice Timely Feedback Outcome Evaluation Long-term: Pre-Testing Monitoring Health Outcomes

36. Outcome Evaluation • Usually requires significant resources, long periods of time, and ongoing data monitoring. • Used less frequently than impact evaluation in NA injury prevention programs. • Focuses on the program’s long-term effect on its target audience. • Conducted after a program has been completed

37. Outcome Evaluation Example:Drowning Rates YK Delta vs. All-Alaska YK Starts PFD Program YK Delta 43% Reduction All Alaskan Rate Per 100,000

38. Evaluation Summary:Identify the evaluation type • Number of PFDs distributed • Decrease in the number of drinking and driving violations • Review of IP materials describing storage of poisons • Number of meetings with Tribal council to discuss possible speed limit ordinance • Number of completed suicides

39. Additional Resources www.nhtsa.dot.gov www.cdc.gov/eval/resources.htm