Evaluating Physical Activity Intervention Programs

1. EvaluatingPhysicalActivityInterventionPrograms Thomas Schmid, PhD Physical Activity and Health Branch CDC Atlanta, Georgia, USA

2. Overview • Evaluation Definitions and Principles • CDC Physical Activity Evaluation Handbook – 6 Steps • Evaluation issues

3. What is Evaluation? The systematic examination and assessment of features of a program and its effects. Evaluation is designed to produce information that can be used by those who have an interest in the improvement or effectiveness of an initiative or program.

4. Why Evaluate? • To assess the effects and value of a program • for accountability • for progress • to identify best practice • as a planning tool • for research

5. Match Evaluation of Health Promotion Programs to: • the program • target groups and needs • realistic achievable objectives • is the program ready for evaluation?

6. Process of conducting evaluation • Identify needs • State goals and measurable objectives • Assess progress towards objectives

7. Evaluation Terminology Efficacy Intervention tested under ideal conditions with optimal study design Effectivenes Field based intervention testing whether the intervention works as intended for a defined population in the real world Efficiency Results in comparison to the efforts required to obtain them – often cost or cost effectiveness

8. EVALUATION Measures short term objectives Formative IMPACT Develop program & resources OUTCOME PROCESS Measures long term effects (goals) Measures activities, program quality, reach

9. Formative Evaluation • What is likely to work • Best practices in health promotion • Idea or concept development • Developing and pilot testing materials • Focus groups

10. Formative Evaluation • Focus groups and interviews with partners about their needs • Reviewing best practices from other programs such as Agita Sao Paulo • Pilot testing in one company

11. Process evaluation • Program implemented as intended ? • Quality of program components • Attendance rates • Audience awareness of message • Appropriate message learned • Participant satisfaction

12. Process Evaluation • Companies enrolled • Workshops held • Promoters trained • Companies participating in community events

13. Impact & Outcome Evaluation • Short and long term effects of program • Hierarchy of effects : what level of effects indicates program success • Assessing the impact upon those who attend or participate • Community effects

14. Impact & Outcome Evaluation • Physical activity level of employees • PA level in community • Disease rates or costs

15. Program Evaluation Framework Physical Activity Evaluation Handbook

16. Program Evaluation Framework Physical Activity Evaluation Handbook • Engage Stakeholders • Include people with a vested interest in the program: implementers, partners, participants, possible opponents and decision makers. • Clearly Describe the Program • The problem being addressed, descriptions of the planning, implementation and maintenance phases of the program, and finally, defining and communicating how all phases work together (eg. the "logic model"). 1. Engage Stakeholders

17. Program Evaluation Framework Physical Activity Evaluation Handbook 3. Clearly Focus Asking a clear, straightforward question and ensuring linkage to the problem being addressed.  Includes clarifying process and/or outcome evaluations of interest 4. Develop and Implement Data Collection Design of data collection efforts, selection of appropriate control groups, designation of appropriate baseline values and decisions about how much time should be built in to adequately measure "change” 3. Clearly Focus

18. Program Evaluation Framework Physical Activity Evaluation Handbook 5. Develop Conclusions 5. Develop Conclusions Data analyses and interpretation. 6. Communicate Promotes the importance of tangible products from the evaluation, dissemination of those products, and follow-up (as appropriate).

19. www.iuhpe.org

21. Observations on Evaluation • All physical activity interventions should be evaluated. Evaluation begins with program planning • There are many good programs, but very few are well evaluated • Involve stakeholders • Community – University partnerships can be very useful

22. Observations on Evaluation • Process as well as outcome evaluation • Quantitative and qualitative • Tailor evaluation to stakeholders and situation • Evaluation is often undervalued compared with “hard science” • Adequate resources should be devoted to evaluation. 10% is recommended

23. Observations on Evaluation • Evaluation results should be used to adjust programs and inform policy • Share results with stakeholders • Academic and practical training are needed • The CDC Physical Activity Evaluation Handbook provides a good framework

24. Gracias/Thank you