1 / 19

Selecting the Right Evaluation Method

Selecting the Right Evaluation Method. Objectives. Why should we evaluate? Which activities should we evaluate? When should we evaluate? How should we evaluate (methods)?. Why Do We Evaluate?. Design a program. (or item; e.g., curriculum, poster) Baseline, formative

slade
Download Presentation

Selecting the Right Evaluation Method

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Selecting the Right Evaluation Method

  2. Objectives • Why should we evaluate? • Which activities should we evaluate? • When should we evaluate? • How should we evaluate (methods)?

  3. Why Do We Evaluate? • Design a program. (or item; e.g., curriculum, poster) • Baseline, formative • Determine if objectives were met.(best practices) • Outcome, process, impact • Build an evidence base for ACSM • Re-design or refine a program.(lessons learned) • Process, outcome

  4. Which Activities Should We Evaluate? • Is this a new activity? • Is this an activity that may be expanded? • Have we invested a lot in this activity? • Is something unexpected happening? • Does the funder want an evaluation? IF YES ….. EVALUATE!

  5. Case Example Is this a priority program for evaluation? Why or why not? Which objectives or activities are most important to evaluate?

  6. IMPACT OUTCOME PROCESS FORMATIVE BASELINE When do we evaluate?

  7. Key Informant Interviews FORMATIVE EVALUATION Client Exit Interviews OUTCOME EVALUATION KAP Surveys Media Scans PROCESS EVALUATION Secondary Analysis Analysis of Program Data ACSM ASSESSMENT Focus Groups IMPACT EVALUATION Simple Surveys

  8. When Can We Use Quantitative Methods? Routine Surveillance Analysis of NTP Data Analysis of Project Data Population-based Surveys Population-Based Surveys Simple Surveys

  9. Use a QUANTITATIVE Method to: • Count things or people. • Measure your program against a benchmark. • Monitor trends over time. • Statistically correlate factors: Is there a connection between variables?

  10. When Can We Use Qualitative Methods? Focus Groups Informant Interviews Exit Interviews Exit Interviews Media Scans Media Scans

  11. Use a QUALITATIVE Method to: • Find out overall concerns, opinions. • Gather initial information or sense of direction. • Get details of/reasons for any problems. • Test communication materials.

  12. Combine BOTH for Best Results: • Identify key issues with a focus group or interviews. • Analyze NTP data to look for correlates or to obtain a baseline measurement. • Use surveys to measure before and after results. • Conduct interviews to assess how your process is going. • Link your final outcome data to surveillance or NTP data.

  13. Which Method and When?

  14. Factors to Consider

  15. What is the purpose of the evaluation? I am at the end of a three-year, district-level ACSM project. Did we produce the expected outputs and short-term outcomes? • Analysis of project data, focus group, or KAP survey? • I am designing a media campaign to inform my workers that TB treatment is free. What is the best way to get this message to them? What do they already know? • Routine surveillance, KAP survey, or focus group?

  16. What is the most rigorous method to answer the evaluation question? The NTP wants an impact evaluation of ACSM because these new strategies have never been used at the national level. • Experimental design, secondary data analysis, exit interviews? I am looking for a successful ACSM model that I can replicate in my district. • Case study, KAP survey, experimental design? We trained 50 DOTS nurses on interpersonal communication and counseling skills and we want to know if they are using them correctly. • Time location survey, exit interviews, secondary data analysis?

  17. How quickly do you need the results? The NTP just announced funding for ACSM to reduce default. I have to design an activity in six weeks! I need to know why defaults are high in my district. • KAP survey, simple survey, analyze surveillance data? • We have Global Fund money to implement a whole “package” of ACSM interventions to improve case detection over five years. We really need to know if the activities are effective! • Research study, informant interviews, KAP survey?

  18. What financial and human resources do I have? I am SO lucky! My donor wants a formal evaluation of ACSM and gave us a lot of money for it. AND we have partners at the School of Public Health to help us! • KAP survey, routine surveillance, research study? • My donor thinks that radio/TV/newspapers are the best way to reach our population and is giving me a lot of money to improve coverage of TB issues. But they want all of the money to go toward implementation. How can I know if it is effective? • KAP survey, media scan, focus group?

  19. Evaluation Template

More Related