1 / 26

Evaluation Methods

Evaluation Methods. Objectives. Describe the most common quantitative and qualitative methods used to evaluate ACSM. Demonstrate the role of these methods in supporting ACSM evaluation. Evaluation of Treatment Support Progra m. Quantitative. Qualitative.

retta
Download Presentation

Evaluation Methods

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation Methods

  2. Objectives • Describe the most common quantitative and qualitative methods used to evaluate ACSM. • Demonstrate the role of these methods in supporting ACSM evaluation.

  3. Evaluation of Treatment Support Program Quantitative Qualitative Interviews with 20 patients who left treatment early. _____________________________ What they experienced during treatment. Why they left treatment. Ideas for improving the treatment support system. Patient data from 10 TB clinics. _____________________________ • How many patients left treatment early. • How many weeks of treatment were completed. • % default by gender, ethnicity, age.

  4. What are Quantitative Methods? • Analyze numerical data. • Use standard tools (e.g., questionnaires). • Need precision and reliability! • Often use large sample sizes with sophisticated methods to select participants. • Results are generalized to a population. Example: The TB prevalence survey planned for Nigeria will include 49,000 people selected from 700 clusters representing six zones.

  5. What are Qualitative Methods? • Provide in-depth, detailed information on behavior, opinions, ideas, etc. • Analyze text, words, pictures, or interview transcripts instead of numbers. • Use open-ended data collection tools. • Usually smaller sample sizes.

  6. Quantitative Methods • Analysis of routine surveillance and NTP data. • Analysis of program or project data. • Simple surveys. • Population-based surveys: • KAP, prevalence.

  7. Method: Analysis of Routine Surveillance and NTP Data • Existing Ministry of Health reporting system data: • Vital registration data. • Routine reports from disease-specific programs. • Hospital data. • Data usually available; no special data collection needed. • Quality may be a concern: completeness, timeliness, accuracy may not be good.

  8. Examples: Routine Surveillance • Number of deaths due to TB in State B in 2010. • Prevalence of TB/HIV co-infection by age and gender. • Number of TB cases reported in 2012. • Percentage of MDR-TB cases among new reported cases in 2012.

  9. Using NTP Data to Prioritize ACSM Quarterly review of SS+ case notification • Which districts report an increase in SS+ case notification? • Which district reported the highest number of retreatment cases? • Where is TB/HIV co-infection the highest?

  10. Method: Analysis of ACSM Program Data • Collect and analyze key output and outcome indicators over the life of the project. • Data come directly from your project. • Quantitative indicators and sources: • Advocacy. • Communication. • Social mobilization.

  11. TB Screening Referrals from Community Health Workers, District X, 2011 57% 55% 40% 8% 18% 20% 15%

  12. Contribution of Community Health Workers to TBCase Notification in District X, 2011

  13. Give Your Data Meaning and Importance • Interpret results within a larger context. • Link project data with routine/NTP data to show contribution. • Monitoring: How many did we refer and how many were diagnosed with TB? • Evaluation: How much did our referrals ultimately contribute to overall case notification in the district?

  14. Method: Population-Based Surveys • Provide data on priority indicators that can be generalized to a population. • % of adults 15 years and older who correctly identify chronic cough as a TB symptom. • Usually requires a large research team, specialized expertise, and a full-time survey manager. • Nigeria 2011/2012 prevalence survey includes 49,000!!!

  15. What is a KAP Survey? • KAP = knowledge, attitudes, and practices • Population-based survey—very large sample size! • Face-to-face interviews with standard tool. • Provides summary data on KAP. • Averages, percentages, totals, ranges, etc.

  16. What does a KAP survey tell us? Knowledge, attitudes, and practices • What does the population know about TB? • What myths or misinformation are common? • What stigmas are related to TB? • Where do people seek care? • What media channels are used by this population?

  17. Essential Components: KAP Survey • Appropriate sampling strategy • Standardized, pre-tested questionnaire • Field protocol and manual • Trained, UNBIASED data collection team • Data analysis plan • Data use strategy • BUDGET!

  18. Method: Simple Surveys • Information from a smaller, targeted sub-group: • Village surveys. • Basic questionnaires. • Polls. • Faster and easier, but less scientifically rigorous than KAP surveys. • Cannot generalize the results to a larger population.

  19. Qualitative Methods • Focus group discussions • Key informant interviews • Exit interviews • Media scans

  20. Method: Focus Group Discussions • Small group (6–10 people). • Gather opinions, insights, thoughts, and feelings about a topic. • Led by a moderator; assisted by a notetaker. • Semi-structured guide with key questions. • May be recorded, transcribed, formally analyzed with software. • Useful for pre-testing communication materials.

  21. Method: Key Informant Interviews • Face-to-face interview with critical individuals: • Policymakers, opinion leaders. • Those who represent key stakeholders (e.g., NGO directors, association leaders). • Marginalized, vulnerable populations who do not wantto participate in groups. • Good for collecting sensitive, personal, or confidential information. • Limited perspectives.

  22. Interviews or Focus Groups? • What do patients think of a new brochure on treatment adherence? Focus Group • How can we get the directors of TB clinics and HIV centers to collaborate more? Interviews • What do HIV-positive injection drugs users know about TB symptoms and diagnosis? Focus Group or Interviews

  23. Exit Interviews • Evaluate short-term outcomes of changes to service delivery. • Standardized instrument used to gather data on patient experiences with a clinic or specific provider. • Can measure patient satisfaction and monitor provider behavior after training.

  24. Media Scans • Search media for how a specific topic is covered: • What are the most common messages about TB? • How often is TB discussed in the media? Is it a “hot” topic? • Is the information about TB accurate? • Are the messages stigmatizing? • Useful to evaluate interventions with journalists. • Formal or informal.

  25. Q&A/Resources

More Related