PEN: CDC Guidance for Developing Plans and Reports 2011 TB ETN/PEN Conference

1. PEN: CDC Guidance for DevelopingPlans and Reports2011 TB ETN/PEN Conference Awal Khan, PhD Lead, Program Evaluation Team Field Services and Evaluation Branch (FSEB) Division of Tuberculosis Elimination (DTBE) OID/NCHHSTP

2. This talk – a quick overview: • Program evaluation (PE) activities • DTBE guidance on PE • DTBE guidance on cohort review • Developing PE plan & reporting results • Current and on-going PE initiatives • Questions

3. Program Evaluation (PE) Activities

4. Program Evaluation (PE) Plans Submitted- 2010 Annual Report

5. PE Reports Following Application Guidance- 2010 Annual Report

6. Program Using NTIP to Select Evaluation Focus Area, 2010

7. Focus Area of Program Evaluation

8. CDC/DTBE Guidance on Program Evaluation

9. CDC Funding Opportunity Announcement (FOA) Requirements, 2012 • Monitor their progress toward meeting national TB objectives and performance target using NTIP reports and other data sources • When applicable, use NTIP to report their performance on national objectives in the annual and interim progress reports for the TB cooperative agreement • Where appropriate, grantees should continue to report and describe progress toward project-specific objectives and activities using their own data and/or NTIP.

10. Guidance for Program Evaluation- CDC 2012 FOA • A systematic and in-depth study of priority program areas to enhance a program’s ability to meet its objectives and performance targets • The focus of evaluation should be on where the program needs improvement • Collect information that can be used to inform, enhance, and facilitate programmatic decision making process.

11. Guidance for Program Evaluation • Programs should monitor their progress toward meeting national TB objectives and performance target using NTIP reports and other data sources • Identify a focus area and provide a justification • Clearly defined objectives, purposes, and methods of the evaluation • Uses and dissemination of findings including strategy and plan to work with local programs.

12. Program Evaluation Focal Points • Serve as the point of contact in their respective jurisdiction for evaluation activities • Build program evaluation capacity • Work closely with TB program staff • Engage in TB PEN activities • Share PE experiences and lessons learned with stakeholders and other programs.

13. CDC/DTBE Guidance on Cohort Review

14. Programs Implementing Cohort Review

15. Guidance for Conducting Cohort Review • Programs will be required to conduct at least onecohort review per year at the state or local levels or both • Programs should continue ongoing case reviews as usual • Programs are recommended to conduct cohort reviews • Semi-annually, if 0-50 cases reported • Quarterly, if over 50 cases reported

16. Developing PE Plan & Reporting Results

17. Format for Developing Evaluation Plan • Purpose of the evaluation: • 3 purposes/ goals including focus area • Objectives: • What do you want to do? should be “SMART”: Specific, Measurable, Achievable, Realistic, and Time-bound • Activities: • An objective may have multiple activities

18. Format for Developing Evaluation Plan – Contd. • D. Measures of activities: • Specific indicators for each activity • Criteria for success/failure in each activity • Expected outcomes / impact • Potential challenges and opportunities • Use of results.

19. Program Evaluation Report • I. Introduction: Describe Evaluation Objectives • Describe the purpose of your evaluation, rationale for selecting evaluation focus area, and clearly define the evaluation objectives • II. Describe data collection and analyses • how the evaluation was conducted (activities and timeline) • data sources, data collected, and methods

20. Program Evaluation Report – Contd. • III. Conclusion and Discussion of results. • Provide results of the objectives. • If not achieved, why not?  • What were the barriers to and facilitators? • IV. Recommendations to Program Administrators for Future Program Focus and Evaluation Efforts • Describe the best and/or evidence-based practices including identifying the next focus area.

21. PET’s Feedback Process for Interim/Annual Reports • PET provides feedback reports only for program evaluation • Timeline – usually May and October • Feedback is reviewed by program consultant and evaluation representative • TB Control Officer and PE Focal Point receive the feedback • Program are encouraged to contact the program consultant and PE representative to discuss the feedback.

22. Feedback Format

23. Conceptual model for Evaluation Focus Area “Implementation Gap” The “Implementation Gap” Use of new Tools or Technology for evaluation activities The “Implementation Gap” Some goals and objectives are achieved and implemented and some or not TB Elimination

24. Current and on-going PE initiatives

25. Current and Ongoing PE Initiatives • Identify PE technical assistance needs collaboratively with TB control programs, PE focal points, and PET representatives • Program develops PE plan and discusses with designated PET representative for additional support and shares with the respective program consultant for collaboratively implementing the PE activities • Field Operations and Program Evaluation Teams meet quarterly to share evaluation & programmatic perspectives for meeting the grantee’s needs

26. Current and Ongoing PE Initiatives – Contd. • Identify ways to provide technical assistance for sustaining PE efforts • Developed template for contact investigation evaluation plan and report in response to requests from the program evaluation focal point • Plan to develop evaluation plan and report templates for all national TB objectives.

27. Number of Abstracts Received, 2009-2011

28. Program Evaluation Focal Points • Serve as the point of contact in their respective jurisdiction for evaluation activities • Build program evaluation capacity • Work closely with TB program staff • Engage in TB PEN activities • Share PE experiences and lessons learned with stakeholders and other programs.

29. Al Forbes Victor Balaban Gail Grant Judy Gibson Regina Gore Lakshmy Menon Bruce Health Brandy Peterson Andy Heetderks Sylvia Trigoso (PHPS Fellow) Mark Miner Christina Dahlstrom (PHPS) Vic Tomlinson Lauren Polansky Awal Khan Paul Tribble Greg Andrews Dawn Tuckey Dan Ruggiero Joe Scavotto Terry Chorba TB PEN Focal Points and TB Control Officers Acknowledgements

30. Thank you

31. Good Program Evaluation Shifts Our Focus From Did our efforts work? to Are our efforts working?

32. Additional Slides on Cohort Review

33. What is Cohort Review ? • Cohort review is a systematic review of patients with TB disease and their contacts • The “Cohort” is a group of TB cases counted over a specific time and the review occurs after the cases are counted • Case reviews are not cohort reviews.

34. Purposes of Cohort Review • A useful tool for ensuring accountability, informing and education program staff about TB protocol and national objectives • Review of comprehensive TB case management procedures for all cases • Opportunity for reviewing TB data for reliability and quality checking, and identifying deviations • Opportunity for reviewing completeness of key data variables of RVCT and making corrections.

35. Purposes of Cohort Review – Contd. • Help program staff • Understand the challenges patients face as they try to complete TB treatment • Undertake activities required for program improvement • Enable programs to verify data for national program objectives and understand activities most likely to impact program targets.