Guidance on Evaluation of Second Generation Surveillance Systems

1. Guidance on Evaluation of Second Generation Surveillance Systems Virginia Loo & Tobi Saidel 2nd Global HIV/AIDS Surveillance Meeting 4 March 2009

2. Evaluation of an SGS system – What is in the scope of SGS?

3. Why evaluation? • To ensure the SGS system is effective • Addresses the key objectives of HIV surveillance • Following the updated guidance on SGS • To make optimal use of limited resources • Financial • Managerial • Technical • Political will

4. What do we mean by evaluation • To use an objective set of criteria for assessing success or progress • Using a logical framework • Defining impact, outcomes, outputs, and input indicators • To apply a systematic approach for measuring the key indicators • To document the findings in a way to facilitate taking action to strengthen the system • Evaluation in terms of testing the validity of new methods and approaches, i.e. operations research is not included in this guidance.

5. Resources for guidance on evaluation of surveillance systems • Current guidance for evaluation: • Guidelines for Evaluating Surveillance Systems. CDC. MMWR 1998. w/ training curriculum tailored for HIV surveillance. • Review articles • Quality of sero-surveillance in low & middle income countries…Lyerla, Gouws, Calleja. STIs 2008 • Case examples: • Reports/Review documents • Presentations: to be shared

6. Logical Framework for Evaluating SGS systems • Impact: • Does HIV surveillance contribute to preventing infections? • Outcomes: • Can the system describe where new infections are coming from? • Can the system describe how the epidemic is changing? • Is the data being used to inform programming?

7. Structure of New Guidelines for Evaluating SGS – Outcome Level • Ex: System describes where the most new infections are coming from? and how the epidemic is changing? • Scope – All geographic areas have some form of AHEAD, • Knows the high risk groups that are present; Size of the high risk groups; • If risk groups are present, knows the risk profile and approximate prev of HIV • Site-group selection – • Decisions of where to put sites are based on epidemiologic criteria. • Size of the population • Size of the risk groups • AIDS case reporting data • Data are collected according to prioritization criteria

8. Indonesia example: Availability of size estimation data for IDU

9. Indonesia Example of Setting & Applying Criteria for Site-Group selection

10. Structure of New Guidelines for Evaluating SGS – Level 1(cont) • Ex: Data is being used to inform programming • Documentation – Written summary of the epidemic situation is available, clearly described for lay audiences, and data-driven • Dissemination – Analyzed data are available widely, actively distributed and reviewed with appropriate stakeholders • Ownership – National and local stakeholders “believe” the surveillance conclusions, cite them when appropriate,

11. Level 2: Outputs/Inputs – Data Utility, Quality, Efficiency • For each surveillance activity (mapping, BSS, sentinel surveillance, AIDS case reporting, etc.): Example Outputs: • Are the right data elements being collected? • Is the choice of methodology appropriate for the situation? • Does the data collection process adhere to protocol? • Has the analysis been conducted with appropriate tools/methods? Including weighting, adjustment for design effects? etc. • Are the data collection restricted to essential data? • Is there appropriate process documentation? • Is the activity conducted meeting ethical standards? Example Inputs: • Training • Supervision/Quality assurance procedures are in place • Protocols • Budget allocation

12. Evaluation Process- Options • Desk review • Gather design/protocol documents, process documentation, result reports • Organize data by geographic areas • Field visits • Observation of field activities • Interviews with local implementers and other stakeholders • Consultation • Review findings • Discussion by technical experts and implementers • Recommendations and Action planning In addition, evaluation of how successfully a method was implemented is an integral step for any kind of analysis…

13. Evaluating the application of the method – “Did it work?” • What do we mean when we ask, “Did it work?” • Were the most appropriate methods applied for the surveillance question asked and the local context? • Was the method implemented properly? • Did it work? = Should we throw it out? But rather – • How can we use the results appropriately? • What can we improve about the application next time? • Assess success every time, • not just when, we don’t get the answer we want • and especially when the answer is “complicated” • All the methods for data collection “work” • But they are based on assumptions that must be tested as part of analysis • Are subject to bias if applied “incorrectly”

14. Tim Gunn’s Fashion advice for surveillance types… • Invest in classic pieces, but individualize your look with accessories • Good fashion = good fit + appropriate for the occasion • Look in the mirror before you leave the house

15. Thank you Acknowledgments: Many thanks to the Indonesia NAC and FHI and World Bank offices for use of the data synthesis example.