BSS From Surveillance to Evaluation: advances and new uses

1. BSS From Surveillance to Evaluation: advances and new uses Carl Kendall Tulane University UTAP

2. Presented at the joint meetings of the World Federation of Public Health Associations and the Associação Brasileira de Pós-Graduação em Saúde Coletiva/Abrasco, Tuesday, August 22, 2006 Rio de Janeiro Tulane University UTAP

3. Evaluating the impact of the global effort to control AIDS • Elizabeth Pisani et al. (BMJ 326:1384-7,2003) have argued that the reason we aren’t doing well combating the global epidemic is that we don’t have better knowledge of incidence in different subpopulations; therefore the solution is better surveillance and modeling Tulane University UTAP

4. What is the state of surveillance today? • HIV prevalence data most common data collected • But as Ties Boerma argues, no gold standard for this data, which is of highly variable quality (Lancet,362:1929-1931,2006) • T Diaz et al. argue for the need for enhanced surveillance: • HIV incidence • HIV drug resistance • Deaths due to AIDS • Integrated analysis of data • Behavioral surveillance • Use of data for action (T Diaz, et al. AIDS 2005) Tulane University UTAP

5. Behavioral Surveillance • 2nd Generation Behavioral surveillance is critical: • Where epidemic potential is highest • Where risk is occurring • Where “combined programs” are reducing risk Tulane University UTAP

6. What is second generation behavioral surveillance? • In the early 1990’s surveillance was in its early stages and focused on unlinked anonymous blood testing among sentinel groups. This is referred to as first generation surveillance. • It was soon found that: • Measuring the prevalence of HIV does not provide all the information needed to designing effective policy and programs. e.g.Without understanding HIV risk behaviors we do not know the potential for HIV to spread further • Without data on behaviors HIV prevalence data was difficult to interpret • Second generation surveillance was developed to improve surveillance systems. Tulane University UTAP

7. Goals of Second Generation Surveillance • Monitors trends in behaviors in addition to HIV (early warning) and understand the behaviors that are driving the epidemic. • Increased focus on sub-populations at highest risk of infection. • Better use of surveillance data to plan prevention and care interventions (including integrating behavioral and biological data). Tulane University UTAP

8. BSS and impact monitoring • Continuous 2nd Generation BSS can: • Provide levels of risk behavior • Track seroprevalence • Collect some information about exposure to program • And BSS is proposed as the primary tool for impact monitoring=impact evaluation Tulane University UTAP

9. Realistic Expectations for M&E Monitoring and Evaluation Pipeline All Most Some Few * # of Projects Impact Monitoring/Evaluation Outcome Monitoring/ Evaluation Process Evaluation Input/Output Monitoring * Supplemented with impact indicators from surveillance data. Levels of Monitoring & Evaluation Effort Slide courtesy of Dr. Deborah Rugg, UNAIDS Adaptation of Rehle/Rugg M&E Pipeline Model, FHI 2001 Tulane University UTAP

10. BSS and impact monitoring • Why no evaluation? • Large, complex, multi-intervention, multi-agency national programs • Traditional impact evaluation questionable: • Program uses proven treatment protocols • Design issues: e.g. attribution • Expense • Ethical concerns Tulane University UTAP

11. BSS and impact monitoring • BSS data can be modeled: • EPP and Spectrum (WHO/UNAIDS) • abcDIM (UNPOP) • ASSA2002 (South African) • AEM (Asian Epidemic model) • HIVMM (HIV and TB) • SPEHS (Pop. Dynamics) • Populate (HIV and fertility) • Baggaley (timing of introduction of therapy) • GOALS (allocation of resources) Tulane University UTAP

12. BSS and impact monitoring • But, BSS models’ primary focus is: • Projections and estimates for epidemic • No single model fulfills all requirements • 5 of 11 can utilize some estimate of program effectiveness, but this is only estimated effect Tulane University UTAP

13. Limitations of Behavioral Surveillance • Many surveys are one-off affairs, no system • BSS may be conducted by contractors • Linking test results and behaviors still problematic • Substantial proportion of reduction in prevalence may be due to mortality • Populations needing surveillance may not be included • Half of all Asian countries with BSS system do not include MSM and IDU Tulane University UTAP

14. Limitations of BSS • Sampling: • Convenience samples often used to record high risk behaviors • New methods are available: • RDS • Recent demonstration in 8 sites in Brazil • Available methods can produce probability samples if conducted correctly • TLS • Cluster sampling Tulane University UTAP

15. Experience in Brazil • Since 2004 the PN has supported an experiment in BSS • 8 sites RDS (Fortaleza, Pernambuco, Sao Paolo, Porto Alegre, Santos, Campinas, Curitiba and Manaus) • 2 sites TLS (Pernambuco, Porto Alegre) Tulane University UTAP

16. PN supported BSS sites in Brazil Tulane University UTAP

17. Evaluating the global AIDS epidemic • But what if the reasons that the global program is failing is that we fail to evaluate program interventions? • We need rigorous outcome evaluations of intervention as well as surveillance data Tulane University UTAP

18. Evaluating the global AIDS epidemic • “To be meaningful, this analysis must include issues of prevention coverage and effectiveness…” (Diaz et al. s5) • But no such system exists Tulane University UTAP

19. Outcome evaluations • Are interventions working (e.g.): • Stigma and discrimination • Sexual violence and exploitation (incl. ovc) • Sex worker interventions • Drugs and risky sex • Risk population mixing • Condom use • Partner reduction • Abstinence • Harm reduction • STI • Not just ABC – can’t be decided politically Tulane University UTAP

20. Outcome evaluations • Evaluations of programs • Evaluation of synergistic effects • Understanding community response Tulane University UTAP

21. Characteristics of outcome evaluations • Rigorous designs to measure effects • But also rigorous application of evaluation models • Relate design to decisions (Habicht, Victora and Vaughn, IJE 1999;28:10-18) • Use evaluation theories Tulane University UTAP

22. Rigorous Study Designs • Experimental • Quasi-experimental • Ex post facto/ Non experimental • Qualitative • New methods derived from econometrics • Propensity scoring Tulane University UTAP

23. Rigorous evaluation designs • Example: steps in Utilization-focused Evaluation: • Stakeholder analysis • Conceptualize outcomes • Design and implement study • Analyze findings and involve stakeholders • Make decisions and write reports • Evaluate the outcome-based management system Tulane University UTAP

24. Stakeholder analysis • Identify key actors and leaders • Establish leadership group • Commit to an outcome oriented management system • Agree on intended use • Map out users and uses, set priorities Tulane University UTAP

25. Conceptualize outcomes • Select indicators • Set targets (e.g. coverage, effectiveness) • Establish work team • Establish and involve advisory group • Engage line staff/workshops • Finalize outcomes Tulane University UTAP

26. Design and implement study • Develop design, including analysis and dissemination plan • Pretest methods, instruments, including review of available data • Train staff and implement data collection • Collect data • Monitor and supervise data collection Tulane University UTAP

27. Analyze findings and involve stakeholders • Prepare team: • Review, validate management uses, potential actions, such as decision options • Conduct training with managers on data use • Analyze results to compare with baseline/ targets/other sources • Identify additional information for interpretation • Involve stakeholders in processing the information • With stakeholders, judge performance Tulane University UTAP

28. Make decisions and write report • With stakeholders, make management decisions • Identify audiences, make links between internal and external use • Revise and implement dissemination plan • Prepare versions of report for audiences Tulane University UTAP

29. Evaluate outcome-based management system • Assemble review team to explore: • Process • Use of information to make management decisions, including policy • Intended vs. actual use • Expectations for evaluation system • Criteria for success of the system • Make recommendations Tulane University UTAP

30. Summary • Not enough to have surveillance data to understand success or failure of programs • Rigorous outcome evaluations of program effects are required • Evaluation models are available that can be combined with conventional research designs to answer questions at the global, national, regional and local levels Tulane University UTAP

31. Challenges • Contextual/structural factors and intervention synergies appear to influence outcome • A new hermeneutic of program and context/structure • Need to explore new methods/theories to capture these factors Tulane University UTAP

32. Recommendations • Surveillance needs to continue to improve • It is not enough for the community to argue that impact monitoring is sufficient • A major effort – on the scale of surveillance - needs to be directed to evaluation: • Reviewing estimates of program effectiveness • Developing tools and training programs • Developing new methods to open the black box of intervention Tulane University UTAP