From Efficacy to Effectiveness: the Safe in the City Model

1. From Efficacy to Effectiveness:the Safe in the City Model 2009 Meeting of the International Society for STD Research June 28 – July 1, 2009 London 1

2. Efficacy vs. Effectiveness • Efficacy: How well an intervention works in optimal settings • Well-trained and paid research staff • Controlled Setting • Carefully recruited study subjects • Carefully constructed and executed research protocol 2

3. Efficacy vs. Effectiveness • Effectiveness: How well an intervention works in the real world • Staff not specifically trained for intervention; and need to meet competing demands • Targets all patients/clients, not selected subjects • Adaptation of intervention protocol often necessary to make it practical for the setting (threat to fidelity) 3

4. Effectiveness Research • Implicitly takes into account the acceptability, feasibility, implementation, and impact of an intervention in the settings for which the intervention was designed 4

5. Effectiveness Research3 Main Components • Participatory research model • Efficacy/Effectiveness evaluation • Scale-up process and evaluation 5

6. Participatory Research Model • Involves all stake holders that determine the ultimate success of the intervention: • Academics / intervention developers • Target population representatives • Representatives from the organizations responsible for implementation 6

7. Efficacy/Effectiveness Evaluation • Evaluation of efficacy/effectiveness should take place in an environment that closely mimics the environment for which the intervention is ultimately intended 7

8. Scale – Up Process and evaluation • Marketing strategies and implementation • Evaluation of intervention up-take by target audience • Short to long term assessment of implementation and sustainability 8

9. 9

10. Safe in the CityProject Investigators Centers for Disease Control and Prevention Lee Warner, Andrew Margolis, Jocelyn Patterson, Craig Borkowf Denver Public Health Cornelis Rietmeijer, John Douglas, Doug Richardson Education Development Center, Inc. Lydia O’Donnell, Athi Myint-U, Carl O’Donnell Long Beach, California State University and Department of Health and Human Services Kevin Malotte, Shelley Vrungos, Nettie DeAugustine San Francisco Department of Public Health Jeffrey Klausner, Gregory Greenwood, Carolyn Hunt 10

11. Safe in the City: Focus on Effectiveness • Collaborative, participatory research process involving all stakeholders • Formative process • Intervention outcome study that involved entire clinic populations • Use of STI markers of effectiveness • Closely-guided and evaluated post-study dissemination phase 11

12. Safe in the CityDevelopment of the Intervention Cornelis A. Rietmeijer, MD, PhD Denver Public Health Department Denver, Colorado, USA 12

13. Study Rationale • 340,000,000 incident STDs worldwide annually • STD clinics provide access to men and women likely to be infected and to acquire new infections over time • Yet behavioral interventions with counseling or multiple sessions are difficult to implement in busy medical settings • Recent interest in simple, easy to use, and low cost interventions for waiting rooms 13

14. Rationale continued • Previous research suggests benefits of video-based approaches, but subject to limitations: • Controlled research settings • Tailored videos • Single site • Inclusion of group counseling • Effectiveness of stand-alone video in ‘real-world’ setting is unknown 14

15. Safe in the City Project Overview • 5-year CDC-funded multi-site study • Develop a brief video-based STD clinic waiting room intervention to reduce (or eliminate) STI and risky sexual behavior • Evaluate effectiveness in 3 publicly funded STD clinics in Denver, San Francisco, and Long Beach, CA. 15

16. Intervention Development Considerations Waiting rooms in medical settings provide an underused opportunity to reach patients who are thinking about their health. Yet to be feasible and sustainable, interventions must: • Be easy and inexpensive to administer • Result in minimal interruption of patient flow • Require few clinic resources, especially staff time • Be acceptable to diverse clients 16

17. Formative Process • Identification of intervention medium, theoretical framework, and key messages by research team • Collaboration with award-winning film maker to integrate framework in an appealing product • Multi-step participatory process involving target audience, clinic staff, and community advisors • Intervention research study in 3 STD clinics 17

18. Intervention Development: Integrated Theoretical Framework Theory of Planned Behavior Information Motivation Behavior Model Social Cognitive Theory Core constructs grouped into interconnected elements → HIV/STD risk, knowledge, perception→ Positive attitudes toward condom use→ Self-efficacy/skills for condom negotiation, acquisition, use→ Modeling of appropriate behaviors 18

19. Focus Groups • 3 sites held 12 focus groups with 176 participants • 3 different stages of video development: • Story line development • Script development • Post-production editing 19

20. What Is the Intervention? • 23-minute video • 3 story lines • 2 cartoon animations • Condom variety and selection • Instructions for use • Posters in waiting and exam rooms 20

21. Things are getting more serious between Paul and Jasmine, but Paul “slips” and has a sexual encounter with Teresa. Teresa gets an STD and tells Paul. Now Paul has to tell Jasmine. Story Line – Paul and Jasmine 21

22. Story Line – Rubén, Tim and Christina Rubén’s girlfriend Christina doesn’t know about his interest in men. Rubén and Tim have a casual sex encounter after meeting in a bar. Days later, Christina suspects something is wrong. She insists on a visit to the STD clinic. 22

23. Story Line – Teresa and Luis Teresa has recently met Luis. After her STD scare with Paul, Teresa is serious about wanting to use condoms. Now she has to convince Luis. 23

24. 24

25. 25

26. Assessing Intervention Effectiveness in an STD Clinic Population:the Safe in the City Model Lee Warner, PhD Centers for Disease Control and Prevention Atlanta, Georgia, USA 26

27. Examples of Challenges in Study Design • Examining “real-world” effectiveness prohibits active patient enrollment *still need to deliver intervention, include entire clinic population • Large sample size required, given effectiveness of brief intervention likely modest • Biologic markers (STI) preferred effectiveness measure to self-reported behavior • Unable to randomize in waiting room setting, but need balance between study conditions 27

28. Maximizing Intervention Delivery and Exposure • Identify environmental characteristics of waiting rooms • Observe waiting room flow • Determine appropriate playback frequency • Identify factors to increase viewership (goal: 80%) • Assess and adjust to clinic staff acceptance of video 28

29. Denver Waiting Room 2nd TV 29

30. Long Beach Waiting Room 30

31. San Francisco Waiting Room 31

32. Video Viewership Viewership as defined by watched most or all of the video + identified a main message 32

33. StudyObjective • To determine whether this brief, structural intervention reduced incident laboratory-confirmed infection among typical visitors to an STD clinic 33

34. Overview of Study Design • Population: = all patients attending 3 STD clinics from December 2003 – August 2005 (N = ~40,000) • Study design: 2 arm non-randomized controlled trial • Arm assignment: alternating 4-week control & intervention periods • Data collection: retrospective review of clinic data & external surveillance records to ascertain new STI diagnoses, conducted under waiver of informed consent 34

35. Study Flow Diagram Patient Presents at Clinic N=38,635 Intervention: 23 Minute Video Shown in Clinic Waiting Room Control: Standard Waiting Room Experience Assignment Based on Clinic Visit Date Clinical Evaluation Clinical Evaluation Review of Medical Records & Surveillance Registry Data for STD outcomes Review of Medical Records & Surveillance Registry Data for STD outcomes 35

36. Characteristics of Clinic Populations, by Condition 36

37. Approach to Statistical Analysis • Outcome: Incident laboratory-confirmed infection • gonorrhea, chlamydia, trichomoniasis, syphilis, and HIV • Analysis: Survival analyses estimating time to infection • Kaplan-Meier analyses comparing survival at specific points in time by condition* • Cox proportional hazards regression comparing hazard ratios by condition* 37

38. First New Laboratory-Confirmed Infection, By Condition 38

39. Overall Effect of Intervention on Laboratory-Confirmed Infection* * = 9% reduction in STI incidence 39

40. Intervention Effect, by Characteristic 40

41. Source of STI Diagnosis, By Condition p = 0.67 41

42. Return Visits to Clinic, By Condition p = 0.86 p = 0.39 42

43. Factors Significantly Associated with Incident STI, by Strength of Effect Measure • MSM • Baseline STI diagnosis • Minority race / ethnicity • Age < 25 yrs Highest Lowest 43

44. Meeting Challenges in Study Design • Evaluation of intervention effectiveness in “real-world” setting with “passive enrollment” of patient population • Efficient examination of laboratory-confirmed STI using existing medical / surveillance records • Inclusion of entire patient population, ensuring generalizability • Non-randomized design with systematic allocation balanced all measured characteristics 44

45. Factors Critical to Success of Evaluation • Ability to examine study population in advance: • Patient flow (e.g., number of patients, waiting room time) • Reaction of patient / staff to video • STI prevalence and incidence of population known • Access to electronic medical records and surveillance registries to identify incident STI • Waiver of informed consent from IRB critical 45

46. Conclusions “Safe in the City” associated with reduction in incident STI *** important -- estimate = effectiveness, not efficacy. This reduction can have significant public health benefit, but requires sufficient availability of intervention and wide adoption by clinics. Relatively* easy-to-implement, low cost interventions can reach large numbers of high-risk populations with minimal effort. Type of design used to examine effectiveness of “Safe in City” promising for studies of STD prevention interventions. 46

47. How to Use a Condom, Christina & Ruben 47

48. 48

49. 49

50. Safe in the CityDiffusion of the Intervention in the U.S. Doug Richardson, MAS Denver Public Health Department Denver, Colorado, USA 50