1 / 16

Beyond the checklist: Assessing understanding of HIV vaccine trials in South Africa

Beyond the checklist: Assessing understanding of HIV vaccine trials in South Africa. Graham Lindegger Xolani Xaba Cecilia Milford Catherine Slack Michael Quayle Eftyhia Vardas HIV AIDS Vaccines Ethics Group (HAVEG) www.saavi.org.za/haveg.htm. Outline. Background

brasen
Download Presentation

Beyond the checklist: Assessing understanding of HIV vaccine trials in South Africa

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Beyond the checklist: Assessing understanding of HIV vaccine trials in South Africa Graham Lindegger Xolani Xaba Cecilia Milford Catherine Slack Michael Quayle Eftyhia Vardas HIV AIDS Vaccines Ethics Group (HAVEG) www.saavi.org.za/haveg.htm XVI International AIDS Conference 13-18 August 2006

  2. Outline • Background • Study aim, design and results • Implications XVI International AIDS Conference 13-18 August 2006

  3. Background: Complexities • What should be understood? What level? How? How best to measure? • Limitation of traditional methods: • Focus on memory of technical information • May be less culturally congruent • Why alternate methods? • More personal understanding; less technical focus • More culturally sensitive XVI International AIDS Conference 13-18 August 2006

  4. Aims and process • Primary objective: • To compare four methods of assessing understanding of trial participation (i.e., self-report, checklist; story; case) • A collaborative process was followed: • Series of 8 meetings to develop protocol and obtain site buy-in • Ethical approval; Sponsor negotiation; Pilot XVI International AIDS Conference 13-18 August 2006

  5. What concepts did we test? • Collaborative meetings with stakeholders: • What must TPS understand? Prioritize? • Key components: • Trial aims • Eligibility to participate • Methodology (randomisation, placebo, blinding) • Risk of false sense of security • Risk of false positivity • Compensation for research related injury • Informed consent (right to withdraw) XVI International AIDS Conference 13-18 August 2006

  6. What measures did we develop? • Self reported understanding: How do you rate your understanding of [concept]? [no/little, good enough] • Formal checklist with true-false items for each concept • Narrative (story) describing trial participation • Responses to hypothetical vignettes (case examples) of eachconcept (or part thereof) XVI International AIDS Conference 13-18 August 2006

  7. How did we develop a scoring system? • Developed collaboratively during stakeholder meetings • Asked the question: • What would satisfy you that a participant understood enough of this component to be enrolled in the trial? • Discussion – disagreement – resolved by discussion – consensus • Ended up with detailed operational criteria for scoring responses • Allowed development of a “Gold standard” of understanding for each component XVI International AIDS Conference 13-18 August 2006

  8. Using the scoring system • Two point scale with little/no or good enough understanding used • Self-report: Participants estimated own understanding • Checklist: Correct scores on certain items • Narrative: description of essential information • Vignette: describe part of concept XVI International AIDS Conference 13-18 August 2006

  9. Sample and procedure • Sample: • 53 Vaccine Discussion Group/Information Session Members • 26 male, 33 female • 18-55 years • Procedure: • All assessment methods administered in 1 sitting • Self report first, then varied order XVI International AIDS Conference 13-18 August 2006

  10. Data analysis • 2 independent scorers • Understanding of each trial concept across the different measures compared using Cochran’s Q • Overall understanding of trial participation across measures compared using Friedman’s (expressed as %) • Level of agreement between different components on each measure (Cohen’s Kappa) XVI International AIDS Conference 13-18 August 2006

  11. Results • Significant differences across measures of scores of overall understanding • Overall scores were ranked as follows: • Self-report: 86% • Checklist: 82% • Narrative: 69% • Vignette: 67% • Significantly different proportions of ‘good enough’ scores across all measures XVI International AIDS Conference 13-18 August 2006

  12. Component Component Percentage of participants rated as “Good Enough” per section Percentage of participants rated as “Good Enough” per section Cochran’s Q or Friedman’s Cochran’s Q or Friedman’s Asymp Sig. Asymp Sig. Self Report Self Report Checklist Checklist Vignette Vignette Narrative Narrative Trial aims Trial aims 95% 95% 93% 93% 53% 53% 60% 60% Q = 45.192 Q = 45.192 0000* 0000* Eligibility to participate Eligibility to participate 96% 96% 86% 86% 84% 84% 95% 95% Q = 7.929 Q = 7.929 0048** 0048** Methodological considerations Methodological considerations 72% 72% 83% 83% 67% 67% 67% 67% Q = 7.714 Q = 7.714 0052 0052 Risk (False sense of security) Risk (False sense of security) 88% 88% 67% 67% 72% 72% 76% 76% Q = 8.509 Q = 8.509 0037** 0037** Risk (False positivity) Risk (False positivity) 87% 87% 75% 75% 55% 55% 60% 60% Q = 23.76 Q = 23.76 0000* 0000* Compensation for research related injury Compensation for research related injury 70% 70% 85% 85% 44% 44% 43% 43% Q = 32.353 Q = 32.353 0.000* 0.000* Right to withdraw Right to withdraw 96% 96% 95% 95% 93% 93% 98% 98% Q = 3.333 Q = 3.333 0343 0343 Overall/Total understanding (%) Overall/Total understanding (%) 86% 86% 82% 82% 67% 67% 69% 69% 0.000** 0.000** *significant at  = 0.01; **significant at  = 0.05 *significant at  = 0.01; **significant at  = 0.05 Frequency of “good enough” understanding for each subscale XVI International AIDS Conference 13-18 August 2006

  13. Results: Closed versus open XVI International AIDS Conference 13-18 August 2006

  14. Discussion • Measured level of understanding is a function of the assessment method used • Forced-choice checklists and self-reports may be over-estimating levels of understanding, as measured against “gold standard” developed by staff • Self report – social desirability? • Checklist - acontextual rote memorizing of technical information? • Vignettes and narratives - contextually embedded task? • Possibly closer to actual decision-making for trials XVI International AIDS Conference 13-18 August 2006

  15. Implications • Limits of open ended methods: • more time consuming • require greater training to implement • For critical concepts where consequences of misunderstanding are great – supplement with open-ended measures XVI International AIDS Conference 13-18 August 2006

  16. Acknowledgements and thanks • Trial site staff • Study participants • Sponsors THANK YOU XVI International AIDS Conference 13-18 August 2006

More Related