1 / 18

Kenneth F. Schulz, PhD, MBA

Kenneth F. Schulz, PhD, MBA. Triangle Global Health Consortium Breakfast Discussion. Participants with Infection. Randomize. Placebo. Treatment. 60% Compliance. 40% Non-compliance. 25% Non-compliance. 75% Compliance. Policy of no treatment. Policy of treatment. Outcome Outcome.

fritz-hester
Download Presentation

Kenneth F. Schulz, PhD, MBA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Kenneth F. Schulz, PhD, MBA Triangle Global Health Consortium Breakfast Discussion

  2. Participants with Infection Randomize Placebo Treatment 60% Compliance 40% Non-compliance 25% Non-compliance 75% Compliance Policy of no treatment Policy of treatment OutcomeOutcome

  3. Participants with Infection Randomize Treatment B Treatment A 97% Treatment A 3% Treatment B 33% Treatment A 67% Treatment B Policy of Treatment A Policy of Treatment B OutcomeOutcome

  4. Participants without HIV Infection Randomize 16,395 8,197 Vaccine 8,198 Placebo • 25% Non-compliant • < 4 doses • Outside time period • 15 HIV+ • 22% Non-compliant • < 4 doses • Outside time period • 24 HIV + • 78% Compliant • 50 HIV+ • 75% Compliant • 36 HIV+ Policy of administering placebo Policy of administering vaccine OutcomeOutcome

  5. Per-protocol traditionally used in preventive vaccine trials • Completed immunization series • With the assigned vaccine • Within the allotted window of time • Moreover, follow-up for counting cases of disease typically begins a few weeks after the last dose • Vaccine-induced response is believed “adequate”

  6. Per-protocol similar to ITT in vaccine trials • Levels of full compliance are frequently as high as 90-95% in vaccine trials • Moreover, compliance is easy to assess • Vaccinations given by health care providers • And instances of noncompliance usually appear to be unrelated to treatment • Conjecture . . . cannot be proven to be • Dropping out of a vaccine trial due to toxicity is uncommon • Vaccines extremely benign compared to therapies

  7. Per-protocol similar to ITT in vaccine trials • If few cases of disease occur before the participants are “fully immunized” and the conditions are met from the prior slide • Then, “it may be possible to estimate the biological efficacy of a full vaccination series without a large magnitude of bias.” • “. . . There should be little risk that an ineffective preventive vaccine will be licensed if conclusions are drawn based on a per-protocol analysis.” Horne AD, Lachenbruch PA, Goldenthal KL. FDA

  8. Excluding Post-Randomization, Before Series Completion, Outcomes • Intuitively attractive • However, the same argument could be used to exclude all outcomes, e.g., in a placebo group • At best, if a priori, cannot improve upon randomization • Only can worsen internal validity . . . Additional bias • If a posteriori, certainly biased • Investigators observe results, then conjure up theoretically justifiable rules that favor their hypotheses • Severely biased, although will appear logical in the paper • Post hoc lucidity

  9. Exclusions After Randomization (Per-protocol) • Damage internal validity • Can introduce bias • Avoided in design and conduct • Carefully scrutinized in reports • All randomized patients should be analyzed • and analyzed as part of the group to which they were initially assigned • ITT (intent-to-treat) analysis • Secondary, non-ITT can be performed • Labeled as non-randomized comparisons

  10. Participants without HIV Infection Randomize 2,000 1,000 Placebo 1,000 Vaccine 200 High Risk 40 HIV+ 800 Low Risk 8 HIV+ 200 High Risk 10 HIV+ 800 Low Risk 2 HIV+ Per-protocol Vaccine 190 High Risk 798 Low Risk Per-protocol Placebo 160 High Risk 792 Low Risk OutcomeOutcome

  11. EndSlides after this will only be used to clarify a point if brought up in discussion

  12. RCT Compared the Effectiveness of Clofibrate in Preventing Cardiac Deaths in Men Who Had Survived a Myocardial Infarction • Authors state that: • One can justify almost any conclusion, dependent upon the analysis chosen • Manipulating deviates leads to severe bias • Can you ever do so?

  13. Dear Dr Schulz (email) Thank you for your thoughtful and comprehensive treatment of randomization issues in controlled trials. This email comes with hope that you have a few minutes to respond to a query. I am currently an investigator at the end of a 4 year . . . trial, writing up a paper I would like to submit to JAMA. I prepared the randomization list and concealment according to procedures you recommended in the 1995 article. . . During the course of the study, however, my colleagues and I (naïve about applying an intention-to-treat analytic plan) deviated from the original generation list. When, three quarters of the way through sample recruitment, treatment participants were missing all sessions, we assumed this group could be treated as "controls," as they received no treatment. Thus, I took any new "control" envelopes out of the randomization sequence, leaving only the treatment assignments.

  14. Dear Dr Schulz (email, continued) Soon thereafter, realizing our mistake, but with only 8-10 slots left in enrollment, I put all the controls back in (all of the last assignments were control, in an attempt to re-balance the randomization). Again, my assignment staff were never aware of any deviations in the protocol. The overall effect is that there are approximately 8 fewer controls than there are for the two treatment groups, but we have worked closely with a statistician to take this deviation into account in the analysis. My question is to what extent (in how much detail) would you describe this problem to the JAMA reader, and how egregious a mistake is this, in your opinion? Many thanks in advance for your time and consideration.

  15. Flow Chart: Withdrawals but ITT The Lancet 2004; 364: 772

  16. Exclusions of LFU Damage Internal Validity • Without outcomes from those lost to follow-up, investigators have little choice but to exclude them from the analysis • Any losses damage internal validity • However, differential rates of loss among comparison groups cause major damage • Investigators must minimize their losses to follow-up

  17. Summary: Per-protocol analyses and “Non-Analyzable” Outcomes • Per-protocol anlayses, if specified a priori, cannot improve upon randomization • But, could lead to bias • If instituted a posteriori, biased and unethical • Even though it may sound logical in paper • Critically, readers cannot determine if per-protocol analysis was really instituted before (protocol) or after • Therefore, using just a rule can taint an article

More Related