1 / 61

THE RELIABILITY AND UNRELIAB ILITY OF SUB-GROUP ANALYSES

THE RELIABILITY AND UNRELIAB ILITY OF SUB-GROUP ANALYSES JEFFREY L. PROBSTFIELD, MD, FACP, FACC, FAHA, FESC, FSCT Professor of Medicine (Cardiology) University of Washington Research grants- Abbott, Boehringer Ingelheim, King, Sanofi-Aventis Pharmaceuticals, NHLBI, NCI;

ull
Download Presentation

THE RELIABILITY AND UNRELIAB ILITY OF SUB-GROUP ANALYSES

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. THE RELIABILITY AND UNRELIAB ILITY OF SUB-GROUP ANALYSES JEFFREY L. PROBSTFIELD, MD, FACP, FACC, FAHA, FESC, FSCT Professor of Medicine (Cardiology) University of Washington Research grants- Abbott, Boehringer Ingelheim, King, Sanofi-Aventis Pharmaceuticals, NHLBI, NCI; Consultantship-King Pharmaceuticals; no stocks, options or BOD positions

  2. DILEMMA “The response of the average patient to therapy is not necessarily the response of the patient being treated.” Bernard, 1865

  3. SUBGROUP ANALYSES DRIVEN BY: “Should all patients be given XYZ before, during, or after ABC or can/should treatment be limited to a select group?”

  4. SUBGROUPS- PETO • Only one thing is worse than doing subgroup analyses---believing the results

  5. PETO: HOW TO SPOIL A GOOD TRIAL RESULT • Undertake many data-dependent subgroup analyses. • Find some subgroups where treatment has no significant effect (or even, perhaps, no apparent effect whatsoever). • Publish the findings in such a way that many readers believe them.

  6. CARE RESULTS - NO CHD RISK REDUCTION BELOW LDL-C 125 mg/dL “…Although our finding cannot be considered definitive and requires confirmation, it suggests that an LDL cholesterol level of 125 mg per deciliter may be an approximate lower boundary for a clinically important influence of the LDL cholesterol level on coronary heart disease…” NEJM 1996;335:1001-1009

  7. CHOLESTEROL LEVELS AND CHD RISK REDUCTION Cholesterol and Recurrent Events (CARE) 4159 Participants Participants Plasma With Events % Risk LDL-C Placebo Pravastatin Reduction  137 269 210 23 (8 to 36) > 137 280 220 24 (10 to 36) NEJM 1996;335:1001-1009

  8. CHOLESTEROL LEVELS AND CHD RISK REDUCTION Cholesterol and Recurrent Events (CARE) 4159 Participants Participants Plasma with Event % Risk LDL-C Placebo Pravastatin Reduction  125 93 89 -3 (-38 to 23) 125-150 311 239 26(13 to 38) > 150-175 145 102 35(17 to 50) NEJM 1996;335:1001-1009

  9. CARE SUBGROUP ANALYSIS RISK REDUCTION BELOW BASELINE LDL-C mg/dL Concerns about divisions described Plasma Participants %CHD Risk LDL-C N Reduction 95% CI > x(20%) 850 N/A N/A > 150mg/dL 953 35 17 to 50 125 - 150 2355 26 13 to 38 < 137.5 2090 23 8 to 36 < 130 1386 15 N/A < 127 1034 10 N/A < 125(20%) 851 -3 -38 to 23

  10. HPS LDL-C N RRR CHD <3 (<116) 6793 33% >3<3.5 5063 25% >3.5 8680 42% LDL-C N Δ LDL-C E RRR CHD <100 3421 -35 69 22% 100-130 7068 -37 86 28% >130 9927 -39 104 24% No interaction with Vitamin Cocktail

  11. PROPER SUBGROUP “A common set of baseline parameters”

  12. IMPROPER SUBGROUP: “Characterized by a variable measured after randomization”

  13. Hypokalemia Associated With Diuretic Use and CV Events in SHEP Franse, et al. Hypertension. 2000;35:1025-1030

  14. 0.15 Y1 K+ = < 3.5 Y1 K+ = 3.5-5.4 0.12 Y1 K+ = > 5.4 Cumulative CHD Event Rate 0.09 0.06 0.03 0.00 0 1 2 3 4 5 Years to CHD CHD Event Rate by Year 1 K+ Strata HR 95% CI Hyper/Normo-K+ 1.28 0.69, 2.40 Hypo/Normo-K+1.03 0.82, 1.30

  15. EXAMPLES OF IMPROPER SUBGROUPS: 1. Responders vs. nonresponders 2. Adherers vs Non-adherers

  16. FIVE-YEAR MORTALITY ACCORDING TO BASE-LINE CHOLESTEROL AND CHANGE FROM BASE-LINE, ADJUSTED FOR 40 BASE-LINE CHARACTERISTICS

  17. FIVE-YEAR MORTALITY: PATIENTS GIVEN CLOFIBRATE OR PLACEBO, ACCORDING TO CUMULATIVE ADHERENCE TO PROTOCOL PRESCRIPTION

  18. STRUCTURED HYPOTHESES • Carefully state hypothesis • Allow analyses to capture the effect

  19. INTERACTION (Differential Subgroup Effect) “A treatment effect that differs by subgroup.”

  20. QUANTATIVE INTERACTION Different amount (quantity) of benefit in various subgroups.

  21. QUALITATIVE INTERACTION True Benefit in some subgroups and True Harm in others (1 of over 700)

  22. QUALITATIVE DIFFERENCES -WHY NOT? • Extremes excluded • Lack of replication in other studies

  23. BIASES AND ERRORS IN DETERMINING SUBGROUP EFFECTS

  24. FURBERG AND BYINGTONCIRCULATION 1983;67:I98-I101 • 146 Subgroups in BHAT: Few defined a priori • Distribution of subgroup results - Gaussian • Impact of change on data set inversely related to sample size. (Participants or deaths gives similar distribution)

  25. 3 CRITERIA FOR CONFIDENCE IN SUBGROUP FINDING • Dose response relationship • Independent findings within the study • Replication by outside trial

  26. EXPECTED EFFECTS OF TRIAL SIZE ON TRIAL RESULTS

  27. Actual effects of trial size on trial results. Relationship between the total number of deaths in the two treatment groups and the result actually attained, in the 24 trials of a treatment (long-term beta-blockade) that reduces the odds of death by about 22 + 4% No. of trials resulting in:

  28. SUBGROUP EFFECT Treatment effect in a specific proper subgroup. Must be significantly different from overall effect!!

  29. HYPOTHETICAL SUBGROUP EFFECTS ILLUSTRATING THE “PLAY OF CHANCE” IN A TRIAL THAT SHOWS CLEAR OVERALL BENEFIT

  30. MULTIPLE COMPARISONS Example: 1,000 participants Mortality Rate = 10% Treat A Treat B Treatments equally effective 10 subgroups (equal size) randomly formed Relative Risk Probability Reduction to: (percent) .5 99 .33 80 .1 5 Nominal p value - inappropriate Conservative approach - p/Sn Especially important in trial where main outcome is not statistically significant

  31. Subgroup Results Mortality and Morbidity Subgroup % pts Overall 100 Beta blocker (yes) 35 no beta blocker 65 ACEI (yes) 93 no ACEI* 7 .87 .97 .77 RR of death P no ACEI  41% <0.05* ACEI + Beta blocker 42% 0.009 44.0% , P=.0002 *n=366 1.0 1.25 0.50 0.75 Valsartan worse Valsartan better Cohn. N Engl J Med. 2001; *FDA analysis/package insert

  32. CV Death or Hospitalization for CHF Test for interaction Candesartanevent/n Placeboevent/n Diabetes No 680/2715 815/2721 Yes 470/1088 495/1075 Hyper- No 484/1710 579/1703tension Yes 666/2093 731/2093 ACEIs No 586/2230 688/2244 Yes 564/1573 622/1552 Beta No 611/1701 710/1695blocker Yes 539/2102 600/2101 Spirono- No 880/3160 1041/3167lactone Yes 270/643 269/629 Overall 1150/3803 1310/3796 P=0.09 P=0.17 P=0.51 P=0.32 P=0.19 0.6 0.8 1.0 1.2 1.4 candesartan better Hazard ratio placebo better

  33. EXAMPLE OF “SUBGROUPING” IN INTERNATIONAL SOCIETY FOR THE INVESTIGATION OF STRESS-2: ASTROLOGY AND ASPIRIN

  34. ORDERED SUBGROUPS • Strong biological rationale • Reflects natural ordering • Correct for multiplicity • Only indicate those as significant which have a p value less than p/Sn

  35. Reduction of Stroke According to Sex Stroke Rates

  36. GISSI-1 • Overall result - streptokinase treatment, 20% reduction in total mortality • Benefit confined to: • Anterior MI • Age  65 years • Treatment  6 hours • Subsequent trials and pooled results do not confirm

  37. HYPOTHETICAL EXAMPLE OF ORDERED SUBGROUPS: RELATIVE RISK AS A FUNCTION OF TIME OF THROMBOLYTIC THERAPY

  38. SUBGROUPS DEFINED A-PRIORI Suggestive differential subgroup effect State in design of new trial Publish (multiplicity, design analysis, plan over-sampling) Test within an existing data set

  39. STROKE SUBGROUP HYPOTHESIS On BP Meds at ICVOff BP Meds at ICV 35% of participants 65% of participants Net reduction in Net reduction in stroke rate =10% stroke rate =40% 80% power to detect 30% treatment difference

  40. STROKE EVENTS BY MEDICATION STATUS GROUP N NFS FS CS Not on Medications Active1584 64 5 67 Placebo 1577 88 11 96 א2 = 6.72 Relative Risk (active/placebo)= 0.69 P = .0096 95% CI = 0.51-0.95 On Medications Active781 32 5 36 Placebo 794 61 3 63 א2 = 5.11 Relative Risk (active/placebo)= 0.57 P = .0237 95% CI = 0.38-0.85

  41. MRFIT RESEARCH GROUP, AM J CARDIOL 1985;55:1-15ECG ABNORMALITIES AT BASELINE

  42. SUBGROUP HYPOTHESIS Will the treatment of ISH reduce the frequency of major coronary events more in those free of baseline ECG abnormalities than in those with such abnormalities?

  43. OTHER COMBINED EVENTS BY TREATMENT GROUP

  44. NONFATAL MI & CHD DEATH BY BASELINE ECG ABNORMALITIES Treatment Number Events Rate per 100 (SE) With baseline ECG abnormalities Active 1429 67 6.0 (0.8) Placebo 1426 96 8.0 (0.9) א2 = 5.73 Rel. Risk = 0.69 P = 0.02 95% CI = 0.50-0.94 Without baseline ECG abnormalities Active 903 35 4.5 (0.8) Placebo 922 43 4.6 (0.7) א2 = 0.70 Rel. Risk = 0.83 P = 0.40 95% CI = 0.53-1.29

  45. SUDDEN DEATH BY BASELINE ECG ABNORMALITIES Treatment Number Events Rate per 100 (SE) With baseline ECG abnormalities Active 1429 15 1.2 (0.3) Placebo 1426 17 1.3 (0.4) א2 = 0.14 Rel. Risk = 0.88 P = 0.71 95% CI = 0.44-1.75 Without baseline ECG abnormalities Active 903 8 1.2 (0.4) Placebo 922 5 0.6 (0.3) א2 = 0.77 Rel. Risk = 1.64 P = 0.38 95% CI = 0.54-5.01

  46. SUBGROUPS DEFINED A-POSTERIORI • “Grist” for formulating hypothesis • Watch for alternative definitions! • Should be clearly stated and reported as an estimate of effect with appropriate confidence interval

  47. SUBGROUPS AND MONITORING TRIALS • Use statistically sound monitoring method • Interference with main trial endpoint - rare • Formulate hypothesis and test prospectively • Terminate subgroup • “Mega trials” - special problems

More Related