Gatekeeping Testing Strategies in Clinical Trials

1. Alex Dmitrienko, Ph.D. Eli Lilly and Company FDA/Industry Statistics Workshop September 2004 Gatekeeping Testing Strategies in Clinical Trials

2. Outline • Gatekeeping strategies • account for the hierarchical structure of multiple analyses (comparisons) and preserve the overall false positive rate • Clinical trial examples • registration trials with multiple primary and secondary endpoints (trials in patients with depression and acute lung injury) • dose-finding trials (trial in patients with hypertension)

3. Primary versus secondary findings • Findings with respect to secondary endpoints provide much useful information • useful to prescribing physicians and patients • FDA guidance “Clinical studies section of labeling for prescription drugs and biologics” • “The CLINICAL STUDIES section should present those endpoints that are essential to establishing the effectiveness of the drug (or that show the limitations of effectiveness) and those that provide additional useful and valid information about the activities of the drug”

4. Primary versus secondary findings • Dilemma • regulatory agencies and pharmaceutical companies have long debated what secondary findings should be included in the product label • regulatory agencies are concerned that pharmaceutical companies tend to present favorable data and ignore unfavorable data • Gatekeeper strategies offer one potential solution to the dilemma

5. Types of gatekeeping strategies • Two types of gatekeeping strategies • sequential strategies have been used for 10 years • parallel strategies were introduced by Dmitrienko, Offen and Westfall (2003) • Introduce general gatekeeping framework • based on the closed testing principle (Marcus et al, 1976) • focus on strategies derived using Bonferroni’s test • easily extended to more powerful tests that account for the correlation among the endpoints (Dunnett’s test, resampling tests)

6. Example 1: One primary endpoint • Depression trial • Experimental drug is compared to placebo • Single primary endpoint • 17-item Hamilton depression rating scale (HAMD 17 score) • Trial is declared successful if the drug is superior to placebo • Two important secondary endpoints • response rate based on the HAMD 17 score • remission rate based on the HAMD 17 score • Can the secondary findings be included in the product label?

7. Sequential gatekeeping strategy • Step 1: Perform the primary analysis • Step 2: Perform the secondary analyses with an adjustment for multiplicity if the primary analysis yielded a significant result Family 1 (gatekeeper) Family 2 A1: HAMD17 A2: Response rate A3: Remission rate

8. Sequential gatekeeping strategy • Primary analysis: No adjustment for multiplicitySecondary analyses: Stepwise Holm’s test • All primary and secondary findings are significant at 5% level

9. Example 2: Multiple primary endpoints • Clinical trial in patients with acute lung injury • Experimental drug is compared to placebo • Two primary endpoints • number of days patients are off mechanical ventilation (vent-free days) • 28-day all-cause mortality rate • Trial is declared successful if the drug is superior to placebo with respect to either endpoint • Two important secondary endpoints • number of days patients are out of ICU (ICU-free days) • overall quality of life at the end of the study • Can the secondary findings be included in the product label?

10. Parallel gatekeeping strategy Family 1 (gatekeeper) Family 2 • Step 1: Perform the primary analyses with an adjustment for multiplicity • Step 2: Perform the secondary analyses with an adjustment for multiplicity if at least one primary analysis yielded a significant result A1: Vent-free days A3: ICU-free days A4: Quality of life A2: Mortality

11. Parallel gatekeeping strategy Family 1 (gatekeeper) Family 2 • Step 1: Perform the primary analyses with an adjustment for multiplicity • Step 2: Perform the secondary analyses with an adjustment for multiplicity if at least one primary analysis yielded a significant result A1: Vent-free days A3: ICU-free days A4: Quality of life A2: Mortality

12. Parallel gatekeeping strategy Family 1 (gatekeeper) Family 2 • Step 1: Perform the primary analyses with an adjustment for multiplicity • Step 2: Perform the secondary analyses with an adjustment for multiplicity if at least one primary analysis yielded a significant result A1: Vent-free days A3: ICU-free days A4: Quality of life A2: Mortality

13. Statistical details • Sequential families of null hypotheses • Null hypotheses H11, H12, H21, H22 grouped into two families F1={H11, H12} and F2 ={H21, H22} • Parallel gatekeeping testing procedure • Overall Type I error rate is no greater than  (e.g., 0.05) • The null hypotheses in F2 can be tested if at least one hypothesis in F1 has been rejected [parallel testing] • Adjusted p-values for the hypotheses in F1 do not depend on the p-values associated with the hypotheses in F2 [primary analyses cannot depend on secondary analyses]

14. Closed testing • Closed family of hypotheses • Consider all 15 intersections of the null hypotheses • {H11or H12 or H21or H22}, {H11or H12 or H21}, etc • Specify a test for each intersection hypothesis that controls the Type I error rate, e.g., Bonferroni test • Establish implication relationships • Intersection hypothesis {H11or H12} implies H11 and H12, etc • Tests for original hypotheses • Reject a null hypothesis if all intersection hypotheses implying it have been rejected

15. Decision matrix

16. Parallel gatekeeping strategy • Two scenarios • significant improvement in the mean number of ventilator-free days and 28-day all-cause mortality • significant improvement in 28-day all-cause mortality but not in mean number of ventilator-free day • Weighted analyses • primary endpoints are unequally weighted to reflect their relative importance (likelihood of success) • weight=0.9 for number of ventilator-free days • weight=0.1 for 28-day all-cause mortality

17. Parallel gatekeeping strategy • Scenario 1 • significant improvement in the mean number of ventilator-free days and 28-day all-cause mortality • Primary analyses: Bonferroni’s testSecondary analyses: Stepwise Holm’s test • All primary and secondary findings are significant at 5% level

18. Parallel gatekeeping strategy • Scenario 2 • significant improvement in 28-day all-cause mortality but not in mean number of ventilator-free day • Primary analyses: Bonferroni’s testSecondary analyses: Stepwise Holm’s test • The primary mortality analysis and secondary quality of life analysis are significant at 5% level

19. Example 3: Dose-finding study • Clinical trial in patients with hypertension • Four doses of an experimental drug are compared to placebo • doses are labeled as D1, D2, D3 and D4 • Primary endpoint • reduction in diastolic blood pressure • Objectives of the study • find the doses with a significant reduction in diastolic blood pressure compared to placebo • study the shape of the dose-response curve

20. Example 3: Dose-finding study Family 1 (gatekeeper) Family 2 (gatekeeper) Family 3 • Step 1: Compare doses D3 and D4 to placebo • Step 2: Compare doses D1 and D3 to placebo if at least one comparison at Step 1 is significant • Step 3: Perform various pairwise dose comparisonsif at least one comparison at Step 2 is significant A1: D4 vs. P A3: D2 vs. P Pairwise comparisons A2: D3 vs. P A4: D1 vs. P

21. Example 3: Dose-finding study Family 1 (gatekeeper) Family 2 (gatekeeper) Family 3 • Step 1: Compare doses D3 and D4 to placebo • Step 2: Compare doses D1 and D3 to placebo if at least one comparison at Step 1 is significant • Step 3: Perform various pairwise dose comparisonsif at least one comparison at Step 2 is significant A1: D4 vs. P A3: D2 vs. P Pairwise comparisons A2: D3 vs. P A4: D1 vs. P

22. Example 3: Dose-finding study Family 1 (gatekeeper) Family 2 (gatekeeper) Family 3 • Step 1: Compare doses D3 and D4 to placebo • Step 2: Compare doses D1 and D3 to placebo if at least one comparison at Step 1 is significant • Step 3: Perform various pairwise dose comparisonsif at least one comparison at Step 2 is significant A1: D4 vs. P A3: D2 vs. P Pairwise comparisons A2: D3 vs. P A4: D1 vs. P

23. Example 3: Dose-finding study

24. Parallel gatekeeping strategy • Doses D2, D3 and D4 are significantly different from placebo at 5% level

25. Summary • Gatekeeping strategies can be successfully used in • pivotal trials with multiple primary and secondary endpoints • dose-finding studies • Registration trials • a priori designation of gatekeeping strategy allows additional data useful to physician and patient to be presented in the product label • Dose-finding studies • efficient tests of dose-response relationship

26. Extensions • More powerful gatekeeping tests • based on more powerful tests, e.g., Simes test • based on tests accounting for the correlation among the endpoints (exact parametric tests such as Dunnett’s test and approximate resampling-based Westfall-Young tests) • Software implementation • SAS programs for gatekeeping tests can be found in Dmitrienko, Molenberghs, Chuang-Stein, Offen. (2004). Analysis of Clinical Trials: A Practical Guide. SAS Publishing, Cary, NC.

27. References • Papers • Dmitrienko, Offen, Westfall. (2003). Gatekeeping strategies for clinical trials that do not require all primary effects to be significant. Statistics in Medicine. 22, 2387-2400. • Marcus R, Peritz E, Gabriel KR. (1976). On closed testing procedure with special reference to ordered analysis of variance. Biometrika. 63, 655-660. • Westfall, Krishen. (2001). Optimally weighted, fixed sequence, and gatekeeping multiple testing procedures. Journal of Statistical Planning and Inference. 99, 25-40.