OTC-TFM Monograph: Statistical Issues of Study Design and Analyses

1. OTC-TFM Monograph: Statistical Issues of Study Design and Analyses Thamban Valappil, Ph.D.Mathematical StatisticianOPSS/OB/DBIII Nonprescription Drugs AC Meeting March 23, 2005

2. Outline • Introduction • Summary of Statistical Issues • Current TFM trial Design and Analyses with surrogate endpoints • Statistical Issues of Study Design and Analyses • Options for Trial Design and Efficacy Criteria using surrogate endpoints

3. Introduction • Previous presentations on issues involved in validating surrogate endpoints. • In the absence of clinical trials data, FDA still needs to address current products under review. • This talk discusses issues related to analysis of data obtained on surrogate endpoints. • Does not address clinical relevance of statistical findings or differences in analysis of data based on surrogate endpoints.

4. Summary of Statistical Issues • Primary Endpoint of Log reduction in Bacterial Counts from baseline. • Data Analyses & Variability Issues • Binary Outcomes • Log Reduction (Mean vs. Median) outcomes • Variability in methodology • Study Design and Controls • Active • Vehicle

5. Current TFM Recommendations

6. Current TFM Recommendations: Issues TFM Recommends: • Randomized • Blinded (to persons determining counts only) • Use of Active Control • Use of Vehicle or Placebo Control (role not clearly specified) However, in the current TFM • A non-comparative study design is used in which the test product is not directly compared to the Active Control. • Mean log reduction meeting the threshold log reduction has been used to demonstrate efficacy.

7. Current TFM Recommendations: Issues • Although vehicle and placebo controls are mentioned in the TFM , majority of the NDAs only have test product and active control arms. • Active controls have only been used for internal validation of study methods. • Efficacy assessment does not include a direct comparison of Test product performance to Active control, vehicle or placebo.

8. Statistical Issues of Study Design and Analyses • Primary Endpoint: Log Reduction • Mean Log Reduction can be influenced by few extreme observations. ( Suggestion: Median log reduction may be another option. Median Log Reduction is less sensitive to few subjects with extreme log reductions or outliers. )

9. Statistical Issues of Study Design and Analyses • The efficacy criteria in the current TFM are based on point estimates and do not include confidence intervals to evaluate variability. • Consequently, a few extreme observations can potentially drive the efficacy results.

10. Example: Log Reduction: Mean = 2.0, Median = 1.7 Below 2-log: 78%

11. Example: Log Reduction in Bacterial Counts Active Control Threshold Test . Vehicle

12. Upper Limit Mean Lower Limit Threshold Active Control Test . Vehicle

13. Upper Limit Mean Lower Limit Vehicle Threshold Test Active Control .

14. Primary Endpoint Using Binary Response Subject will be classified as a ‘success’ or a ‘failure’ based on meeting the threshold reduction. Advantages: • Outcome centered on number of subjects and not on organisms may be more clinically relevant. • Effect of Variability is reduced. Disadvantage: This method does not differentiate the magnitude of log reductions among those who meet the criteria for “success”.

15. Example:

16. Study #1 (Surgical Hand Scrub) Treatment Day Mean Median Success (%) Test 1 (1 log) 2.63 2.95 20/21 (95%) 2 (2 log) 3.25 3.17 21/21 (100%) 5 (3 log) 3.51 3.88 13/20 (65%) Active #1 1 (1 log) 1.51 1.69 9/12 (75%) 2 (2 log) 2.37 2.36 8/12 (67%) 5 (3 log) 3.47 3.59 9/12 (75%) Active #2 1 (1 log) 1.17 1.40 14/22 (64%) 2 (2 log) 2.02 1.80 10/22 (45%) 5 (3 log) 1.66 1.58 2/21 (10%)

17. Sample Size Issues • In current TFM, sample size is estimated based on allowing a test product to be as much as 20% worse than active control in the mean log reductions. However, the basis for 20% margin is not clearly stated. • Majority of the current submissions do not follow the recommended sample size as specified in the TFM and only use a sample size of ~30 subjects per treatment arm.

18. Options for Trial Design and Efficacy criteria • Issue 1 • How to analyze the data obtained on the surrogate endpoint of log reductions in bacteria? • Issue 2 • How to take into account the variability in the data collected, when measuring effect of the product? • Issue 3 • How to take into account the variability in the test methodology?

19. Options for Trial Design and Efficacy criteria • Issue 1: How to analyze the data obtained on the surrogate endpoint of log reductions in bacteria? • Mean log reduction • It can be influenced by few extreme observations. • Median log reduction • Less sensitive to outliers or extreme observations. • Percentage of subjects who meet log reduction criteria • Outcome centered on number of subjects who meet the threshold and may provide incentive to study “conditions of use” that provides highest success rates.

20. Options for Trial Design and Efficacy criteria • Issue 2: How to take into account the variability in the data collected? • Examine confidence intervals around the “outcomes” as defined on previous slide with a threshold for lower bound of confidence interval • PRO: Improvement over examination of point estimates alone. • CON: Does not take into account the variability in test method. • Examine confidence intervals around treatment difference between test product and some control • PRO: Allows for examination of variability in methodology across treatment arms. • CON: May require larger sample size for products with lower success rates.

21. Options for Trial Design and Efficacy criteria • Issue 3: How to take into account variability in the test methodology? • “Equivalence”/non-inferiority showing test product is no worse than active control by some margin • PRO: Allows comparison with an active treatment to rule out loss of effect relative to active control • CON: Lack of constancy of effect of active control in previous studies, possible overlap of effect of active and test product with vehicle, hence no basis to select a non-inferiority margin • Superiority of test product to vehicle AND superiority of active control to vehicle • PRO: Given lack of constancy of effect with both active and vehicle controls, allows internal validity of comparisons • CON: May require larger sample size than current TFM standards (how much larger depends on product efficacy over vehicle)

22. Controlling Variability in Test Methodology ? Vehicle Active Control Test Product S S S = Superiority

23. Sample Size: Superiority Test

24. Thank you!