Bayesian Sample Size Determination in the Real World

1. Bayesian Sample Size Determination in the Real World John Stevens AstraZeneca R&D Charnwood Tony O’Hagan University of Sheffield

2. Reference: • Bayesian Assessment of Sample Size for Clinical Trials of Cost-Effectiveness • O’Hagan A, Stevens JW • Medical Decision Making 2001;21:219-230

3. Contents • The Sample Size Problem • The Utility Function • Study Objectives • The Prior Distribution • An Example • Discussion

4. The (Real World) Sample Size Problem • A study is to be designed to compare the efficacy of two treatments, Treatment 2 (the experimental treatment) and Treatment 1 (the control treatment). • Consideration is to be given to • the gain (i.e. profit) achieved from sales of the new treatment if it is successfully marketed, • the loss (i.e. costs) associated with setting up the study and the cost per patient in the study, allowing for any delay in coming to market. • It will be assumed that all gains and losses are associated with the conduct of a single study. • What is the optimal sample size?

5. The Utility Function • In order to determine sample size using decision theory it is necessary to define a Utility Function. • The Utility Function can be written as : • U(n1, n2, x) = u . L(x) - c . (n1 + n2) -c0 • where, • n1 and n2 are the sample sizes in the two treatment arms, • x denotes the data obtained in the study, • L(x) takes values one if the data, x, are convincing enough to the regulator to approve the drug and zero otherwise, • u is the profit to the company if the drug is approved, • c is the cost per patient in the study (allowing for delays in marketing), • c0 is a set-up cost for the study.

6. Sample Size - The Two Stages • There are two stages in the conduct of a study: • Design stage. Plan the study to maximise the expected utility associated with the desired outcome. • Analysis stage. Analyse the data from the study to see whether we can report the desired outcome. • These two stages are reflected in • setting the objectives, • how we do the sample size calculations.

7. Study Objectives Analysis Objective: • We will regard the outcome of the study as positive if the data obtained are such that there is a probability of at least ω that Treatment 2 is more efficacious than Treatment 1. Design Objective: • We wish to maximise the expected utility, U(n1, n2), associated with the desired outcome.

8. Analysis Objective • Frequentist formulation • We wish to reject the null hypothesis that μ2 - μ1 = 0, at the 100(1 - w)% level of significance. • e.g. w = 0.95 corresponds to usual 5% significance test in a one-sided test • Bayesian formulation • We wish to have at least a 100w% posterior probability that μ2 - μ1 > 0.

9. Design Objective • Bayesian formulation • We want to choose sample sizes that maximise the expected utility associated with achieving the desired posterior probability that μ2 - μ1 > 0.

10. Expected Utility • L(x) is a function of the sample size in each treatment group. • The expected value of L(x) is P(n1, n2), and is the probability of obtaining data to convince the regulator. • The expected Utility is then, • U(n1, n2) = u . P(n1, n2) - c . (n1 + n2).

11. Understanding the Bayesian Formulation • At the Analysis stage: • The Bayesian and frequentist formulations are similar (particularly if we employ weak prior information in the analysis of the data). • The Bayesian statement is often how the p-value is interpreted anyway. • At the Design stage: • The frequentist and Bayesian formulations are different. • The frequentist approach fixes the parameters at (more or less) arbitrary values. • The Bayesian formulation defines prior distributions for the parameters.

12. The Prior Distribution • When advocating a Bayesian approach, the usual question arises. • What about the prior distribution? • Various options are available: • Realistic beliefs of the wider community • The company’s genuine prior beliefs • Non-informative priors • Sceptical priors

13. Which Prior Distribution?

14. The Two Prior Distributions • We will allow different prior distributions at the two stages of design and analysis. • Analysis prior distribution. • This will typically be non-informative, sceptical or some kind of consensus. • Design prior distribution. • This should generally be based on all information available to the company.

15. Example Modified from Briggs and Tambour (1998). • Bayesian formulation, • analysis and design priors informative and identical, • prior means μ1 = 5, μ2 = 6.5; • prior variance = 4; prior co-variance = 3, • ω = 0.975 (1-sided, equates to 5% 2-sided test). • Profit : £5bn • Cost per patient : £1000; £10,000; £100,000; £1,000,000 • (Ratio : £5,000,000; £500,000; £50,000, £5,000)

18. Example - Conclusions • We have a prior probability of 85.55% that Treatment 2 is more efficacious than Treatment 1, which we approach rapidly at sample sizes above 2000 per treatment group. • The maximum utility (return on investment) depends on the ratio of the profit to the cost per patient. • Unnecessarily large sample sizes reduce the return on investment with no major increase in the probability that the regulator will be convinced to approve the registration of the treatment.

19. Discussion • The Bayesian approach allows considerable flexibility to represent the (real world) problem. • The Bayesian approach encourages the assessment of the genuine beliefs regarding the true treatment means. • Unequal sample sizes could be used is there is a prior belief that the variances are different between treatment groups. • The design objective allows the incorporation of prior information in determining the optimal sample size to maximise the return on investment. • Using the decision-based approach could provide an informed basis for management to use when allocating limited clinical resources to different clinical projects.