Results 2 (cont’d) c) Long term observational data on the duration of effective response

1. A TWO LEVEL MONTE CARLO APPROACH TO CALCULATING EXPECTED VALUE OF SAMPLE INFORMATION (EVSI):- HOW TO VALUE A RESEARCH DESIGN. Alan Brennan, , Jim Chilcott, Samer Kharroubi, Tony O’Hagan, University of Sheffield, England. a.brennan@sheffield.ac.uk . • Purpose:describe a generalised two level Monte Carlo simulation for • calculating partial EVSI for parameters in a health economic decision model. • Background: Claxton has promoted the EVSI concept as a measure of the • societal value of research designs to help identify optimal sample sizes for primary data collection.1,2 The method (using unit normal loss integral formula to calculate EVSI for sample size n), only applies if the net benefit function is normally distributed, and if the proposed sample exercise measures all the model parameters. • Such conditions are not universal or necessarily common. • A generalised method is needed for partial EVSI. Mathematical Formulation: EVSI for Parameters  = the parameters for the model (uncertain currently). d = set of possible decisions or strategies. NB(d, ) = the net benefit for decision d, and parameters  Step 1: no further information (the value of the baseline decision) Given current information chose decision giving maximum expected net benefit. Expected net benefit (no further info) = (1) i = the parameters of interest for partial EVPI  -i = the other parameters (those not of interest, i.e. remaining uncertainty) Step 6: Sample Information on i Expected Net benefit, sample on i = (6) Step 7: Expected Value of Sample Information on i (6) – (1) Partial EVSI = (7) This is a 2 level simulation due to 2 expectations How to Do it: Case 1 = Normal 0= prior mean for the parameter 0= prior uncertainty in the mean (standard deviation) = precision of the prior mean (B1) 2pop = patient level uncertainty from a sample ( needed for Bayesian update formula)   = sample mean (further data collection from more patients / clinical trial study entrants). = precision of the sample mean (B2). = sample variance = implied posterior mean (the Bayesian update of the mean following the sample information) = implied posterior standard deviation (the Bayesian update of . the mean following the sample information) Note implied posterior variance will always be smaller than the prior variance because the denominator of the adjustment term is always larger than the numerator. If the sample size is very small then ,in formula 5, the adjustment term will almost be equal to 1 and posterior variance is almost identical to the prior variance. If the sample size is very large, the numerator of the adjustment term tends to zero, the denominator tends to the prior variance and so, posterior variance tends towards zero. How to Do it: Case 2 = Beta / Binomial (e.g. % responders) Suppose prior for % of responders is ~ Beta (a,b) If we obtain a further n cases, of which y are successful responders then Posterior ~ Beta (a+y,b+n-y) How to Do it: Case 3 = Gamma / Poisson (e.g. no. of side effects a patient experiences in a year ) Suppose prior for mean number of side effects per person is ~ Gamma (a,b) If we obtain a further n samples, (y1, y2, … yn) from a Poisson distribution then Posterior for mean number of side effects per person ~ Gamma (a+ yi, b+n) How to Do it: Case 4 = Lognormal (e.g. cost of hospitlaisation) Convert to normal variables i.e. exp(lognormal), then do Bayesian update as in Normal (Case 1 above), then convert back to lognormal • The Algorithm: Part 1 – baseline decision and EVPI • 0) Set up a decision model comparing different strategies and set up a decision rule e.g. • “select strategy if marginal cost per QALY is < $50,000”. This is equivalent to “chose • strategy with highest net benefit” where, Net benefit = $50,000 * QALY – Cost • 1) Characterise uncertain parameters with probability distributions e.g. normal(μ,σ2), • beta(a,b), gamma (a,b), triangular(a,b,c) … etc (prior knowledge) • 2) Simulate say 1,000 sample sets of uncertain parameter values (monte carlo). • 3) Work out the baseline decision – i.e. strategy giving the highest net benefit • (on average over the 1,000 simulations). • 4) For each sample set of parameters, work out the optimal strategy given that particular • “possible future world”, record the net benefit • 5) With “perfect” information (i.e totally accurate estimates of each parameter) we would • always choose the optimal strategy. Overall expected value of perfect information is • Part 2 – Two Level Algorithm for EVSI for Parameters (one parameter version) • 6) Simulate a data collection exercise for your parameter of interest by: • sampling the true underlying value 1 for parameter of interest • (e.g. the underlying true % response rate to a drug) • once from its prior range of uncertainty, • then deciding on a sample size (n) for the parameter of interest, and then • sampling a mean value for the simulated data obtained on the parameter of interest • (e.g. the response rate found in a trial of n=100, given 1). (1st level) • 7) combine the prior knowledge with the simulated data for each parameter using • Bayes formula to update the probability distribution for the parameter of interest • and produce a simulated posterior distribution for the parameter of interest • This has a revised mean and a smaller variance than the ‘prior’ because new • information has been obtained. • 8) simulate the new combined data set allowing theparameters of interest to vary • across the uncertain range defined by their simulated posteriorand the other • remaining unknown parameters to vary according to prior uncertainty (2nd level) • 9) calculate the best strategy given this new knowledge on the parameter of interest • i.e. strategy which gives the most net benefit over all the simulations in (8) • 10) calculate the mean net benefit of this new best strategy • 11) Repeat steps 6 to 10, say 1,000 times, and calculate the average of the net benefits • obtained in (10) • 12) The average value of gaining the specified sample information on the parameter of • interest is then • EVSI (n)= average net benefit given sample information on the parameter (11) - • average net benefit given no further information i.e. NB of baseline decision (3) • This same algorithm can be used for sets of parameters, each parameter can have a different sample size, giving the ability to reflect complex research designs. Result 1: Individual Variables EVSI can be done for individual variables e.g. response rate to drug T0. The value of the knowledge increases as sample size increases but reaches a plateau at the perfect information ceiling. Results 2: Sets of Parameters (Proposed research designs) a) Trial Only – measure response rates T0,T1 Trial of n = 50 has EVSI = £330 b) Utility – measure utility improvement for responders to T0, T1 Utility study of n = 50 has EVSI = £716 Results 2 (cont’d) c) Long term observational data on the duration of effective response Observational data on n=50 has EVSI = £867 d) Collect data on all variables Data on all variables n=50 has EVSI = £1319 Conclusions 1) The mathematical expression for partial EVSI shows the 2 level monte carlo simulation needed. 2) Given sample data, Bayesian updating requires an analytic expression to revise parameters of the distribution (given here for normal, beta, gamma and lognormal distributions). 3) More general distributions may require other approximations or MCMC methods. 4) The algorithm applies equally for sets of parameters as for a single one and provides a significant step towards a generalised method for partial EVSI 6) Further work is needed on computational issues- how many samples? Non-linear approximations? - see Matt Stevenson, Jeremy Oakley posters 1Claxton and Posenettand Posnett J. "An economic approach to clinical trial design and research priority setting", Health Economics, 1996(5); 6: 513-524. 2 Claxton et al. Bayesian Value of Information Analysis: An application to a Policy Model of Alzheimer's Disease, International Journal of Technology Assessment in Health Care, Volume 17 Issue 1, 2001 Acknowledgements: Co-authors: Jim, Samer and Tony (for help in thinking, working through maths and encouragement to get it done). I am a member of CHEBS: The Centre for Bayesian Statistics in Health Economics. Particular thanks to Karl Claxton and Tony Ades who helped finalise our thinking at a CHEBS “focus fortnight” event in April 2002.