Statistical considerations in small proof-of-concept trials, including crossover designs

1. Statistical considerations in small proof-of-concept trials, including crossover designs Stephen Senn

2. People look down on marketing men • It’s not true that they are not scientists • They work in sell biology • I would like to take this opportunity to draw your attention to a book I rather like • In fact I wrote it myself

3. Outline • Decision analysis and proof of concept • Value of information perspective • Place of cross-over trials • Carry-over • The potential for cross-over trials in studying individual response

4. A Model Probability proof of concept (POC) study successful Probability proof of efficacy study (POE) successful if POC successful Probability POE study successful if POC unsuccessful Probability POE study successful Cost of POC including any lost sales through extra delay Cost of POE study Expected NPV revenue if POE initiated immediately and successful Value of strategy of POE only Value of strategy of POC + POE Value of POC study

5. Model Continued

6. Example Value of two strategies plotted against , the probability POE successful if POC successful

8. Value of Biomarker Information in Terms of Posterior Variance • Suppose that over all products for this indication the correlation of true therapeutic and biomarker outcomes is 0.9 • Let the prior means be zero in this class • Let the prior variances be 1 • Let the data variance of a minimal experiment be also 1 • Implies prior information equivalent to one minimal experiment

9. Here n is the number of minimal experiments we run Of course we expect a biomarker experiment to be cheaper than a therapeutic one Nevertheless note that fairly rapidly there is no interest in getting further biomarker information

10. A Serious Warning to Statisticians In the mathematical formulation of any problem it is necessary to base oneself on some appropriate idealizations and simplification. This is, however, a disadvantage; it is a distorting factor which one should always try to keep in check, and to approach circumspectly. It is unfortunate that the reverse often happens. One loses sight of the original nature of the problem, falls in love with the idealization, and then blames reality for not conforming to it. De Finetti 1975 ‘The only way that human beings could ever have survived as a species for long as we have is that we’ve developed another kind of decision-making apparatus that’s capable of making very quick judgements based on very little information. Malcolm Gladwell, Blink, 2005

11. My Gloomy Take on This • We don’t really understand this topic • There may be less value in proof of concept studies than we propose • Therapeutic studies may be valuable even if they have low power • There is no point in undertaking POC studies unless you can see circumstance under which they would cause you to cancel projects

12. Appropriate Attitudes for Cross-over Trials • They are not suitable for all indications and questions • They are extremely valuable for some indications and questions • Carry-over has to be dealt with by washout • Don’t pre-test for carry-over • Don’t rely on classical statistical approaches to carry-over • Cross-over trials have great potential in investigating individual response

13. Carry-over Definition: Carry-over is the persistence (whether physically or in terms of effect) of a treatment applied in one period in a subsequent period of treatment. If carry-over applies in a cross-over trial we shall, at some stage, observe the simultaneous effects of two or more treatments on given patients. We may, however, not be aware that this is what we are observing and this ignorance may lead us to make errors in interpretation.

14. The simple carry-over model. This is a very popular model amongst “applied” statisticians of a mathematical bent. The model assumes that if a carry-over effect is present 1) it lasts for one period exactly 2) it depends on the engendering treatment only and not on the perturbed treatment.

15. Three Period Bioequivalence Designs • Three formulation designs in six sequences common. • Subjects randomised in equal numbers to six possible sequences. • For example, 18 subjects, three on each of the sequences ABC, ACB, BAC, BCA, CAB, CBA. • A = test formulation under fasting conditions, • B = test formulation under fed conditions • C = reference formulation under fed conditions.

16. Weights for the Three Period Design: not Adjusting for Carry-over

17. Properties of these weights • Sum to 0 in any column, • eliminates the period effect. • Sum to 0 in any row • eliminates patient effect • Sum to 0 over cells labelled A • A has no part in definition of contrast • Sum to 1 over the cells labelled B and to -1 over the cells labelled C • Estimate contrast B-C

18. Weights for the Three Period Design: Adjusting for Carry-over B-C contrast: illustration of treatment effect and elimination of period and patient effects

19. Weights for the Three Period Design: Adjusting for Carry-over Illustration of elimination of ‘carry-over’ effects

20. Have We Got Something for Nothing? • Sum of squares weights of first scheme is 1/3 (or 4/12) • Sum of squares of weights of second scheme is 5/12 • Given independent homoscedastic within- patient errors, there is thus a 25% increase in variance • Penalty for adjusting is loss of efficiency

21. The difference between mathematical and applied statistics is that the former is full of lemmas whereas the latter is full of dilemmas

22. The Dangers of Pre-testing • Situation with AB/BA design • Two-stage procedure is very badly biased • CARRY and PAR are highly correlated • 1/2 <  < 1 • Three treatment design • Same problem • Carry-over and adjusted estimates correlated •  = 0.45

23. The Phoenix Bioequivalence Trials • Analysed by D’Angelo, Potvin & Turgeon * • 20 drug classes • 1989-1999 • 12 or more subjects • 96 three period designs • 324 two period designs D'Angelo, G.Potvin, D.Turgeon, J. J Biopharm Stats, 11, 27-36, 2001

24. Three Treatment Designs P-Values for Carry-Over “Significant” results in bold Senn, S. J., G. D'Angelo, et al. (2004). "Carry-over in cross-over trials in bioequivalence: theoretical concerns and empirical evidence." Pharmaceutical Statistics3(2): 133-142.

25. Two Treatment Designs “Significant” results in bold

26. Test of Uniformity of P-Values

27. Galling as this may appear to statisticians, the cure for carry-over is more biological and pharmacological understanding not more statistics

28. Conclusions • Distribution of P-values uniform • no evidence of carry-over • Carry-over a priori implausible • presence testable by assay • No point is testing for it • leads to bias • Or adjusting for it • increased variance

29. Possible Strategy • Run multi-period cross-overs • Patient by treatment interaction becomes identifiable • This provides an upper bound for gene by treatment interaction • Because patients differ by more than their genes

32. Cheap Low tech Insight into sources of variation gained Only suitable for chronic diseases Demanding of patient’s time Unglamorous Does not produce diagnostic patents Advantages and DisadvantagesPROCON

33. An Overlooked Source of Genetic Variability • Humans may be classified into two important genetic subtypes. • One of these suffers from a massive chromosomal deficiency. • This is expressed in. • Important phenotypic differences. • A massive disadvantage in life expectancy. • Many treatment strategies take no account of this. • The names of these subtypes are...

34. Men and Women

35. A Difficult Decision • You have $100 • Should you spend it on beer? • US 20 beers • UK 15 beers • Or on books? • In particular 1 book • Have I mentioned this before?