Lost Opportunities for Design Theory in Drug Development

1. Lost Opportunities for Design Theory in Drug Development Stephen Senn

2. Basic Thesis • Design theory has great potential in drug development • But this potential is unrealised • Those working in so-called optimal design are so ignorant of application realities that where their influence is not zero it is harmful • On the other hand the understanding of design theory by biostatisticians is pitifully inadequate • We must cooperate properly to cure this parlous state of affairs

3. Outline • Quick tutorial on cross-over trials • I shall then give two introductory examples of nonsense • By leading design theoreticians • By leading biostatisticians • I shall then consider ‘design nonsense’ further • Some conclusions • After lunch a case-study

4. Warning • I am a biostatistician • We are used to thinking of data matrices with rows as subjects and columns as measurements • That means that we write sequences for designs with rows representing subjects and columns representing periods

5. Cross-over Trials Definition: A cross-over trial is one in which subjects are given sequences with the object of studying differences between individual treatments.

6. An Example of an AB/BA cross-over in asthma

7. An Example from Rheumatism:2 doses of diclofenac and placebo

8. Carry-over Definition: Carry-over is the persistence (whether physically or in terms of effect) of a treatment applied in one period in a subsequent period of treatment. If carry-over applies in a cross-over trial we shall, at some stage, observe the simultaneous effects of two or more treatments on given patients. We may, however, not be aware that this is what we are observing and this ignorance may lead us to make errors in interpretation.

9. Simple Carry-over • Carry-over lasts for exactly one period • It depends only on the engendering treatment and is unmodified by the perturbed treatment • There is a huge literature proposing ‘optimal’ designs for this model • There is no empirical evidence that any of this has been useful

10. Three Period Bioequivalence Designs • Three formulation designs in six sequences common. • Subjects randomised in equal numbers to six possible sequences. • For example, 18 subjects, three on each of the sequences ABC, ACB, BAC, BCA, CAB, CBA. • A = test formulation under fasting conditions, • B = test formulation under fed conditions • C = reference formulation under fed conditions.

11. Weights for the Three Period Design: not Adjusting for Carry-over

12. Properties of these weights • Sum 0 in any column, • eliminates the period effect. • Sum 0 in any row • eliminates patient effect • Sum 0 over cells labelled A • A has no part in definition of contrast • Sum to 1 over the cells labelled B and to -1 over the cells labelled C • Estimate contrast B-C

13. Weights for the Three Period Design: Adjusting for Carry-over

14. Weights for the Three Period Design: Adjusting for Carry-over

15. Properties of These Weights • As before • Estimates B-C contrast • Eliminates, period and patient effect • Eliminates A • Sum to zero over cells labelled a,b, and c • Eliminate simple carry-over

16. Have We Got Something for Nothing? • Sum of squares weights of first scheme is 1/3 (or 4/12) • Sum of squares of weights of second scheme is 5/12 • Given independent homoscedastic within- patient errors, there is thus a 25% increase in variance • Penalty for adjusting is loss of efficiency

17. First ExampleSome Design Theory Nonsense John, J. A., Russell, K. G., and Whitaker, D. (2004), "Crossover: An Algorithm for the Construction of Efficient Cross-over Designs," Statistics in Medicine, 23, 2645 - 2658. A cross-over experiment involves the application of sequences of treatments to several subjects over a number of time periods. It is thought that the observation made on each subject at the end of a time period may depend on the direct effect of the treatment applied in the current period, and the carry-over effects of the treatments applied in one or more previous periods. Various models have been proposed to explain the nature of the carry-over effects. An experimental design that is optimal under one model may not be optimal if a different model is the appropriate one. In this paper an algorithm is described to construct efficient cross-over designs for a range of models that involve the direct effects of the treatments and various functions of their carry-over effects. The effectiveness and flexibility of the algorithm are demonstrated by assessing its performance against numerous designs and models given in the literature.

18. What’s wrong here? “Sometimes in a clinical trial it may be necessary to modify or extend an ongoing trial. For example, suppose that after a few periods have been completed one of the treatments is dropped from the trial. It will then be necessary to re-allocate the other treatments to the remaining periods of the trial.” (John et al, 2004 p. 2653) “Jones and Donev [17] also consider augmenting a design to account for the removal of a treatment. The initial trial to compare four treatments A, B, C, and D in five periods using four groups of subjects used a Williams square [3] with the fourth period repeated. After the first two periods had been completed it was decided to drop treatment D from the remainder of the trial.”

20. The reality ..in a single-dose cross-over trial in asthma in 12 patients reported by Palmqvist et al. [5] patients were treated in the first period of a cross-over on dates ranging from 5 May to 12 November 1987 [4]. They were treated in a second period on dates ranging from 18 May to 26 November 1987. Eleven of the patients had completed period two of treatment before the 12th patient was recruited. (Senn, 2005, p3675.) A basic fact of clinical trials You treat patients when they fall ill

21. Multi-Story

22. Conclusion • Multi-dose trials real scope for design theory. • These will employ active wash-out • Design problem is trade-off between exploiting correlation and eliminating carry-over. • Short vs long active wash-out periods

23. Second ExampleSome Biostatistics Nonsense • Chow, S. C., and Liu, J. P. (2000), Design and Analysis of Bioavailability and Bioequivalence Studies (2nd ed.), New York: Marcel Dekker. • Have several discussions of efficiency of designs in their book which are completely beside the point. They compare designs in terms of residual degrees of freedom! • They conclude that Balaam’s design, which uses sequences • TR/RT/TT/RR • Is similar in efficiency to the more conventional • TR/RT design. • They write “The degrees of freedom for the intrasubject residuals for the 2 × 2 and 4 × 2 design are 22 and 21 respectively. Therefore there is little difference in testing power.” • This is nonsense

24. Second ExampleSome Biostatistics Nonsense .

25. What is wrong 1. It’s not correct design theory • As any design expert knows residual degrees of freedom are (nearly) irrelevant to efficiency • It is the impact of adjustment on the degree of orthogonality of the design matrix that is important

26. What is wrong 2.It’s not realistic biostatistics • In fact as any biostatistician who has had to think about it will know from a practical point of view far from being optimal Balaam’s design is simply inadmissible • The reasons is that only half of the resources are devoted to actually measuring the treatment • The rest are devoted to providing an adjustment for a form of carry-over that is itself implausible

27. Allocation of patients for two designs

28. Investigation of the real efficiency of Balaam’s design

29. Five Reasons why the Simple Carry-over Model is not Useful • If it applies then the investigator can design a trial which eliminates it . (double the periods) • Implausible given pk/pd theory. (obvious) • Leads to inefficient estimators. (see investigation to follow) • Can lead to poor designs. (ditto) • The models which incorporate it are self-contradicting. (example: factorial X-overs)

30. Dose response: the pharmacokineticist’s version

31. Dose Response:The Statistician’s Version Response This is what the simple carry-over model implies Dose

32. The Models Which use Simple Carry-over are Inconsistent Consider a factorial cross-over in four periods comparing A, B and the combination of A and B to placebo. We can represent the four treatments by: **, A*, *B and AB. Suppose we consider a patient who has received the sequence AB ** A* *B. A standard parameterisation for treatment and carry-over would be as in the following table.

33. Paramaterisation of a factorial cross-over Period

34. The Rhinoceros The rhinoceros has a kind heart, if you doubt it here’s the proof That thing on his nose is for taking stones out of a horse’s hoof He seldom ever meets a horse, it is this that makes him sad When he does, then it hasn’t a stone in its hoof But he would, if he did and it had Flanders and Swann

35. The Phoenix Bioequivalence Trials • Analysed by D’Angelo & Potvin • 20 drug classes • 1989-1999 • 12 or more subjects • 96 three period designs • 324 two period designs

36. Three Treatment Designs P-Values for Carry-Over

37. Two Treatment Designs

38. Test of Uniformity of P-Values

39. Conclusions • Distribution of P-values uniform • no evidence of carry-over • Carry-over a priori implausible • presence testable by assay • No point is testing for it • leads to bias • Or adjusting for it • increased variance

40. Do Bayesians do Better? • In principle the Bayesian approach ought to allow us to be more flexible about nuisance parameters such as carry-over • However, the Bayesian track record is not impressive here • Realistic models have not been employed

41. Cross-over trial in Eneuresis Two treatment periods of 14 days each Treatment effect significant if carry-over not fitted 2.037 ( 0.768, 3.306) Treatment effect not significant if carry-over fitted 0.451 (-2.272, 3.174) 1. Hills, M, Armitage, P. The two-period cross-over clinical trial, British Journal of Clinical Pharmacology 1979;8: 7-20.

42. Identical ‘uninformative’ prior placed on carry-over as for treatment NB Parameterisation here means that values of  need to be doubled to compare to conventional contrasts

43. Identical Priors for Treatment and Carryover? • Patients treated repeatedly during trial • Fourteen day treatment period • Average time to last treatment plausibly 4 hours • Average time to previous treatment seven days • Saying that it is just as likely that carry-over could be greater than treatment is not coherent • In any case the two cannot be independent • Is negative carry-over as likely as positive carry-over?

44. So What are Acceptable Models for Carry-over? • Ignoring carry-over altogether (not allowing for it because one believes one has taken adequate steps to eliminate it) • This is always a reasonable strategy • Using an integrated pharmacokinetic pharmacodynamic model (Sheiner et al, 1991) • This may work for dose-finding trials • Very difficult to implement where more than one molecule is involved

45. The Sheiner model PD dose response PK model for dose-concentration as a consequence of previous dosing history Steady state concentration for patient i in period l

49. The difference between mathematical and applied statistics is that the former is full of lemmas whereas the latter is full of dilemmas

50. Advice for Design-Theoreticians • Resist the temptation to give advice if you are unfamiliar with the application area • Seek collaborators • Ground your models in pharmacology • Remember that the goal is good medicine not elegant mathematics • Don’t defend the indefensible