Veterinary clinical studies Key issues for statistical analysis

1. Ecole Nationale Vétérinaire de Toulouse Veterinary clinical studiesKey issues for statistical analysis Didier Concordet d.concordet@envt.fr ECVPT Workshop July 2009 Can be downloaded at http://www.biostat.envt.fr/spip/spip.php?article34

2. Vocabulary • Bias (Statistical & Operational) • Blind Review • Content Validity • Double-Dummy • Dropout • Equivalence Trial • Frequentist Methods • Full Analysis Set • Generalisability, Generalisation • Global Assessment Variable • Independent Data Monitoring Committee (IDMC) (Data and Safety Monitoring Board, • Monitoring Committee, Data Monitoring Committee) • Intention-To-Treat Principle • Interaction (Qualitative & Quantitative) • Inter-Rater Reliability • Intra-Rater Reliability • Interim Analysis • Meta-Analysis • Multicentre Trial • Non-Inferiority Trial • Preferred and Included Terms • Per Protocol Set (Valid Cases, Efficacy Sample, Evaluable Subjects Sample) • Safety & Tolerability • Statistical Analysis Plan • Superiority Trial • Surrogate Variable • Treatment Effect • Treatment Emergent • Trial Statistician Boring From ICH Topic E 9

3. Aim of clinical trials Sample Population To assess the efficacy of a drug in a (target) population Issues at each stage Population : the set of individuals that can receive the drug Practically Design/Sampling Inference

4. ISSUES • When designing the trial • When collecting data • When analysing data • When interpreting results

5. ISSUES • When designing the trial • When collecting data • When analysing data • When interpreting results • Sampling the target population • Different kinds of clinical trials • How to detect bias

6. ISSUES • When designing the trial • When collecting data • When analysing data • When interpreting results • Sampling the target population • Different kinds of clinical trials • How to detect bias

7. Sampling the target population There exist sources of variation that make the judgment criterion vary Example with two breeds Judgment criterion

8. Sampling the target population The two same breeds with different proportions Judgment criterion

9. Sampling the target population The sample should be representative of the target population Target population Sample .<1 year 1<= . <2 years The sample has the same structure as the population Female 2<= . <3 years Male breed 6 breed 1 breed 4 breed 2 breed 3 breed 5

10. Two main ways to sample the population Randomization: leave chance make the job the percentage of the animals in each subgroup should be close to the population's one. Stratification: help the chance to do the job Build a sample of animals that has exactly the same percentage of individuals in each subgroup as the population. This requires to know the repartition of subgroups in the population.

11. Target population definition An experiment in 2 years old beagles showed that the temperature of dogs treated with the antipyretic drug A decreased by 2 °C. What assumptions do we need for this result to hold for all 3 years old beagles beagles dogs man 11

12. ISSUES • When designing the trial • When collecting data • When analysing data • When interpreting results • Sampling the target population • Different kinds of clinical trials • How to detect bias

13. Different kinds of clinical trials • Non inferiority • Superiority • Equivalence

14. Reference – D (penalty) Reference D : non inferiority margin Superiority New treatment Efficacy Reference + D Reference D : superiority margin Different kinds of clinical trials Non inferiority New treatment Efficacy low high

15. Different kinds of clinical trials Equivalence trial New treatment Efficacy Reference – D Reference Reference + D

16. Reference – D (penalty) Reference Non inferiority trial New treatment Efficacy The new treatment can have a smaller efficacy than the reference treatment

17. Reference – D (penalty) Reference Non inferiority trial New treatment • the reference treatment is not efficacious • animals included in the trial are not sick • the judgment criterion is not relevant (e.g. does not vary) • delta is too large Efficacy

18. Is there a problem ? Is there a problemorganizing a non inferiority trial able to demonstrate Reference treatment New treatment Decrease of rectal temperature of at least 1.5°C Decrease of rectal temperature of at least 1.2°C Cure rate = 75 % Cure rate = 65 %

19. A clinical trial should avoid bias Bias : the difference between the compared drugs at the end of the trial due to other things than the drugs • Confusion bias • Selection bias • Follow-up bias • Attrition bias

20. Confusion bias Arises when one do not taking into account a confusion factor. To avoid such bias, the trial should be comparative and should have a contemporarycontrol group used as a reference group. Questions Warning • Despite a control group the treatment effect is measured with a "before-after" comparison. • Is there a control group ? • Is the treatment effect determined with respect to this control group ?

21. Selection bias Arises when the two groups to be compared are different (with respect to the endpoint before the beginning of the trial. To avoid it one uses a randomisation : a random allocation of animals into treatment groups Questions Warning • There is a historical control group (no randomisation) • The investigators were able to select the animals for a group • Is there a randomisation procedure ? • Are the two groups balanced ?

22. Follow-up bias Arises when the follow-up is not the same for the two drugs to be compared. Destroy initial comparability. To avoid it : double blind Warning Questions • The treatments were discernable • The investigators were able to select the animals for a group • The judgment criterion was subjective (eg : the animal feels better ) • Is the trial double blind ? • Is the rate of concomitant medications the same for the two groups ? • Are the protocol deviations similar ? • Are the drop-out number similar ?

23. Attrition bias Arises when some randomised animals are excluded. To avoid it Analysis of the Intention to Treat dataset Questions Warning • Is the number of analysed animals equal to the number of randomized animals ? • Was an imputation method used for missing data ? • Intention to treat analysis • Per Protocol analysis (only the animals alive and non excluded were analysed) • High rate of concomitant treatments ? • High rate of protocol deviations ? • High rate of drop-out ?

24. 41 Temperature (°C) 38 Before treatment After treatment Example Efficacy of an antipyretic drug. Inclusion of 30 dogs with at least 39.5°C of temperature.

25. ISSUES • When designing the trial • When collecting data • When analysing data • When interpreting results Missing data

26. Missing data should be adequately reported Ignorable missing data: Data imputation does allow to treat such missing data Leads to the ITT dataset Non-Ignorable missing data: The missingness mechanism has be clearly described Three kinds of missingness mechanism • data Missing Completely At Random (MCAR) • The missingness is independent of data • data Missing At Random (MAR) • The missingness depends on observed data • data Missing Not At Random (MNAR) • The missingness depends on the non observed data 26

27. Missing Completely At RandomMCAR Missingness and outcome are independent • the owner of the animal missed a visit to the vet • the investigator forgot to write the results • the owner moves house Unlikely to occur in a clinical trial 27

28. Missing At RandomMAR Missingness depends on data that have been observed but not on the unobserved (missing) data • dropout related to baseline characteristics • the animal health has markedly improved or deteriorated since inclusion Assumes that the future trajectories of animals who dropout are similar to those who share the same measurements whether or not they dropout. Frequent in clinical trials. 28

29. Missing Not At RandomMNAR Missingness depends on data that have been unobserved (missing data) • sudden decline or improve in health that has not been observed in the previous visits Assumes that the future trajectories of animals who dropout are different to those who share the same measurements Occurs in clinical trials. 29

30. Can you classify these missing data ? • The battery of the thermometer is discharged. I cannot measure the temperature. • At the last visit, the dog was well. I called the owner by phone, he did not want to come because he said that the dog was cured. • The owner did not come back. I don't know why. 30

31. ISSUES • When designing the trial • When collecting data • When analysing data • When interpreting results • Statistical tests • Multiple comparisons • Data drying off • What dataset to analyse ?

32. ISSUES • When designing the trial • When collecting data • When analysing data • When interpreting results • Statistical tests • Multiple comparisons • Data washing • What dataset to analyse ?

33. Statistical analysis Sample Population Objective : To draw conclusions on the target population from observation of a sample Inference

34. There is a difference in the population. This conclusion is drawn with less than 5% risk. Non significant difference. We would take too much risk by claiming a difference in the population. Things to know about statistical tests Sample Target population P<5% Test Observed difference P≥5%

35. Test 1 Test 2 Test 3 Test 4 Risk to wrongly conclude to a difference= 5% Risk to wrongly conclude to a difference= 5% Risk to wrongly conclude to a difference= 5% Risk to wrongly conclude to a difference= 5% Repetition of testsalso called multiple comparisons n global risk 1 0.05 2 0.10 3 0.13 5 0.23 10 0.40 Globally, the risk to wrongly conclude to a difference for 4 comparisons is 18%. Risk inflation

36. Multiple comparisons One wants to compare the ADG obtained with 5 different diets in pig Ten T-tests A risk of 5% for each comparison : the global risk can be very large here 40% 36

37. Choosing the question to get an answer Occurs frequently in the analysis of clinical trials results The question becomes random : it changes with the sample of animals. The question is chosen with its answer in hands… Think about a flip coin game where you win 1€ when tail or head occurs. You choose the decision rule once you know the result of the flip ! Such an approach increases the number of false discoveries. 37

38. Target population?! Data drying off:Analysis in subgroups • Dog (eff. NSAID) P difference with placebo • Age<10 0.92 • Age>=10 0.95 • Male 0.81 • Female 0.78 • Format small 0.63 • 5 Format medium 0.91 • 6 Format Large 0.74 • Food dry 0.01 • Food wet 0.63

39. a priori definition of a main criterion A single statistical test Risk to wrongly conclude to efficacy = 5% No definition of a main criterion 7 statistical test Risk to wrongly conclude to efficacy of the new treatment : 30% Data drying off: a posteriori choice of the judgment criterion • Death all causes • Death cardiovascular origin • Sudden death • Infarct • Vascular cerebral accident • Surgery • Main criterion • Death all causes • Secondary criteria • Death cardiovascular • Sudden death • Infarct • Vascular cerebral accident • Surgery From Cucherat 2005

40. What dataset to analyse ? • Intention To Treat dataset is based on the initial treatment intent, not on the treatment eventually administered regardless the drop-out. • Per Protocol dataset contains animals who have not dropped out for any reason regardless of initial randomization.

41. Example : complete data 41

42. Example : complete data 42

43. Drop-out (MAR) Drop-out 43

44. Intention To Treat dataset with Last Observation Carried Forward (LOCF) 44

45. Per Protocol dataset Only the animals that did not dropped-out were used 45

46. ISSUES • When designing the trial • When collecting data • When analysing data • When interpreting results • Standard error and standard deviation • P-Values

47. Standard error / standard deviation The clairance of the drug was equal to 68 ± 5 mL/mn Two possible meanings depending on the meaning of 5 If 5 is the standard error of the mean (se) there is 95 % chance that the population mean clearance belongs to [68 - 2  5 ; 68 + 2  5 ] If 5 is the standard deviation (SD) 95 % of animals have their clearance within [68 - 2  5 ; 68 + 2  5 ] 47

48. P values NO The difference between the effect of the drugs A and B is not significant (P = 0.56) therefore drug A can be substituted by drug B. The only conclusion that can be drawn from such a P value is that you didn't see any difference between the effect of the drugs A and B. That does not mean that such a difference does not exist. Absence of evidence is not evidence of absence 48

49. P values NO The drug A has a higher efficacy than the drug B (P = 0.001) The drug C has a higher efficacy than the drug B (P = 0.04) Since 0.001<0.04 the drug A has a higher than the drug B. The only conclusion that can be drawn from such a P value is that you are sure than A>B and less sure than C>B. This does not presume anything about the amplitude of the differences. Significant does not mean important 49

50. How to avoid these problems ? • Consult your preferred statistician for help in the design of complicated experiments • Use basic descriptive statistics first (graphics, summary statistics,…) • Use common sense • Consider to learn more statistics 50