Measures of Clinical Significance Employed in Dementia Drug RCTs

1. Measures of Clinical Significance Employed in Dementia Drug RCTs Dr. Frank Molnar Associate Professor of Medicine, University of Ottawa, Canada Staff, The Ottawa Hospital Division of Geriatric Medicine Medical Director, The Ottawa Hospital Geriatric Day Hospital (focus on Dementia) Co-chair, Champlain Dementia Network ( www.champlaindementianetwork.org ) Medical Director, Regional Geriatric Program of Eastern Ontario ( www.rgpeo.com )

2. Conflict of Interest Disclosure Frank Molnar MSc, MDCM, FRCPC Has no real or apparent conflicts of interest to report.

3. Statistical vs. Clinical Significance • Statistically significant differences only tell us whether the effect of the drug can be mathematically distinguished from the effect of its comparator. • Clinically significant (or clinically important) differences detect much larger differences - whether patients, families and/or physicians feel improvement on the treatment is sufficiently superior to the comparator to merit the cost and the risk of side-effects. • Not all statistically significant differences are sufficiently large to be judged clinically significant

4. I. Empirically derived independent of a RCT: -generalizable to other studies (stable from study to study) • Minimally Clinically Important Difference (MCID) • Interpatient Minimally Detectable Difference (Interpatient MDD) • Intrapatient Minimally Detectable Difference (Intrapatient MDD)

5. Minimally Clinically Important Difference (MCID) • The smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and cost, a change in the patient’s management. • Risk / benefit assessment

6. Minimally Detectable Difference (MDD) • Intra-patient MDD has been determined by following patients over time and asking them whether they feel better or worse than previously while concurrently measuring change in outcome scales. • When patients feel slightly better or slightly worse, the corresponding change on the scale is the intra-patient MDD

7. Minimally Detectable Difference (MDD) • Inter-patient MDD involves asking patients to compare themselves to other patients with the same condition. It requires assembling a group of patients so that they can watch each other and converse to determine each other’s quality of life and function. • When patients can detect a difference in state between themselves and others, the corresponding change on the scale is the inter-patient MDD • For highly personalized aspects of health, interpersonal judgments may be unreliable, which may be particularly relevant in dementia

8. Minimally Detectable Difference (MDD) • We do not usually treat patients to merely obtain a change (effect) that they or their families can barely perceive. • We treat to obtain a larger change (effect) that is worth the cost and worth risking the side-effects. • We treat to obtain an MCID not and MDD

9. II. Empirically derived within a RCT: - not generalizable to other RCTs • Goal Attainment Scaling (GAS) • Global Measures • Clinician Interview-Based Impression of Change plus caregiver input (CIBIC plus) • Clinician’s Global Impression of Change (CGIC)

10. Goal Attainment Scaling • GAS asks patient or caregiver ‘‘what specific changes would be considered meaningful to you?’’ and creates a scale for these goals • Because GAS does not traditionally counterbalance the benefits with risks and cost, it also falls short of a traditional MCID.

11. Goal Attainment Scaling • Another limitation of GAS is that each patient, caregiver, or clinician may select a different type of outcome (mobility, function, independence …). • GAS results cannot be generalized to other studies, because GAS measures are specific to particularpatients in a particular study.

12. Global Measures • CGIC • Clinician’s Global Impression of Change • CIBIC-plus • Clinician’s Interview-Based Impression of Change plus Caregiver input

13. Global Measures • 7 point Likert scales – overall status • 1. Markedly worse • 2. Moderately worse • 3. Minimally worse • 4. Unchanged • 5. Minimally better • 6. Moderately better • 7. Markedly better

14. Global Measures • Global measures are limited, because it is often difficult to relate improved global scores to individual domains or specific outcome measures within domains. • An individual may be rated as improved because of the cumulative effect of small improvements in several domains (e.g., cognition, behaviour, function). • Conversely, an individual may be rated as unchanged or as deteriorated if specific improvements are counterbalanced by deteriorations in other domains. • Therefore, it is difficult, if not impossible, to use global measures to determine the clinically important difference within one domain or on one outcome measure (cannot convert global measures to an MCID for a particular scale).

15. III. Opinion-Based • Conference • Survey • Regulatory Committee (e.g. FDA) • Opinion of authors of articles

16. Opinion-Based measures • Limited by lack of empirically derived alternatives • selecting from among unproven non-empirically derived measures of clinical importance • May be biased by process used to select participants • Does not directly incorporate patient or caregiver perspectives • Opinions of authors may be prone to bias in favour of positive study results • Some approaches (committee, authors) limited to opinions of small non-representative groups of experts

17. IV. Mathematical • Neither patient, caregiver nor clinician perspectives considered • Only use when none of previous options available • Effect Size

18. Effect Size Cohen’s D, Standardized Response Means (SRMs) etc • Calculated by taking the difference in treatment effect between the treatment group and the comparator (e.g. placebo) and divides this by a measure of data dispersion such as standard deviation (this varies from study to study). It is essentially a measure of ‘signal-to-noise ratio’ • ES = (Treatment – Control) = Signal standard deviation Noise • ES estimates the probability of a clinically significant response. • The higher the signal-to-noise ratio (i.e. the larger the effect size), the more likely / probable the results are to be clinically important.

19. Effect Size • Effect size does nottell us that a result actually is clinically important and does not set evidence-based thresholds at which outcome measures are felt to be clinically important. • Traditional non-evidence based definitions • 0.2 – 0.3 small ES • 0.5 medium ES • 0.8 – 1.0 large ES

20. Effect Size • Effect size is not stable. • The amount of change on an outcome scale that will generate a specific effect size will vary from study to study as the standard deviation of outcome measures in the different studies changes • ES = (Treatment – Control) = Signal standard deviation (SD) Noise • Same group difference; • Homogeneous population → ↓ SD → ↑ ES • Heterogeneous population → ↑ SD → ↓ ES

21. The Systematic Review • Systematic Review of Measures of Clinical Significance Employed in Randomized Controlled Trials of Drugs for Dementia • Journal of the American Geriatrics Society 57:536–546, March 2009

22. JAGS Review - Results • Of the 1,146 articles identified using the search strategies, 191 were selected for full text review. Of these, 57 articles met eligibility criteria for the systematic review. • 26 of these 57 RCTs (46%) discussed clinical significance.

23. JAGS Review – Results Summary • Of the 26 trials that discussed clinical significance; • most employed opinion-based estimates of clinically significant changes on specific cognitive scales (n = 19) OR • clinical importance was captured using global measures of change (n = 8).

24. JAGS Review – Results Summary • Some global measures, such as the Clinician’s Interview- Based Impression of Change Plus Caregiver Input (CIBIC-plus), allow physicians and investigators to incorporate caregiver input when forming their opinions. • Seven of the eight studies employing global measures used the CIBIC-plus and hence incorporated caregiver input. • Only one RCT directly measured thresholds for clinical importance from the patient perspective (e.g., patient Goal Attainment Scale).

25. JAGS Review – Results Summary • The most commonly cited measures of clinical significance were; • 4-point change in the Alzheimer’s Disease Assessment Scale-Cognitive Subscale (ADASCog) as recommended by a U.S. Food and Drug Administration committee (n = 7) • linking clinical significance to global scales (n = 8) such as the Clinician’s Global Impression of Change (CGIC) and the CIBIC-plus.

26. Conclusions • Within the minority of studies that discussed clinical significance, there were a variety of investigator opinions regarding how to measure clinical importance in dementia trials. • Only one study (Rockwood’s study using GAS) directly measured patient opinion • None of the studies employed MCIDs that incorporate patient or family perspectives regarding whether the risk: benefit ratios of the medications studied favoured clinical use

27. QUESTIONS? • How do you think clinically significant changes should be captured in dementia drug RCTs? • Cognitive scale • CIBIC like approach • GAS-like approach • Multiple outcomes • ??????