290 likes | 588 Views
Assessment of Reliable Change: Methods and Assumptions. Michael Basso, Ph.D. Associate Professor and Director of Clinical Training Department of Psychology—University of Tulsa Clinical Associate Professor Department of Psychiatry—University of Oklahoma. Objectives.
E N D
Assessment of Reliable Change: Methods and Assumptions Michael Basso, Ph.D. Associate Professor and Director of Clinical Training Department of Psychology—University of Tulsa Clinical Associate Professor Department of Psychiatry—University of Oklahoma
Objectives • Provide background concerning methods of assessing reliable change • Describe assumptions and applications of reliable change scores • Illustrate use of reliable change scores
Assessment of Clinical Change • Two Basic Approaches • Assessment of Group Differences Across Time • Assessment of Individual Differences Across Time
Assessment of Group Differences Across Time • Assessment of statistically reliable change • “Does the treatment yield significant benefits for groups of patients?” • i.e., do average scores at T1 and T2 come from different distributions • This approach describes the average rate of change over groups primarily • It is accomplished with repeated measure ANOVA • Problem: You could have a statistically significant difference with a very small effect size, but it might not be a clinically meaningful change
Assessment of Individual Differences Across Time • Assessment of Clinically Meaningful Change • “Did the patient’s change in performance at T2 exceed base rates of change?” • i.e., did the individual show change that exceeded expectations based on measurement error, practice effects, and regression to the mean? • This method describes the base rate of change • Change that is exceeds the base rate is not normal, and is therefore clinically meaningful • Our focus is on the assessment of clinically meaningful change in individuals, but this method can be applied to group data as well
Assessment of Clinical Change for Individuals • How do you establish the base-rate of change? • Bear into consideration that: • It would be improbable to obtain the exact same score twice • There is no perfect test-retest correspondence because of • measurement error • regression to the mean • practice effects
Two Methods of Assessing Base Rates of Change • Reliable change Index scores • “Does change exceed what would be expected based on measurement error alone?” • This method is based on Reliability of measurement • It is used for typical performance tests • i.e., attitude, personality, psychopathology, etc. • Standardized Regression-Based Change Scores • “Does change in scores exceed expectations based on T1 (baseline) scores?” • This method is based on a validity coefficient (i.e., what T2 score is predicted by the T1 score) • It is used for maximal performance tests • i.e., IQ, neuropsychological, etc.
Reliable Change Index Scores • Elaborated by Jacobson and Truax (1991) • Based on the standard error of the difference • Which in turn is based on the reliability coefficient • This reflects the sampling distribution of difference scores • it implies the magnitude of differences between two test scores that vary by chance alone • Assumptions • Error components are mutually independent and independent of true pretest and posttest scores • Error is normally distributed with a mean of 0 • SE of error is equal for all subjects • These assumptions are questionable in clinical settings (cf. Maassen, 2004)
Reliable Change Index Scores--Method • To use the RCI, you must compute the SE of difference between two scores • SEdiff=(2(SD(1-rxx)1/2)2)1/2 • Then, compute a confidence interval for change scores • for 95% confidence, you multiply 1.96 * SEdiff • for 90% confidence, you multiply 1.60 * Sediff • Does the raw score change between T2 and T1 exceed the confidence interval? • If so, it represents change that exceeds the base rate expected based on measurement error • Thus, clinically meaningful change has occurred • If not, then the change is consistent with the base rate expected based on measurement error • Thus, no clinically meaningful change has occurred
Reliable Change Index Scores—An Example • Ferguson, Robinson, & Splaine (2002) • SF-36 in 200 patients who had undergone a Coronary Artery Bypass Grafting (CABG) surgery • SF-36 contains 8 scales • Physical Functioning • Role Functioning Physical • Bodily Pain • General Health • Vitality • Social Functioning • Role Functioning-Emotional • Mental Health
Reliable Change Index Scores—An Example • Ferguson, Robinson, & Splaine (2002) • Physical Functioning • Reliability=.93 (from normative sample of 2474) • Mean of normative sample=84.15 • SD of normative sample=23.28 • SEdiff=(2(SD(1-rxx)1/2)2)1/2 • SEdiff=(2(23.28(1-.93)1/2 ) 2))1/2=9.85 • 95% CI: (SEdiff)*1.96=19.32 • T1 Mean=40.97 • T2 Mean=63.39 • Mean Diff=22.42 • The mean difference exceeds 19.32 • Thus, clinically meaningful change has occurred as a result of surgery
Reliable Change Index Scores—An Example • Ferguson, Robinson, & Splaine (2002) • Mental Health • Reliability=..84 (from normative sample of 2474) • Mean of normative sample=75.01 • SD of normative sample=21.40 • SEdiff=(2(SD(1-rxx)1/2)2)1/2 • SEdiff=(2(21.40(1-..84)1/2 ) 2))1/2=10.92 • 95% CI: (SEdiff)*1.96=21.40 • T1 Mean=72.08 • T2 Mean=71.84 • Mean Diff=-0.24 • The mean difference fails to exceed 21.40 • Thus, no clinically meaningful change has occurred as a result of surgery
Standardized Regression Based Change Scores • Elaborated by Charter (1996) • Used primarily for maximal performance tests • The RCI of Jacobsen and Truax is used for typical performance tests • It assumes that errors between test scores at baseline and time 2 are uncorrelated • This assumption is untenable in maximal performance tests because of practice effects
Standardized Regression Based Change Scores • Based on the standard error of prediction • SEpred=SDY2((1-rY1Y22)1/2) • The SE reflects the sampling distribution of predicted scores • It implies the range of scores that might be expected at time two that may be expected from the baseline score and prediction error • This method requires you to compute the estimated true score • Y2True=M+((rY1Y2)(Y1-M)) • The T2 score is prone to error, and this formula permits an unbiased estimate of the true score • The SEpred is used to compute a confidence interval around the estimated true score
Standard Normal Curve—Distribution of Standard Error of Prediction Around Estimated True Score
Standardized Regression Based Change Scores--Method • To use the SRB, you must compute the estimated true T2 score • Compute the confidence interval around this estimated true T2 score • For 95% confidence, you multiply 1.96 * SEpred • For 90% confidence, you multiply 1.60 * SEpred • Does the obtained T2 score fall outside the confidence interval around the estimated true score for T2? • If so, it represents change that exceeds the base rate expected based on measurement error, regression to the mean, and practice • Thus, clinically meaningful change has occurred • If not, then the change is consistent with the base rate expected based on measurement error, practice, and regression to the mean • Thus, no clinically meaningful change has occurred
Standardized Regression Based Change Scores--An Example • Basso, Carona, Lowery, & Axelrod (2002) • WAIS-III re-tested in a group of control subjects over a 3-6 month interval • FSIQ • Test-Retest Reliability=.90 • T1 Mean T1=109.4 (11.6) • T2 Mean T2=115.0 (12.1) • SEpred=SDY2((1-rY1Y22)1/2) • SEpred=(12.1((1-.902) 1/2))=5.29 • 95% CI: (SEdiff)*1.96=10.36 • Mean Diff=5.60 • The mean difference fails to exceed the 95% CI • No individual had a score exceeding the 95% CI • To apply the SRB, the T2 True Score is estimated • If the obtained score falls within the CI around the T2 True score, then no clinically meaningful change has occurred
Standardized Regression Based Change Scores--An Example • Basso, Carona, Lowery, & Axelrod (2002) • An example application: • T1 obtained score=104 • T2 obtained score=116 • Estimated True T2 Score • YTrue=M+((rY1Y2)(Y1-M)) • YTrue=100+(.90)(104-100)=103.6 • 116 exceeds 10.36 points from 103.6 • Thus, meaningful change has occurred
Standardized Regression Based Change Scores--An Example • Basso, Carona, Lowery, & Axelrod (2002) • An example application: • T1 obtained score=103 • T2 obtained score=106 • Estimated True T2 Score • YTrue=M+((rY1Y2)(Y1-M)) • YTrue=100+(.90)(106-100)=105.4 • 105 falls within 10.36 points of 106 • Thus, no meaningful change has occurred