Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery

1. Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery Martin L. Rohling, Ph.D. Associate Professor Department of Psychology University of South Alabama

2. Clinical vs. Mechanical Diagnosis • Much research has been conducted since Meehl (1954) found clinical judgment to be less accurate than mechanical or “actuarial” judgement • e.g., Dawes, Faust, & Meehl (1989); Filskov (1981); Garb (1989); Garb (1994); Garb (1998); Grove et al. (2000);Sawyer (1966); and Wedding & Faust (1989) • Such results influential in causing NPs to turn to different versions of the HRB (Russell, 1998). • “Batteries” have been defined as the method by which one can avoid the clinical errors highlighted by Meehl an others, using “actuarial” rules for diagnosis (Russell, 1995; Russell et al., 2005). Rohling - CCPN Orlando, FL

3. Rohling’s Interpretive Method (RIM): Development History • Conducted several meta-analysis with Dr. Laurence Binder at the Portland, OR – VA • The last of these focused on the residual cognitive effects of mild head injury. • Binder, Rohling, & Larrabee (1997) • Binder et al. grouped effect sizes (ES) into domains of neuropsychological functioning based on factor analytic studies. • e.g., Leonberger, Nicks, Larrabee, & Goldfader (1992) Rohling - CCPN Orlando, FL

4. RIM Generated fromMeta-Analytic Procedures • Meta-analysis (MA) combines effect sizes (ES) across samples assuming that they all sample the population M for the particular effect of interest. • Common method ES calculation is a standardized mean difference score (e.g., Glass’ delta). • delta = difference between con. & exp. group’s M’s divided by con. group’s SD. • delta analogous to Z score - linear equivalent of T score used in clinical neuropsychology Rohling - CCPN Orlando, FL

5. RIM Generated fromMeta-Analytic Procedures • Binder et al. (1997) combined ES’s generated from various tests into cognitive domains. • Why not similarly combine ES’s, or T scores, from a single patient into cognitive domains in the same way that it is accomplished in MA. • Each test score is treated as a ES that reflects the individual’s ability within a domain. • ES can be combined based on homogeneity of variance, so as to avoid combining apples and oranges. Rohling - CCPN Orlando, FL

6. Introduction to the RIM Analysis • Flexible battery (multiple measure) use: • Is the most frequently cited model of assessment among neuropsychologists. • Only 7% of neuropsychologists use a fixed battery (Rabin et al, 2006, ACN). • Regarding the suitability, practicality, and usefulness of any fixed battery: • “We know of no batteries that fully satisfy these criteria.” (Lezak, Howieson & Loring 2004, Neuropsych. Assess., 4th ed, p 648.) Rohling - CCPN Orlando, FL

7. Advantages of Flexible Battery • Dynamic & responsive to clinician’s needs • Covers 1 or many domains • “Flexible”, can be adapted for each patient • Can “oversample” domains • Well suited for hypothesis-driven approach Rohling - CCPN Orlando, FL

8. Potential Problems with aFlexible Battery • Inflated error rates • Multicollinearity • Weighting decision problems • Unknown veracity/reliability of sets of tasks • Human judgment errors Rohling - CCPN Orlando, FL

9. Human Judgment Errors(Wedding & Faust, 1989, ACN) • Hindsight bias • Confirmatory bias • Overreliance on salient data • Under-utilization of base rates • Failure to take into account co-variation Rohling - CCPN Orlando, FL

10. Potential Benefits withRohling’s Interpretive Method (RIM) • Judgment errors can threaten reliability & validity of multiple measure test batteries. • RIM was designed to reduce these effects. • Based on meta-analytic techniques. • Uses a linear combination of scores placed on a common metric. Rohling - CCPN Orlando, FL

11. Potential Benefits of RIM • A strategy that produces summary results analogous to those generated in a fixed-battery approach (e.g., HII, GNDS, AIR). • Takes advantage of psychometric properties of same metric data, e.g., T Scores. Rohling - CCPN Orlando, FL

12. Today’s Presentation - Intent • Present a set of procedures that allows for a quantitatively-based comparison of an overall battery of measures. • Non-specific to battery measures themselves. • Can be used for any individual patient. • Demonstrate importance and practicality of use of established statistical indices. • (e.g., alpha, beta, effect size). Rohling - CCPN Orlando, FL

13. Today’s Intent (cont’d) • Present a data format for any set of measures to be inspected at: • Global level (OTBM) • Domain level (DTBM) • Test measure level (ITBM) • Present a series of calculations to assist in the generation of these indices. • Present Steps in conjunction with clinical judgment from an informed position. Rohling - CCPN Orlando, FL

14. Common RIM Domains of Functioning • Symptom Validity (SV) Tests • Emotional / Personality (EP) Measures • Meta-Cognition, Pain, or other self-ratings • Estimated Premorbid General Ability (EPGA) • Test Battery Means • Overall (OTBM), Domain (DTBM), & Instrument (ITBM) • Cognitive Domains: • VC, PO, EF, AML, VML, AW, PS • Non-Cognitive Domains: • PM, LA, SP Rohling - CCPN Orlando, FL

15. Sample RIM: Summary Table Rohling - CCPN Orlando, FL

16. Sample RIM: Graphic Display Rohling - CCPN Orlando, FL

17. Brief of RIM Steps: • There are 24 steps to the RIM process • 17 calculation steps: • Advice on design of the battery • Calculation of summary statistics • Generation of graphic displays • 7 interpretative steps. • Detail a systematic procedure for use of the statistical summary table and graphic displays to: • Assess and verify summary data. • Identify strengths/limitations of current data. • Obtain a reliable diagnosis. • Develop tx plans based on sound judgments. • We briefly review each step in just a moment. Rohling - CCPN Orlando, FL

18. Support for the RIM Process • Rational support/reasoning: Reduce clinical judgment errors. • The RIM is a Process, not a program. • Rather, the RIM is a way of formalizing thinking & interpretation of individual case data. • This is operationalizing what many flexible battery clinicians are already doing in their head. Rohling - CCPN Orlando, FL

19. Support for the RIM Process:Specific Advantages • Psychometric properties at level with fixed, co-normed batteries, without their limitations. • Flexibility of test selection. • Flexibility of theoretical view of cognition (domain structure) Rohling - CCPN Orlando, FL

20. Support for the RIM Process:Specific Advantages • Quantitatively support your conclusions and interpretations • Statistical evaluation • Measure of confidence in findings • Measure of limitations of findings • Ability to present data at different levels of interpretation • Greater defensibility Rohling - CCPN Orlando, FL

21. The RIM has a Set of Procedure or Specific Steps Rohling - CCPN Orlando, FL

22. RIM Steps 1-4: Summary Data • Design & administer battery. • Use well standardized recently normed tests. • Estimate premorbid general ability. • Use Reading (WTAR), Regression (OPIE-III), & academic records (rank, SAT, ACT). • Convert test scores to a common metric. • We recommend T scores, but z or SS OK too. • Assign scores to domains. • Factor analysis to support assignment (Tulsky et al., 2003) Rohling - CCPN Orlando, FL

23. RIM Steps 5-8: Summary Data • Calculate domain M, sd, & n. • Calculate test battery means (TBM). • Overall TBM – All scores, large N & high power. • Domain TBM – Avoids domain over weighting. • (e.g., attention & memory). • Instrument TBM – One score per norm sample. • Calculate p for heterogeneity. • Have you put “apples & oranges” together? • Determine categories of impairment. • Recommend using of Heaton et al. (2003). Rohling - CCPN Orlando, FL

24. RIM Steps 9-12: Summary Data • Determine % of test impaired. • Analogous to Halstead Impairment Index • # scores below cutoff / total # of of scores • Calculate ES for all summary stats. • Use Cohen’s d = (Me – Mc) / SD pooled • Calculate confidence interval for stats. • 90% CI = 1.645 x SEM • Upper limit of performance for impair. • Look for overlap between 90% CI of EPGA (lower) & Summary Stats (upper) Rohling - CCPN Orlando, FL

25. RIM Steps 13-17: Summary Data • Conduct one-sample t tests. • Use EPGA as reference point • Conduct a between-subjects ANOVA. • Looking for strengths & weaknesses • Conduct power analyses. • Only needed for those NS differences • Sort scores for visual inspection. • Graphically display summary statistics. Rohling - CCPN Orlando, FL

26. RIM Steps 18-20: Interpretation • Assess battery validity. • Examine the Symptom Validity scores. • Caution in accepting low power results. • Look at heterogeneity of summary stats. • Normative sample unrepresentative of patient. • Scores assigned to wrong domain. • Inconsistent performance on construct measures. • Examine influence of psychopathology. • Examine scores for heterogeneity. • Check OTBM, DTBM, & ITBM impaired. Rohling - CCPN Orlando, FL

27. RIM Steps 21-24: Interpretation • Examine strengths/weaknesses looking at: • Confidence intervals overlap. • Results from one-sample t tests. • Results of ANOVA. • %TI show differences otherwise not evident. • Determine if pattern existed premorbidly. • Examine non-cognitive domains. • Psychomotor, Lang/Aphasia, Sensory Percept • Explore Type II errors –need more tests? • Examine sorted T-scores • Look for patterns missed by summary stats. Rohling - CCPN Orlando, FL

28. Age: 37 Handed: Left Race: Euro-American Sex: Female Ed: 14 years Occup: Nursing Marital: Sep. 10 yrs Living: Camper in parent’s backyard Reason for Referral: TBI in head-on boat accident. Propeller hit pt in right parietal-occipital lobe (LOC = 7 days; GCS = 3). Eval. to determine capacity for medical & financial decisions, parenting skills, occupational prognosis, & disability status. Significant emotional, behavioral, occupational, and social problems pre-TBI. RIM Sample Case 1: Obvious TBI Rohling - CCPN Orlando, FL

29. RIM Sample Case 1: Obvious TBI Rohling - CCPN Orlando, FL

30. RIM Sample Case 1: Obvious TBI Rohling - CCPN Orlando, FL

31. TBI Dose Response CurvesDikmen ES’s Meyers’ T Scores Rohling - CCPN Orlando, FL

32. Combined Dikmen & Meyers Estimates: ES, T, & Difference Rohling - CCPN Orlando, FL

33. Return to Work Study: OTBM’s for 4 Groups of TBI Survivors Rohling - CCPN Orlando, FL

34. RIM Sample Case 1: Obvious TBI Normal Distribution of T Scores Rohling - CCPN Orlando, FL

35. Reason for Referral: 2 yrs dangerous work habits. Eval to see if atrial fib & Type II diabetes impairs cognition. Hospitalized “TIA-like” Sx. Admitted to problems for 20 yrs, cardiac dysrhythmia & bradycardia, pacemaker, blood sugar difficult to manage, & family Hx of heart disease & diabetes. Age: 55 Handed: Right Race: Euro-American Sex: Male Ed: 13 years Occup: Mechanic Marital: Married 20 yr Living: at home w/wife RIM Sample Case 2: Subtle Diabetes Rohling - CCPN Orlando, FL

36. RIM Sample Case 2: Subtle Diabetes Rohling - CCPN Orlando, FL

37. RIM Sample Case 2: Subtle Diabetes Rohling - CCPN Orlando, FL

38. RIM Sample Case 2: Subtle Diabetes Normal Distribution of T Scores Rohling - CCPN Orlando, FL

39. RIM Critiques: Concern 1 • The method of calculating the standard deviations (SDs) for summary statistics and domain scores is incorrect. • Since many of the remaining steps of the RIM depend on the use of these SDs, this error is magnified in the subsequent steps. • SDs statistically can not exceed 9.99 and are more likely to be around 6.4 Rohling - CCPN Orlando, FL

40. Response 1: RIM Ms 4 Datasets Rohling - CCPN Orlando, FL

41. Inter-Individual Ms & SDs Rohling - CCPN Orlando, FL

42. Response 1: RIM SDs 4 Datasets Rohling - CCPN Orlando, FL

43. Intra-Individual Ms & SDs Rohling - CCPN Orlando, FL

44. RIM Critiques: Concern 2 • More false-positives then clinical judgment. • Palmer et al. (2004) expressed concern that • We failed to distinguish “statistical” from “clinical” significance. • This failure is a critical error that precludes the prudent clinician from using the RIM. Rohling - CCPN Orlando, FL

45. Response 2: RIM vs. Manual Detecting Differences – Overall % Rohling - CCPN Orlando, FL

46. Response 2: RIM vs. Manual Detecting Differences – ESs Rohling - CCPN Orlando, FL

47. Response 2: RIM vs. ManualDetecting Differences Scores Rohling - CCPN Orlando, FL

48. RIM Critiques: Concern 3 • Clinicians who use the RIM will: • Idiosyncratically assign scores to cognitive domains. • This will result in low inter-rater reliability in analysis & diagnosis. Rohling - CCPN Orlando, FL

49. RIM Critiques: Concern 4 • Scores on domains are unit weighted, which introduces error. • Willson & Reynolds (2004) said scores load on multiple domains. Assignment to domains & weights depend on: • Battery of tests administered. • Patients whose test scores are being examined. Rohling - CCPN Orlando, FL

50. Response 4: Cross-Valid. Unit Wts • Conducted 4 multiple reg. on 457 pts’ WAIS-R. • Split sample in ½ - assess shrinkage. • Regressed patients’ verbal subtests onto PIQ. • Generated ideal weights for the 1st ½ of sample. • Used wts to predict PIQs in the 2nd ½ of sample. • Pre-PIQs regressed on actual PIQs 2nd ½ sample. • Also, generated weights for the 2nd ½ of sample. • Use Pre-PIQ’s regress on actual PIQs 1st ½ sample. • Repeated, except performance subtests predict VIQ • split sample ½ & generate same statistics as before. Rohling - CCPN Orlando, FL