1 / 138

Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery

Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery. Martin L. Rohling, Ph.D. Associate Professor Department of Psychology University of South Alabama. Clinical vs. Mechanical Diagnosis.

moke
Download Presentation

Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery Martin L. Rohling, Ph.D. Associate Professor Department of Psychology University of South Alabama

  2. Clinical vs. Mechanical Diagnosis • Much research has been conducted since Meehl (1954) found clinical judgment to be less accurate than mechanical or “actuarial” judgement • e.g., Dawes, Faust, & Meehl (1989); Filskov (1981); Garb (1989); Garb (1994); Garb (1998); Grove et al. (2000);Sawyer (1966); and Wedding & Faust (1989) • Such results influential in causing NPs to turn to different versions of the HRB (Russell, 1998). • “Batteries” have been defined as the method by which one can avoid the clinical errors highlighted by Meehl an others, using “actuarial” rules for diagnosis (Russell, 1995; Russell et al., 2005). Rohling - CCPN Orlando, FL

  3. Rohling’s Interpretive Method (RIM): Development History • Conducted several meta-analysis with Dr. Laurence Binder at the Portland, OR – VA • The last of these focused on the residual cognitive effects of mild head injury. • Binder, Rohling, & Larrabee (1997) • Binder et al. grouped effect sizes (ES) into domains of neuropsychological functioning based on factor analytic studies. • e.g., Leonberger, Nicks, Larrabee, & Goldfader (1992) Rohling - CCPN Orlando, FL

  4. RIM Generated fromMeta-Analytic Procedures • Meta-analysis (MA) combines effect sizes (ES) across samples assuming that they all sample the population M for the particular effect of interest. • Common method ES calculation is a standardized mean difference score (e.g., Glass’ delta). • delta = difference between con. & exp. group’s M’s divided by con. group’s SD. • delta analogous to Z score - linear equivalent of T score used in clinical neuropsychology Rohling - CCPN Orlando, FL

  5. RIM Generated fromMeta-Analytic Procedures • Binder et al. (1997) combined ES’s generated from various tests into cognitive domains. • Why not similarly combine ES’s, or T scores, from a single patient into cognitive domains in the same way that it is accomplished in MA. • Each test score is treated as a ES that reflects the individual’s ability within a domain. • ES can be combined based on homogeneity of variance, so as to avoid combining apples and oranges. Rohling - CCPN Orlando, FL

  6. Introduction to the RIM Analysis • Flexible battery (multiple measure) use: • Is the most frequently cited model of assessment among neuropsychologists. • Only 7% of neuropsychologists use a fixed battery (Rabin et al, 2006, ACN). • Regarding the suitability, practicality, and usefulness of any fixed battery: • “We know of no batteries that fully satisfy these criteria.” (Lezak, Howieson & Loring 2004, Neuropsych. Assess., 4th ed, p 648.) Rohling - CCPN Orlando, FL

  7. Advantages of Flexible Battery • Dynamic & responsive to clinician’s needs • Covers 1 or many domains • “Flexible”, can be adapted for each patient • Can “oversample” domains • Well suited for hypothesis-driven approach Rohling - CCPN Orlando, FL

  8. Potential Problems with aFlexible Battery • Inflated error rates • Multicollinearity • Weighting decision problems • Unknown veracity/reliability of sets of tasks • Human judgment errors Rohling - CCPN Orlando, FL

  9. Human Judgment Errors(Wedding & Faust, 1989, ACN) • Hindsight bias • Confirmatory bias • Overreliance on salient data • Under-utilization of base rates • Failure to take into account co-variation Rohling - CCPN Orlando, FL

  10. Potential Benefits withRohling’s Interpretive Method (RIM) • Judgment errors can threaten reliability & validity of multiple measure test batteries. • RIM was designed to reduce these effects. • Based on meta-analytic techniques. • Uses a linear combination of scores placed on a common metric. Rohling - CCPN Orlando, FL

  11. Potential Benefits of RIM • A strategy that produces summary results analogous to those generated in a fixed-battery approach (e.g., HII, GNDS, AIR). • Takes advantage of psychometric properties of same metric data, e.g., T Scores. Rohling - CCPN Orlando, FL

  12. Today’s Presentation - Intent • Present a set of procedures that allows for a quantitatively-based comparison of an overall battery of measures. • Non-specific to battery measures themselves. • Can be used for any individual patient. • Demonstrate importance and practicality of use of established statistical indices. • (e.g., alpha, beta, effect size). Rohling - CCPN Orlando, FL

  13. Today’s Intent (cont’d) • Present a data format for any set of measures to be inspected at: • Global level (OTBM) • Domain level (DTBM) • Test measure level (ITBM) • Present a series of calculations to assist in the generation of these indices. • Present Steps in conjunction with clinical judgment from an informed position. Rohling - CCPN Orlando, FL

  14. Common RIM Domains of Functioning • Symptom Validity (SV) Tests • Emotional / Personality (EP) Measures • Meta-Cognition, Pain, or other self-ratings • Estimated Premorbid General Ability (EPGA) • Test Battery Means • Overall (OTBM), Domain (DTBM), & Instrument (ITBM) • Cognitive Domains: • VC, PO, EF, AML, VML, AW, PS • Non-Cognitive Domains: • PM, LA, SP Rohling - CCPN Orlando, FL

  15. Sample RIM: Summary Table Rohling - CCPN Orlando, FL

  16. Sample RIM: Graphic Display Rohling - CCPN Orlando, FL

  17. Brief of RIM Steps: • There are 24 steps to the RIM process • 17 calculation steps: • Advice on design of the battery • Calculation of summary statistics • Generation of graphic displays • 7 interpretative steps. • Detail a systematic procedure for use of the statistical summary table and graphic displays to: • Assess and verify summary data. • Identify strengths/limitations of current data. • Obtain a reliable diagnosis. • Develop tx plans based on sound judgments. • We briefly review each step in just a moment. Rohling - CCPN Orlando, FL

  18. Support for the RIM Process • Rational support/reasoning: Reduce clinical judgment errors. • The RIM is a Process, not a program. • Rather, the RIM is a way of formalizing thinking & interpretation of individual case data. • This is operationalizing what many flexible battery clinicians are already doing in their head. Rohling - CCPN Orlando, FL

  19. Support for the RIM Process:Specific Advantages • Psychometric properties at level with fixed, co-normed batteries, without their limitations. • Flexibility of test selection. • Flexibility of theoretical view of cognition (domain structure) Rohling - CCPN Orlando, FL

  20. Support for the RIM Process:Specific Advantages • Quantitatively support your conclusions and interpretations • Statistical evaluation • Measure of confidence in findings • Measure of limitations of findings • Ability to present data at different levels of interpretation • Greater defensibility Rohling - CCPN Orlando, FL

  21. The RIM has a Set of Procedure or Specific Steps Rohling - CCPN Orlando, FL

  22. RIM Steps 1-4: Summary Data • Design & administer battery. • Use well standardized recently normed tests. • Estimate premorbid general ability. • Use Reading (WTAR), Regression (OPIE-III), & academic records (rank, SAT, ACT). • Convert test scores to a common metric. • We recommend T scores, but z or SS OK too. • Assign scores to domains. • Factor analysis to support assignment (Tulsky et al., 2003) Rohling - CCPN Orlando, FL

  23. RIM Steps 5-8: Summary Data • Calculate domain M, sd, & n. • Calculate test battery means (TBM). • Overall TBM – All scores, large N & high power. • Domain TBM – Avoids domain over weighting. • (e.g., attention & memory). • Instrument TBM – One score per norm sample. • Calculate p for heterogeneity. • Have you put “apples & oranges” together? • Determine categories of impairment. • Recommend using of Heaton et al. (2003). Rohling - CCPN Orlando, FL

  24. RIM Steps 9-12: Summary Data • Determine % of test impaired. • Analogous to Halstead Impairment Index • # scores below cutoff / total # of of scores • Calculate ES for all summary stats. • Use Cohen’s d = (Me – Mc) / SD pooled • Calculate confidence interval for stats. • 90% CI = 1.645 x SEM • Upper limit of performance for impair. • Look for overlap between 90% CI of EPGA (lower) & Summary Stats (upper) Rohling - CCPN Orlando, FL

  25. RIM Steps 13-17: Summary Data • Conduct one-sample t tests. • Use EPGA as reference point • Conduct a between-subjects ANOVA. • Looking for strengths & weaknesses • Conduct power analyses. • Only needed for those NS differences • Sort scores for visual inspection. • Graphically display summary statistics. Rohling - CCPN Orlando, FL

  26. RIM Steps 18-20: Interpretation • Assess battery validity. • Examine the Symptom Validity scores. • Caution in accepting low power results. • Look at heterogeneity of summary stats. • Normative sample unrepresentative of patient. • Scores assigned to wrong domain. • Inconsistent performance on construct measures. • Examine influence of psychopathology. • Examine scores for heterogeneity. • Check OTBM, DTBM, & ITBM impaired. Rohling - CCPN Orlando, FL

  27. RIM Steps 21-24: Interpretation • Examine strengths/weaknesses looking at: • Confidence intervals overlap. • Results from one-sample t tests. • Results of ANOVA. • %TI show differences otherwise not evident. • Determine if pattern existed premorbidly. • Examine non-cognitive domains. • Psychomotor, Lang/Aphasia, Sensory Percept • Explore Type II errors –need more tests? • Examine sorted T-scores • Look for patterns missed by summary stats. Rohling - CCPN Orlando, FL

  28. Age: 37 Handed: Left Race: Euro-American Sex: Female Ed: 14 years Occup: Nursing Marital: Sep. 10 yrs Living: Camper in parent’s backyard Reason for Referral: TBI in head-on boat accident. Propeller hit pt in right parietal-occipital lobe (LOC = 7 days; GCS = 3). Eval. to determine capacity for medical & financial decisions, parenting skills, occupational prognosis, & disability status. Significant emotional, behavioral, occupational, and social problems pre-TBI. RIM Sample Case 1: Obvious TBI Rohling - CCPN Orlando, FL

  29. RIM Sample Case 1: Obvious TBI Rohling - CCPN Orlando, FL

  30. RIM Sample Case 1: Obvious TBI Rohling - CCPN Orlando, FL

  31. TBI Dose Response CurvesDikmen ES’s Meyers’ T Scores Rohling - CCPN Orlando, FL

  32. Combined Dikmen & Meyers Estimates: ES, T, & Difference Rohling - CCPN Orlando, FL

  33. Return to Work Study: OTBM’s for 4 Groups of TBI Survivors Rohling - CCPN Orlando, FL

  34. RIM Sample Case 1: Obvious TBI Normal Distribution of T Scores Rohling - CCPN Orlando, FL

  35. Reason for Referral: 2 yrs dangerous work habits. Eval to see if atrial fib & Type II diabetes impairs cognition. Hospitalized “TIA-like” Sx. Admitted to problems for 20 yrs, cardiac dysrhythmia & bradycardia, pacemaker, blood sugar difficult to manage, & family Hx of heart disease & diabetes. Age: 55 Handed: Right Race: Euro-American Sex: Male Ed: 13 years Occup: Mechanic Marital: Married 20 yr Living: at home w/wife RIM Sample Case 2: Subtle Diabetes Rohling - CCPN Orlando, FL

  36. RIM Sample Case 2: Subtle Diabetes Rohling - CCPN Orlando, FL

  37. RIM Sample Case 2: Subtle Diabetes Rohling - CCPN Orlando, FL

  38. RIM Sample Case 2: Subtle Diabetes Normal Distribution of T Scores Rohling - CCPN Orlando, FL

  39. RIM Critiques: Concern 1 • The method of calculating the standard deviations (SDs) for summary statistics and domain scores is incorrect. • Since many of the remaining steps of the RIM depend on the use of these SDs, this error is magnified in the subsequent steps. • SDs statistically can not exceed 9.99 and are more likely to be around 6.4 Rohling - CCPN Orlando, FL

  40. Response 1: RIM Ms 4 Datasets Rohling - CCPN Orlando, FL

  41. Inter-Individual Ms & SDs Rohling - CCPN Orlando, FL

  42. Response 1: RIM SDs 4 Datasets Rohling - CCPN Orlando, FL

  43. Intra-Individual Ms & SDs Rohling - CCPN Orlando, FL

  44. RIM Critiques: Concern 2 • More false-positives then clinical judgment. • Palmer et al. (2004) expressed concern that • We failed to distinguish “statistical” from “clinical” significance. • This failure is a critical error that precludes the prudent clinician from using the RIM. Rohling - CCPN Orlando, FL

  45. Response 2: RIM vs. Manual Detecting Differences – Overall % Rohling - CCPN Orlando, FL

  46. Response 2: RIM vs. Manual Detecting Differences – ESs Rohling - CCPN Orlando, FL

  47. Response 2: RIM vs. ManualDetecting Differences Scores Rohling - CCPN Orlando, FL

  48. RIM Critiques: Concern 3 • Clinicians who use the RIM will: • Idiosyncratically assign scores to cognitive domains. • This will result in low inter-rater reliability in analysis & diagnosis. Rohling - CCPN Orlando, FL

  49. RIM Critiques: Concern 4 • Scores on domains are unit weighted, which introduces error. • Willson & Reynolds (2004) said scores load on multiple domains. Assignment to domains & weights depend on: • Battery of tests administered. • Patients whose test scores are being examined. Rohling - CCPN Orlando, FL

  50. Response 4: Cross-Valid. Unit Wts • Conducted 4 multiple reg. on 457 pts’ WAIS-R. • Split sample in ½ - assess shrinkage. • Regressed patients’ verbal subtests onto PIQ. • Generated ideal weights for the 1st ½ of sample. • Used wts to predict PIQs in the 2nd ½ of sample. • Pre-PIQs regressed on actual PIQs 2nd ½ sample. • Also, generated weights for the 2nd ½ of sample. • Use Pre-PIQ’s regress on actual PIQs 1st ½ sample. • Repeated, except performance subtests predict VIQ • split sample ½ & generate same statistics as before. Rohling - CCPN Orlando, FL

More Related