Neuropsychological evaluation of sports concussion/ tbi : Historical and psychometric considerations

1. Ruben J. Echemendia Neuropsychological evaluation of sports concussion/tbi:Historical and psychometric considerations FDA WorkshopAssuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington, DC: June, 2011

2. Disclosure Dr. Echemendia is a paid consultant to: • The National Hockey League • Major League Soccer • US Soccer Federation

3. Historical Background • Neuropsychological assessment has been a key component in evaluating/managing TBI for decades. • Neuropsychological evaluations have become a key tool in the evaluation of MTBI since mid 70’s. • Focus on functional vs. structural assessment. • NP assessment of sports concussions began in mid 80’s.

4. Background • Sports Laboratory Assessment Model • Pre-post injury evaluations • Paper & Pencil Assessment Batteries • Computerized Batteries

5. Psychometric Considerations • Validity vs. Reliability • Validity • Does the test measure what it is supposed to measure? • Construct Validity –Operationalization of the construct, measurement of the construct • Cognitive tests DO NOT measure concussion! • Assess cognitive domains that are believed to be affected by concussion • Convergent/Divergent Validity • Does the test correlate with other known measures of the construct? • Does the test not correlate with domains that it is not supposed to measure Face Validity • Face Validity • Does the test appear to be measuring what it is supposed to measure

6. Psychometric Considerations • Sensitivity vs. Specificity • Sensitivity: Is the measure “sensitive” in detecting the construct being measured? • Do concussed individuals score lower than non-concussed individuals? • What proportion of concussed individuals are identified by the test? Positive Predictive Power • What proportion of Non-concussed individuals are identified by the test? Negative Predictive Power

7. Abnormal NP Scores Normal NP Scores 150 0 Concussed 150 Hypothetical 2X2 Validity Matrix 0 150 150 Normal 300

8. Abnormal NP Scores Normal NP Scores 120/ 80% 30/ 20% Concussed 150 True Positive False Negative Hypothetical 2X2 Validity Matrix 60/ 40% 90/ 60% 150 Normal True Negative False Positive 300

9. Psychometric Considerations • Sensitivity vs. Specificity • Specificity: Is the measure “specifically” measuring the identified construct and not others? • Concussion vs ADHD, Depression, Anxiety, Fatigue, etc. • Ideal: 100% Sensitivity – 100% Specificity • Real: NOT! • Sensitivity vs Specificity trade-off • Typically as sensitivity increases, specificity decreases • The user must decide on what side to err.

10. ROC Curve

11. Psychometric Considerations • Reliability – Estimating Error • Can we expect to get the same results when the test is given…. • To the same person an infinite number of times? (Confidence Intervals) • To the same person under different circumstances? (Sideline vs. Office) • To the same person at different points in time? • To the same person by different test administrators? • In different/alternate forms?

12. Tests Used in Sports TBI….

13. Paper & Pencil Tests • Selected from tests already in use and validated for assessing MTBI • Test Requirements: • Short • Easily administered by non-NPs • Easily scored • Sensitive to changes in specified domain • Acceptable psychometrics • Alternate forms. • Assess several cognitive domains. • Typical Tests: HVLT. BVMT. Trails, Digit Span, SDMT, Verbal Fluency, LNS, Cancellation tests.

14. Paper & Pencil Tests Cont. • Advantages: • Direct observation of player behavior • Direct monitoring of effort/motivation & task performance • Superior assessment of memory • Broad normative db with multiple samples. • Disadvantages: • Labor intensive (Expensive) • Variability in standardized admin • Less reliable assessment of info proc speed. • Tests must be hand scored

15. Computerized Batteries • Two approaches: • Adapt existing P&P measures into computer platform • Measure multiple cognitive functions (memory, proc speed, reaction time, etc.) • Develop tests de novo for computers • Limited assessment of domains (largely RT, Info Proc Speed) • New Instruments are being developed rapidly • Existing Instruments: ImPACT, ANAM Sports, HeadMinders CRI, CNS Vital Signs, CogSport.

16. Computerized Tests Cont. • Advantages: • Can be used in groups (Less costly) • Standardized admininistration • Immediate scoring/data capture • Finer assessment of RT/Proc Speed (altho other problems) • Multiple languages • Theoretically infinite alternate forms 

17. Computerized Tests Cont. • Disadvantages: • Less complete assessment of memory • Loss of observational data • Less control of effort/motivation • Diminished ability to assess understanding/track task performance • Computer-specific measurement issues • Data loss

18. Hybrid Approaches • Combine strengths of both P&P and Computerized tests • Once adequate normative data have been established…. • Computer for baseline testing • Computer plus P&P post-injury

19. Sports MTBI specific issues Phase of Evaluation Baseline Acute – On Field Management Sub acute – Office/Training Room Return to Play/Work/School Long Term Follow-up

20. Sports MTBI specific issues • Baseline Testing – Is it necessary/helpful? • Theoretically designed to minimize error due to intra-individual comparison • Does it do this? • Does it increase sensitivity/specificity? • Does it introduce additional sources of error? • What about multiple post-injury tests? • How often should it be repeated? • Age dependent?

21. Sports MTBI specific issues • Test – Retest Reliability • Significant variability across studies • What test-retest intervals should be used? • Reliability estimates for multiple tests

22. Sports MTBI specific issues • Practice Effects • Content Practice vs. Procedural • Use of Alternate Forms • Reliability/Validity of Alternate Forms • Are practice effects bad? • How to control for / appropriately use practice effects? • RCI, regression based estimates

23. Sports MTBI specific issues • Motivation/Effort • Unique aspect of sports was the assumption that players are motivated to perform well. • Different motivation as different phases of assessment • Baseline vs. Post-injury • Individual vs. Group • Identification of poor effort • Test specific indices • Outlier identification

24. User Qualifications • Although not the responsibility of the FDA, clear delineation of user qualifications for administering and interpreting tests is critical. • Failure to do so may lead to scope of practice challenges, medico-legal quandaries and poor practice.

25. Thank you!