Were Prior Mild TBI Meta-Analytic Results Refuted or Replicated?

1. Pertab, James, & Bigler (2009): Limitations of Mild Traumatic Brain Injury Meta-Analysis, BI, 23, 498-508 Were Prior Mild TBI Meta-Analytic Results Refuted or Replicated? Martin L. Rohling, Ph.D. Laurence Binder, Ph.D. U. of South Alabama Oregon Health Sciences Glenn Larrabee, Ph.D. Danielle Ploetz, M.S. Private Practice U. of South Alabama

2. Pertab, J. L., James, K. M., & Bigler, E. D. (2009). Limitations of mild traumatic brain injury meta-analyses. Brain Injury, 23, 498–508. This presentation is a rebuttal to the a paper published in Brain Injury. Rohling et al. Rebuttal to Pertab et al. (2009)

3. Quotes from the introductory section of Pertab, James, & Bigler’s (2009) paper. • Mild traumatic brain injury (mTBI) is one of the most frequently studied topics in clinical neuropsychology and may also be one of the most controversial (p. 428). • The first, and most frequently cited, meta-analysis intending to clarify the effects of mTBI was conducted by Binder, Rohling, & Larrabee (1997). They found only small overall effect sizes and concluded that… ‘the size of the overall effect was unimpressive and suggestive of clinical non-significance’. Note: Binder et al. (1997) only analyzed outcomes for patients who were 3-months or longer post mTBI. Rohling et al. Rebuttal to Pertab et al. (2009)

4. What did Binder, Rohling, & Larrabee (1997) examine? 1. Are there residualdeficits in cognitive functioning that could be attributed to mTBI? 2. If there are not significant overall deficits cognitive deficits caused by mTBI, are there domain specific deficits associated with mTBI (e.g., attention) that might be a “canary in the mine?” Rohling et al. Rebuttal to Pertab et al. (2009)

5. What did Binder et al.’s find?Overall Results and Domain specific findings related to mTBI. Rohling et al. Rebuttal to Pertab et al. (2009)

6. What were Pertab, James, & Bigler’s study objectives? • Reanalyze data reported in 2 prior MAs on mTBI: Binder et al. (1997) & Frencham et al. (2005). • Specifically, focus on previously unanalyzed variables: • (1) mechanism of the injury • (2) diagnostic criteria • (3) assessment instruments utilized • (4) symptomatic vs. non-symptomatic samples • They claimed to have restrict their data analyses to that which was gathered 3-months or greater post mTBI. • Purpose: “…to clarify opposing conclusions in the mTBI literature…” Rohling et al. Rebuttal to Pertab et al. (2009)

7. What did Pertab, James, & Bigler (2009) find? “Of the 25 original studies, 18 reported neuropsychological variables in a form that enabled effect size calculation on the variables of interest for this investigation (see Table I)” (pp 502). Table I is on the next slide. Rohling et al. Rebuttal to Pertab et al. (2009)

8. Table I. Studies with analysable neuropsychological data, the number of comparisons and average effect size for each study. References listed in redare incorrectly matched. Corrected reference numbers are listed in blue. Rohling et al. Rebuttal to Pertab et al. (2009)

9. Table II. Assessment tools — weighted overall effect size 3 months and over, ranked — largest impairment first. Rohling et al. Rebuttal to Pertab et al. (2009)

10. How do these two findings make neuropsychological sense? • Scores on Trails B & List Learning were not affected by mTBI. • Scores on Verbal Paired Associates, Digit Span, & Coding were all negatively affected by mTBI. This suggests that the control groups for some of the included studies were not representative. Studies did not use trauma controls like Dikmen et al (1995) Rohling et al. Rebuttal to Pertab et al. (2009)

11. Rohling et al.’s (2010) Methods: • Acquired Pertab et al.’s (2009) data file • Two coders blindly recoded effect sizes for Binder et al. (1997) & Frencham et al. (2005) • Compared results to Pertab’s calculations • Corrected any questionable procedures • Inappropriate collapsing of ESs across time • Failure to conduct heterogeneity analyses • Failure to include all available cognitive variables • Failure to use inferential statistics to calculate ES Rohling et al. Rebuttal to Pertab et al. (2009)

12. Demographic & injury severity measures that were available within manuscripts Rohling et al. Rebuttal to Pertab et al. (2009)

13. Inter-rater reliability for individual effect sizes coded by Rohling & Pertab ES represents effect sizes coded by Rohling et al. High degree of inter-rater reliability on individual effect size calculations when both teams coded a data point. Rohling et al. Rebuttal to Pertab et al. (2009)

14. Inter-rater reliability for overall study effect sizes coded by Rohling & Pertab ES represents effect sizes coded by Rohling et al. Much lower inter-rater reliability was generated when overall summary ES’s were compared to one another. There is only 1 summary ES per sample. The summary ES is calculated by averaging all of a studies individual ES’s together. Rohling et al. Rebuttal to Pertab et al. (2009)

15. Rohling et al.’s Results • Coding errors were apparent in Pertab et al. • Most critical was the inclusion of data gathered prior to 3-month post mTBI into ESs presented in Table I. • When all errors were corrected, for those data gathered prior to the 3-month post mTBI cutoff, a weighted mean ES was generated: ES = -.33 (n = 32 samples) • By contrast, data gathered at 3-months or greater post mTBI, a weighted mean ES was generated: ES = -.07 (n = 16 samples) Rohling et al. Rebuttal to Pertab et al. (2009)

16. All cognitive measures divided into 3 groups based on Chronicity (i.e., time post injury) and analyzed with a fixed effects model. Rohling et al. Rebuttal to Pertab et al. (2009)

17. All cognitive measures divided into 3 groups based on Chronicity (i.e., time post injury) and analyzed with a random effects model. Rohling et al. Rebuttal to Pertab et al. (2009)

18. Cognitive domains for All Time Frames generated from a fixed effects model. Rohling et al. Rebuttal to Pertab et al. (2009)

19. All cognitive measures divided into 9 groups based on Chronicity and analyzed with a fixed effects model. * Two studies should have been excluded, as they did not meet the inclusion initial criteria as coming from an mTBI non-symptomatic patients. Another study in this subsample had effect sizes incorrectly calculated and the corrected ES’s were smaller than previous calculated. Rohling et al. Rebuttal to Pertab et al. (2009)

20. Rohling et al.’s Conclusions • When analyzed properly, results from Binder et al. (1997) and Frencham et al. (2005) were REPLICATED with an ES = -.07 to -.09 • Pertab et al.’s (2009) results were the product of coding errors, mistakes in statistical analyses, and poor written description of results. • Including < 3 mo. data with data => 3-mo. lead to confusion by many readers. • Pertab found no residual cognitive effects of mTBI that were different from that which had been found earlier. • There was no evidence in the data that a subgroup of mTBI patients remain more significantly impaired. Rohling et al. Rebuttal to Pertab et al. (2009)

21. No new data revealed by Pertab et al. on these noted variables • Mechanism of the injury Could not be analyzed – insufficient data • Diagnostic criteria Could not be analyzed – insufficient data • Assessment instruments used Questionable stability – outliers, small n • Symptomatic patients vs. non-symptomatic Could not be analyzed – insufficient data Rohling et al. Rebuttal to Pertab et al. (2009)