Effect size measures for single-case designs: within-case parametric indices

1. Single-Case Intervention Research Training Institute Madison, WI - June, 2019 James E. Pustejovsky pusto@austin.utexas.edu Effect size measures for single-case designs: within-case parametric indices

2. Effect size • Broadly: a quantitative [index] of relations among variables(Hedges, 2008, p. 167). • In context of SCDs: a quantitative index describing the direction and magnitude of a functional relationship (i.e., effect of intervention on an outcome) in a way that allows for comparison across cases and studies (Pustejovsky & Ferron, 2017)

3. Why do we need effect sizes? “Reporting and interpreting effect sizes in the context of previously reported effects is essential to good research. It enables readers to evaluate the stability of results across samples, designs, and analyses. Reporting effect sizes also informs power analyses and meta-analyses needed in future research.” (Wilkinson & APA Task Force on Statistical Inference, 1999)

4. Characteristics of a good effect size(Lipsey & Wilson, 2001) • Readily interpretable. • Comparable across studies (and cases, for SCDs) that use different operational procedures. • Accompanied by a measure of sampling variability (i.e., a standard error/confidence interval). • Calculable from available data.

5. Comparability across studies and cases(single-case designs) • Imagine several single-case design studies investigating the same intervention, with similar participants, similar outcome construct. • What procedural factors might be different across these studies? • Ideally, effect size indices should not be strongly affected by such factors.

6. Parametric within-case effect size measures • Within-case = characterize magnitude of functional relationship separately for each case in a study. • Parametric = based on a model for the process that generated the data. • Advantage: clear separation between effect size parameter definition and how it is estimated. • Challenge: developing/assessing modeling assumptions.

7. Simplest Possible Scenario • Stable baseline and treatment phase (no trends) • Immediate level shift due to treatment • Independence of outcome measurements (Sorry Joel!)

8. Notation • Mean levels of outcome in each phase: μA, μB • Standard deviation of outcome in each phase: σA, σB • Variability around mean level of outcome for one case • Sample of • m observations in baseline phase, • n observations in treatment phase,

9. Three ways to describe a change in level • Raw difference in levels: μB– μA • Standardized difference in levels (within-case standardized mean difference): (μB– μA) / σA • Proportional change in levels μB/ μA

10. Within-case standardized mean difference • SMD is one of most commonly used effect sizes in between-groups research. • Gingerich (1984), Busk and Serlin (1992) proposed within-case SMD for single-case designs. • Parameter definition: • Difference in means, “standardized” by variability in baseline phase. • Standardization makes δscale-free • NOT equivalent to between-case SMD, because σA only represents within-case variation.

11. Estimating the within-case SMD • As originally proposed, estimate δ by replacing parameters with corresponding sample statistics: • d-estimator has a small-sample bias. A bias-corrected estimator is: • Approximate standard error of g (assuming independence):

12. Estimating the within-case SMD (continued) • If it is reasonable to assume that the SD of outcome is constant across phases (i.e., homogeneity of variance), then SMD can be estimated using pooled sample variance: • A bias-corrected estimator: • Approximate standard error of g (assuming independence):

13. Rodriguez & Anderson (2014). Integrating a social behavior intervention during small group academic instruction using a total group criterion intervention.

14. Rodriguez & Anderson example * Standardized mean difference estimates (d and g) are calculated using standard deviation of baseline phase only. Estimates and standard errors are based on assumption that outcome measurements are mutually independent.

15. Comments on within-case SMD • Appropriate for interval-scale outcome measures • Is variability of outcome measure constant (approximately) for different mean levels? • Within-case SMD involves scaling by test-retest unreliability. Measurement procedures of varying reliability will mean that SMD is not comparable across studies. • Scales with restricted ranges will • ES magnitude and statistical significance are affected by auto-correlation. • Maggin and colleagues (2011) proposed extensions to within-case SMD • Addresses bias due to serial dependence • Adjusts for time trends in baseline and/or intervention phases.

16. Proportional change in levels • Percentage/proportional change from baseline to intervention is a common, easily interpretable “informal” effect size measure (Campbell & Herzinger, 2010). • Occasional applications of percentage change in meta-analyses of SCDs (e.g., Campbell, 2003; Kahng, Iwata, & Lewin, 2002; Marquis et al., 2000). • The log response ratio (LRR) is a formal effect size measure that quantifies functional relationships in terms of proportionate change • Used as effect size for between-groups research in some fields (e.g., ecology, economics). • Pustejovsky (2015) argues for its use in single-case research with behavioral outcome measures.

17. Within-case log response ratio • LRR parameter:: • Appropriate for ratio-scaled outcomes (frequency counts, percentage duration). • Natural logarithm used to make range less restricted. • If treatment has zero effect, then μB / μA = 1 and ψ = 0. • Particularly appropriate for behavioral outcomes measured by direct observation (Pustejovsky, 2015). • Magnitude remains stable even when outcomes are measured using different procedures. • Under certain assumptions, also comparable across dimensional constructs. • Relationship to percentage change:

18. Estimating the within-case LRR (Pustejovsky, 2015) • Basic estimator: • This estimator will be biased if m or n is small • A bias-corrected estimator: • Approximate standard error of R2 (assuming independence):

19. Estimating the within-case LRR (continued) • Approximate confidence interval for ψ (assuming independence):where zα is 2-tailed critical value from standard normal distribution • Approximate confidence interval for % change (assuming independence):

20. Comments on within-case LRR • Serial dependence will affect SE and CI but not the effect size estimator itself • If there is positive auto-correlation, SE and CI will tend to be too small. • When applying to outcomes measured as proportions, need to be careful that direction of improvement is consistent (Pustejovsky, 2018). • E.g., need to re-code “% time on-task” if other cases or studies use “% time off-task” • Partial interval recording data presents further complications (Pustejovsky & Swan, 2015).

21. Rodriguez & Anderson example * Standard errors and confidence intervals are based on assumption that outcome measurements are mutually independent.

22. Dealing with time trends

23. Predictions at a focal follow-up time • : predicted level of the outcome at time Fif intervention never happens. • : predicted level of the outcome at time Fif intervention happens.

24. Defining effect sizes for afocal follow-up time • Define effect sizes as comparisons betweenand . • Within-case SMD at time F: • Within-case LRR at time F: Difference in levels at focal follow-up time Residual standard deviation

25. Non-linear models for gradual effects where Ui is cumulative number of treatment sessions

26. Non-linear models for gradual effects • Can be extended for ABAB/treatment reversal designs • Works with LRR and other parametric effect sizes • See Swan and Pustejovsky (2018) for further details • Web-app for effect size estimation: https://jepusto.shinyapps.io/gem-scd/

27. Meta-analysis across cases • Within-case ES estimates describe effects (FRs) for each case. • What if you want to characterize the overall pattern of findings across cases/participants in a study? • Simple fixed effects meta-analysis • Estimate an average effect size across cases you have observed. • SEs/CIs represent the uncertainty of estimates for the observed cases. • Random effects meta-analysis • Estimate an average effect size in a population of cases (similar to the observed cases), as well as heterogeneity of effects in the population. • SEs/CIs represent the uncertainty of estimates for the population.

28. Simple fixed-effect meta-analysis • M cases, with effect size estimates and standard errors • Average effect size estimate: • SE for the average effect size: • Approximate confidence interval:

29. Rodriguez & Anderson example * Standard errors and confidence intervals are based on assumption that outcome measurements are mutually independent.

30. References (1/2) Busk, P. L., & Serlin, R. C. (1992). Meta-analysis for single-case research. In T. R. Kratochwill & J. R. Levin (Eds.), Single-Case Research Design and Analysis: New Directions for Psychology and Education (pp. 187–212). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Campbell, J. M. (2003). Efficacy of behavioral interventions for reducing problem behavior in persons with autism: a quantitative synthesis of single-subject research. Research in Developmental Disabilities, 24(2), 120–138. doi:10.1016/S0891-4222(03)00014-3 Campbell, J. M., & Herzinger, C. V. (2010). Statistics and single subject research methodology. In D. L. Gast (Ed.), Single Subject Research Methodology in Behavioral Sciences (pp. 417–450). New York, NY: Routledge. Gingerich, W. J. (1984). Meta-analysis of applied time-series data. Journal of Applied Behavioral Science, 20(1), 71–79. doi:10.1177/002188638402000113 Hedges, L. V. (2008). What are effect sizes and why do we need them? Child Development Perspectives, 2(3), 167–171. https://doi.org/10.1111/j.1750-8606.2008.00060.x Kahng, S., Iwata, B. a, & Lewin, A. B. (2002). Behavioral treatment of self-injury, 1964 to 2000. American Journal of Mental Retardation : AJMR, 107(3), 212–221. doi:10.1352/0895-8017(2002)107<0212:BTOSIT>2.0.CO;2 Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-Analysis. Thousand Oaks, CA: Sage Publications, Inc. Maggin, D. M., Swaminathan, H., Rogers, H. J., O’Keeffe, B. V, Sugai, G., & Horner, R. H. (2011). A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology, 49(3), 301–321. doi:10.1016/j.jsp.2011.03.004

31. References (2/2) Marquis, J. G., Horner, R. H., Carr, E. G., Turnbull, A. P., Thompson, M., Behrens, G. A., … Doolabh, A. (2000). A meta-analysis of positive behavior support. In R. Gersten, E. P. Schiller, & S. Vaughan (Eds.), Contemporary Special Education Research: Syntheses of the Knowledge Base on Critical Instructional Issues (pp. 137–178). Mahwah, NJ: Lawrence Erlbaum Associates. Pustejovsky, J. E. (2015). Measurement-comparable effect sizes for single-case studies of free-operant behavior. Psychological Methods, 20(3), 342–359. doi:10.1037/met0000019 Pustejovsky, J. E., & Ferron, J. M. (2017). Research synthesis and meta-analysis of single-case designs. Handbook of Special Education: Second Edition. https://doi.org/10.4324/9781315517698 Pustejovsky, J. E., & Swan, D. M. (2015). Four methods for analyzing partial interval recording data, with application to single-case research. Multivariate Behavioral Research, 50(3), 365–380. doi:10.1080/00273171.2015.1014879 Rodriguez, B. J., & Anderson, C. M. (2014). Integrating a social behavior intervention during small group academic instruction using a total group criterion intervention. Journal of Positive Behavior Interventions, 16(4), 234–245. doi:10.1177/1098300713492858 Swan, D. M., & Pustejovsky, J. E. (2018). A gradual effects model for single-case designs. Multivariate Behavioral Research, forthcoming. doi:10.1080/00273171.2018.1466681 Wilkinson, L. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), 594–604.