Autocorrelation in Single-Subject Research and Statistical Analysis Strategies

1. Some Autocorrelation References Baer, D. M. (1988). An autocorrelated commentary on the need for a different debate. Behavioral Assessment, 10, 295-298. Busk, P. L., & Marascuilo, L. A. (1988). Autocorrelation in single-subject research: A counterargument to the myth of no autocorrelation. Behavioral Assessment, 10, 229-242. Crosbie, J. (1993). Interrupted time-series analysis with brief single-subject data. Journal of Consulting and Clinical Psychology, 61, 966-974. Huitema, B. E. (1985). Autocorrelation in applied behavior analysis: A myth. Behavioral Assessment, 7, 107-118. Huitema, B. E. (1988). Autocorrelation: 10 years of confusion. Behavioral Assessment, 10, 253-297. Parsonson, B. S., & Baer, D. M. (1992). The visual analysis of data, and current research into the stimuli controlling it. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis: New developments for psychology and education (pp. 15-40). Hillsdale, NJ: Eribaum. Riviello, C., & Beretvas, S. N. (2011). Detecting lag-one autocorrelation in interrupted time series experiments with small datasets. Unpublished manuscript, University of Texas, Austin. Shadish, W. R., & Sullivan, K. J. (in press).Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods. Shadish, W. R., Sullivan, K. J., Hedges, L. V., & Rindskopf, D. M. (2010, Oct.). A d-estimator for single-case designs. Paper presented at the Conference on Single-Case Intervention Research, Madison, WI. Sharpley, C. F., & Alavosius, M. P. (1988). Autocorrelation in behavioral data: An alternative perspective. Behavioral Assessment, 10, 243-251. Toothaker, L., E., Banz, M., Noble, C., Camp, J., & Davis, 0. (1983). N = I designs: The failure of ANOVA-based tests. Journal of Educational Statistics, 8, 289-309.

2. So Who Cares About Elephants Anyway? • Reflect back on our three initiating questions • Why • The hope for improved data-analysis objectivity and reliability, as well as more “valid” conclusions about intervention effects? • But always keep in mind the IOTT! • Whether • In light of the investigator’s purposes, is statistical analysis really necessary? • Which? • Statistical-conclusion validity issues • Acceptable Type I error control • Acceptable statistical power • The potential to yield informative confidence intervals and effect-size measures

3. Proposed Statistical Analysis Strategies •Standard parametric statistical methods (e.g., t test, analysis of variance, regression analysis, binomial test) • problems problems everywhere… • Time-series analysis (e.g., ARIMA models: Glass, Willson, & Gottman, 1975) • concepts and basics (McCleary & Welsh, 1992) • problems, and problems within problems (Crosbie, 1993) • current limitations • Adapted regression-based and HLM models (e.g., Beretvas & Chung, 2008; Kyse, Rindskopf, & Shadish, 2011) • potential advantages, but wait and see • current limitations • Nonparametric (permutation-based) analyses (e.g., Edgington & Onghena, 2007; Todman & Dugard, 2001) • potential advantages • current limitations • Editorial comment: Tradeoffs between statistical analysis elegance/complexity and parsimony/comprehensibility

4. Proposed Statistical Analysis Strategies References Beretvas, S. N., & Chung, H. (2008). An evaluation of modified R2-change effect size indices for single-subject experimental designs. Evidence-Based Communication Assessment and Intervention, 2:3, 129-128. Edgington, E. S., & Onghena, P. (2007). Randomization tests (4th ed.) Boca Raton, FL: Chapman & Hall/CRC. Glass, G. V., Willson, V. L., & Gottman, J. M. (1975). Design and analysis of time series experiments. Boulder, CO: University of Colorado Press. Kyse, E. N., Rindskopf, D. M., & Shadish, W. R. (2011). Analyzing data from single-case designs using multilevel models: A primer. Unpublished manuscript, University of California, Merced. McCleary, R., & Welsh, W. N. (1992). Philosophical and statistical foundations of time-series experiments. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis: New developments for psychology and education (pp. 41-91). Hillsdale, NJ: Erlbaum. Todman, J. B., & Dugard, P. (2001). Single-case and small-n experimental designs: A practical guide to randomization tests. New York: Erlbaum.

5. “True” Single-Case Applications and Classroom-Based Applications Differences in the two types of application • Definitely not the same species • Conceptual and methodological issues ─ potential confounders in each ─ number of time periods and number of observations per time period • Statistical issues ─consideration of the autocorrelation “elephant” in each, vis-à-vis statistical properties ─ this and other statistical-conclusion validity issues are currently being studied for different statistical-analysis strategies applied to various single-case designs