1 / 20

Moving from Single-Case to Conventional Group Intervention Designs, and Vice Versa

This article discusses the transition between single-case and conventional group intervention designs, exploring the credibility and validity of research in both approaches. It examines the impact on research funding and provides considerations for researchers.

eloisaa
Download Presentation

Moving from Single-Case to Conventional Group Intervention Designs, and Vice Versa

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Moving from Single-Case to Conventional Group Intervention Designs, and Vice Versa Joel R. Levin University of Arizona

  2. On Trials and Scales • Conventional (“group”) educational intervention research • Research credibility (and fundability) in the context of Levin and O’Donnell’s (1999) “stages of educational research inquiry” model: From “tryouts” to “trials” Levin, J. R., & O'Donnell, A. M. (1999). What to do about educational research's credibility gaps? Issues in Education: Contributions from Educational Psychology, 5, 177-229.

  3. Professional Disclaimer Let me state from the outset that I am an unapologetic, dyed-in-the-wool, tried-and-true conventional group intervention researcher who professes (to the extreme) the principles and practices of carefully controlled randomized experimental designs (i.e., “true” experiments, controlling for as many potential confounding variables as is superhumanly possible). Nonetheless, I have elected to play devil’s advocate in the arguments that follow…

  4. Today’s Statistics Qualifying Examination Questions (Adapted from Levin, 1993) Two federally funded researchers from different institutions conduct the same randomly sampled, randomized two-conditions study (intervention vs. control) under equally well-controlled conditions. Both researchers conduct the same statistical test and detect an intervention effect with exactly the same significance probability (p = .048). In addition, Researcher A reports an effect size of d = .0625 whereas Researcher B reports an effect size of d = 3.11. Levin, J. R. (1993). Statistical significance testing from three perspectives. Journal of Experimental Education, 61, 378‑382.

  5. Think Carefully Now… • Which researcher's findings are more credible (i.e., nonchance, or “real”)? • Which researcher's findings are more impressive (e.g., worth getting excited about)? • Assuming that you must recommend the work of only one of these researchers for federal funding, which of the two would you recommend? Why?

  6. Oh yes, in order to produce the results as stated: Researcher A (for whom p =.048 and d = .0625) included a total of 2,000 participants, with 1,000 in each experimental condition. In contrast, Researcher B (for whom p = .048 and d = 3.11) included a total of 4 participants, with 2 in each experimental condition.

  7. On Trials and Scales • Unconventional (single-case) intervention research: Randomized controlled trials (RCTs) on a different scale? • Validity considerations • External: random sampling of cases and contexts • Internal: randomized designs and control of potential confounding variables (including attention to attrition issues) • Statistical conclusion: valid analyses, outcome probabilities, and effect-size estimates

  8. On Trials and Scales • Other considerations • Target(s) of intervention • Mode of intervention • individually vs. group-administered • Nature, extent, and duration of assessment • visual “inspection” of cases, with possibility for idiographic insights • Resource requirements • Practical constraints

  9. Let’s run with the previous example a bit further... Two different intervention studies are proposed to a funding agency. Both studies are comparing a mathematics instructional intervention (Singapore Manipulatives System, SMS) with the standard method of instruction. A. Conventional group RCT study: 8 randomly selected first-grade classrooms, randomly assigned to two instructional conditions (SMS, standard) for 15 weeks of instruction; assessed on a pretest, midtest, and posttest using hierarchical linear modeling (HLM). B. Single-case “RCT” study: 4 randomly selected learning-disabled first-grade students, randomly assigned to both stagger positions and SMS start points in a “regulated randomization” multiple-baseline design, with assessments taken every week for 15 weeks and analyzed by a valid randomization statistical test.

  10. In addition, suppose it can be demonstrated that: (1) the major “intervention effect” research question is associated with respectable statistical power in each of the studies; but (2) only one of the studies can be funded. Which one should be funded, the conventional group study or the single-case study? On what basis?

  11. Bottom Line Who’s to say which is worth more to science and society: A pound of conventional group intervention randomized trials research? OR Sixteen ounces of unconventional single-case intervention randomized trials research?

  12. Summing Up • Credible (and fundable) single-case intervention research: Scaling up, scaling down, and fitting in? • Independent replications within and across studies • Single-case intervention studies embedded within group-intervention studies (“microexperiments”) • Levin, Lall, & Kratochwill’s (2011) adapted example Levin, J. R., Lall, V. F., & Kratochwill, T. R. (2011). Extensions of a versatile randomization test for assessing single-case intervention effects. Journal of School Psychology, 49, 55-79.

  13. Actual Randomized “Group” Intervention Study • Objective: Kratochwill et al.’s (2004) experimental investigation of a community-based program, Family and Schools Together (FAST), was designed to improve the behavioral and academic functioning of Native American students. • Design: In seven different schools, between 10 and 18 students were matched on the basis of several behavioral characteristics and randomly assigned to program and nonprogram conditions (thereby creating between 5 and 9 pairs per school), with teachers ostensibly “blind” to conditions. • One focal outcome measure of the study was on students’ arithmetic skills and so on different assessment occasions throughout the school year, all students were provided with several pages of arithmetic problems to solve. Kratochwill, T. R., McDonald, L., Levin, J. R., Young Bear-Tibbetts, H., & Demaray, M. K. (2004). Families and Schools Together: An experimental analysis of parent-mediated multi-family group program for American Indian children. Journal of School Psychology, 42, 359-383.

  14. What If?: Proposed Randomized Single-Case Intervention Microexperiment • What if some small number of pairs were selected to participate in a microexperiment to assess the differential effect of an instructional intervention on program and nonprogram students? • Suppose that on a single assessment occasion arithmetic problems were randomly distributed over 8 pages (representing 8 outcome observations). • Further suppose that students were required to complete the first p pages according to whatever basic arithmetic procedures they typically used. • Then, following a randomly selected intervention point (out of, say, 5 designated ones), suppose that some specialized arithmetic-skill instruction were provided to all students. • Finally, students were required to complete the remaining 8-p pages of problems using the new procedures they had just been taught.

  15. The differential effect of the instructional intervention can be evaluated by comparing program and nonprogram students’ performance changes from the first p pages to the subsequent 8-p pages.

  16. Statistical Properties Based on a Comparative Intervention Effectiveness Permutation Test • The differential effect of the instructional intervention can be evaluated by comparing program and nonprogram students’ performance changes from the first p pages to the subsequent 8-p pages. • With 8 outcome observations (i.e., 8 pages of arithmetic problems), 5 potential intervention start points, 7 paired program and nonprogram students, and a posited autocorrelation of .30, a researcher would have a 62% chance of detecting a differential instructional intervention effect amounting to a conventional Cohen’s d of 1.60 and a 79% chance of detecting an intervention effect of d = 2.00.* • Of course, these powers would increase with a greater number of pairs. *As was discussed in the previous session, determining the reasonable equivalences of conventional and single-case effect-size measures is a “work still in progress.”

  17. Summing Up Separate credibility standardsshould be applied for each type of intervention research, conventional group and single-case • Guidelines are now in effect for conventional group RCTs (Moher, Schulz, & Altman (2001) • Standards have recently been developed for single-case intervention research based on individual cases (Kratochwill et al., 2010) • the same needs to be done for single-case intervention research based on aggregate or “clustered” cases (e.g., small groups, classrooms, schools, communities) Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M & Shadish, W. R. (2010). Single-case designs technical documentation. Retrieved from the What Works Clearinghouse website: http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf Moher, D., Schulz, K. F., & Altman, D. G. (2001). The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. Annals of Internal Medicine, 134, 657–662.

More Related