Meta-Analysis of Single Subject Designs

1. Meta-Analysis of Single Subject Designs William R. Shadish University of California, Merced Natasha Beretvas University of Texas, Austin Eden Nagler CUNY, The Graduate Center

2. Presentations • William Shadish: “Meta-Analysis of Single Subject Designs: An Overview” • Natasha Beretvas: “Effect Sizes for Meta-Analysis of Single Subject Designs” • Eden Nagler: “Analyzing Data from Small N Designs using Multilevel Models”

3. Single Subject Designs (SSDs) • A time series of observations on one person. • The repeated introduction, removal, and/or reintroduction of treatment. • Demonstration of causality through showing that the effect is present when the treatment is present, but absent otherwise. • Many different variations, for example:

4. A Hypothetical SSD

5. McClannahan et al., 1990 Mean Personal Appearance Score Home 1 No Feedback Feedback Home 2 Home 3 Sessions

6. Meta-Analysis of Single Subject Designs • IES awarded two grants on this topic: • Shadish & Rindskopf • Beretvas • A general interest in using single subject designs as a source of evidence for “Evidence Based Practice” • But methods for the synthesis of such designs are diverse, with no consensus about which method is best.

7. Shadish and Rindskopf Grant • Review the Literature and Create Annotated Bibliographies (DONE) • Past Meta-Analyses of Single Subject Designs • Methods for Meta-Analyses of Single Subject Designs • Identify Sample Data Sets for Meta-Analysis of Single Subject Designs (DONE) • Creation of Coding Protocols (IN PROCESS) • Analytic and Computer Simulation of Pertinent Statistics (YEARS 2 and 3)

8. Past Meta-Analyses of Single Subject Designs • The annotated bibliography is at: http://faculty.ucmerced.edu/wshadish/Past%20MA%20of%20SSDs.htm • We found 22 such meta-analyses • Published between 1985 and 2005, with increasing frequency in the last decade. • Using a wide variety of methods:

9. Methods for Meta-Analysis of Single Subject Designs • We located 33 scholarly works on the topic, which are posted as an annotated bibliography at http://faculty.ucmerced.edu/wshadish/Methods%20for%20MA%20of%20SSD.htm • They were published between 1971 and 2005 • As you can see, the frequency of this work has increased significantly over time:

11. Sample Data Sets • One small and one large data set: • 16 studies reporting SSD’s to improve reading. Annoted Bibliography is at: http://faculty.ucmerced.edu/wshadish/Reading%20Improvement.htm • We are currently extracting data from these • 86 studies reporting SSD’s for treatment of learning disabilities. Annoted Bibliography is at : http://faculty.ucmerced.edu/wshadish/Learning%20Disabilities.htm

12. Two Kinds of Data, Two Kinds of Problems • Multiple Studies, each of which uses a different SSD (the hard problem) • Different participants • Different dependent variables (though perhaps conceptually related) • Different (related) treatments • Different time scales • Different designs: AB, ABAB, ABC, etc. • Multiple SSDs that use the same design within one study on multiple participants • The easy problem • Probably analyze with HLM or LGM

13. Which Leads Nicely Into the Next Talks • Natasha Beretvas will talk about the first problem • Eden Nagler will talk about the second problem. • Then we will have time for open discussion.