Evidence Based Practice University of Utah

1. Evidence Based Practice University of Utah Presented by Will Backner December 2009 Training School Psychologists to be Experts in Evidence Based Practices for Tertiary Students with Serious Emotional Disturbance/Behavior Disorders US Office of Education 84.325K H325K080308

2. The Use of Single-Subject Research to Identify Evidence Based Practice in Special Education Council for Exceptional Children Horner, R., Carr, E., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The Use of Single-Subject Research to Identify Evidence Based Practice in Special Education. Exceptional Children, 71(2), 165-179.

3. Council for Exceptional Children (CEC) • CEC was first established in 1922 by school administrators and faculty at the Teachers College of Colombia University • Its purpose is to improve the success of children with disabilities and gifted/talented children • It is the largest professional organization in the country devoted to special education (SPED) • Focus is on teachers and administrators (applied research)

4. Single Subject Research • Methodology first operationalized over 40 years ago • Has been relevant for defining educational practices at the level of the individual learner • Provides a level of experimental rigor beyond that found in traditional case studies. • Over 45 professional journals report single subject research • Operant principles of behavior have benefitted most from single-subject methodology and been empirically demonstrated with single-subject experiments for over 70 years • Single-subject methodology can be used to evaluate diverse approaches to human behavior such as social learning theory, social psychology, and social work

5. Goals of the Article • Present the defining features of single-subject research methodology • Clarify the relevance of single-subject research methods for special education • Offer objective criteria for determining when single-subject research results are sufficient for documenting evidence-based practices • Goal is to clarify how single-subject research is used to establish knowledge within special education and define the empirical support needed to document evidence-based practices

6. Single Subject Research Methodology • Experimental, rather than correlational or descriptive • Documents casual relationships between independent and dependent variables • Uses between- and within-subject comparisons to control for threats to internal validity • Requires systematic replication to enhance external validity

7. Individual Participant is the Unit of Analysis • May only involve one participant but typically includes multiple participants in a single study • Each participant serves as his own control • Performance prior to intervention is compared to performance during and/or after intervention • Usually the participant is an individual, but it can be a group that generates a single score during a measurement period (e.g. a classroom)

8. Participant and Setting Description • Operational descriptions of the participants, setting, and the selection of participants is required • Another researcher should be able to use the descriptions to recruit similar participants and use similar settings • Global descriptions are usually insufficient

9. Dependent Variable • In educational research is usually some sort of observable behavior • Operationally defined- Need valid and consistent assessment and allow for replication of the assessment process • Measured repeatedly across controlled conditions to allow identification of performance patterns prior to intervention and comparison of performance across conditions • Recording is assessed for consistency by frequent monitoring of interobserver agreement • Variables are selected for their social significance. • Chosen because it is important for the individual participant, those who are in contact with the individual or society

10. Independent Variable • Typically the practice, intervention, or behavioral mechanism under investigation • Operationally defined to allow valid interpretation of results and accurate replication of procedures • Materials and actions are often documented • Variable is actively rather than passively manipulated to demonstrate experimental control • Fidelity of variable implementation fidelity is documented • Can be done by continuous direct measurement or of the IV or an equivalent

11. Baseline Comparison Condition • Performance during this condition is compared to the effects of the intervention condition • Similar to a treatment as usual condition in group design research • Requires measurement during this condition and a detailed description of the condition which is precise enough to allow for replication • Measurement should continue during baseline until the observed pattern of responding is consistent enough to allow prediction of future responding • Requires multiple data points without a substantial trend or a trend in the opposite direction predicted by the intervention. • A trend in the direction predicted by the intervention compromises the ability to document an effect.

12. Experimental Control • Demonstration of a functional relationship between manipulation of the independent variable and change in the dependent variable. • Demonstrated when the design documents three demonstrations of the experimental effect at three different points in time with a single participant or across different participants • Experimental effect is demonstrated when predicted change in the dependent variable covaries with manipulation of the independent variable. • Documentation of experimental control is demonstrated through introduction and withdrawal of the independent variable, the staggered introduction of the independent variable at different points in time, or the iterative manipulation of the independent variable across observation periods.

13. Analysis of Single-Subject Research • Single subject research can be interpreted with statistical analyses, but the traditional approach involves systematic visual comparison of responding within and across conditions of a study. • Visual analysis involves interpretation of the level, trend, and variability of performance occurring during baseline and intervention conditions • Also involves judging: • Immediacy of effects following the onset and/or withdrawal of the intervention • The proportion of data points in adjacent phases that overlap in level • The magnitude of changes in the dependent variable • The consistency of data patterns across multiple presentations of intervention and nonintervention conditions

14. Analysis Continued… • Integration of information from these assessments and comparisons is used to determine whether a functional relationship exists between the independent and dependent variable • A functional relationship is a documentation of an effect. It is compromised when: • There is a long latency between manipulation of the independent variable and change in the dependent variable • Mean changes across conditions are small and/or similar to changes within conditions • Trends do not conform to those predicted following introduction or manipulation of the independent variable • Meta-analysis can also be used to document comparative trends in a field

15. External Validity • Concern whether an effect documented by one study has relevance for participants, locations, materials, and behaviors beyond those defined in the study. • External validity of single-subject research is enhanced through replication of the effects across participant, conditions, or different measures of the dependent variable • Also improved if study includes multiple participants, settings, materials, or behaviors • Enhanced by using operational descriptions of parameters • Generality of an intervention is established through systematic replication of effects across multiple studies • External validity narrowed when selection and attrition bias limit the range of examples available for analysis

16. Social Validity • In education single subject research has been used to document interventions that are functionally related to change in socially important outcomes. • Social validity of single subject findings is enhanced by: • Selection of dependent variables that have high social importance • Demonstration that independent variables can be implemented with fidelity by teachers and parents in typical contexts across time • Demonstration that teachers and parents report the procedures to be acceptable, effective, feasible with available resources, and continue to use interventions after formal support is removed • Demonstration that the intervention produced an effect that met the clinical need • Single subject research can also be used to distinguish between efficacy research and effectiveness research

17. Research Questions Appropriate for Single Subject Research Methods • No research approach is appropriate for all research questions • Research questions appropriately addressed with single-subject methods include: • Examining casual or functional relations by examining the effects of the introduction or manipulation of an independent variable on change in one or more dependent variables • Focus on the effects that altering a component of a multicomponent independent variable has on one or more dependent variables • Focus on the relative effects of two or more independent variable manipulations on one or more dependent variables

18. Quality Indicators for Single-Subject Research • Description of Participants and Setting • Participants are described with sufficient detail to allow others to select individuals with similar characteristics. • The process for selecting participants is described with replicable precision. • Critical features of the physical setting are described with sufficient precision to allow replication. • Dependent Variable • Dependent variables are described with operational precision. • Each dependent variable is measured with a procedure that generates a quantifiable index. • Measurement of the dependent variable is valid and describes with replicable precision. • Dependent variables are measured repeatedly over time. • Data are collect on the reliability or interobserver agreement associated with each dependent variable, and IOA levels meet minimum standards (e.g., IOA=80%; Kappa=60%).

19. Quality Indicators Continued… • Independent Variable • Independent variable is described with replicable precision. • Independent variable is systematically manipulated and under the control of the experimenter. • Overt measurement of the fidelity of implementation for the independent variable is highly desirable. • Baseline • The majority of single-subject research studies will include a baseline phase that provides provides repeated measurement of a dependent variable and establishes a pattern of responding that can be used to predict the pattern of future performance, if introduction or manipulation of the independent variable did not occur. • Baseline conditions are described with replicable precision.

20. Quality Indicators Continued… • Experimental Control/Internal Validity • The design provides at least three demonstrations of experimental effect at three different points in time. • The design controls for common threats to internal validity. • The results document a pattern that demonstrates experimental control. • External Validity • Experimental effects are replicated across participants, settings, or materials to establish external validity. • Social Validity • The dependent variable is socially important. • The magnitude of change in the dependent variable resulting from the intervention is socially important. • Implementation of the independent variable is practical and cost effective. • Social validity is enhanced by implementation of the independent variable over extended time periods, by typical intervention agents, in typical physical and social contexts.

21. Importance of Single Subject Research Methods for Research in Special Education • Single-subject research focuses on the individual • Single-subject research allows detailed analysis of “nonresponders” as well as “responders.” • Single-subject research provides a practical methodology for testing educational and behavioral interventions. • Single subject research provides a practical research methodology for assessing experimental effects under typical educational conditions. • Single-subject research designs allow testing of conceptual theory. • Single-subject research methods are a cost-effective approach to identifying educational and behavioral interventions that are appropriate for large scale analysis.

22. Identification of EBP using Single-Subject Research • The practice is operationally defined. • Described with sufficient proficiency so others can replicate it with fidelity. • The context in which the practice to be used is defined. • Conditions where the practice should be used, individuals qualified to apply the practice, the population for whom the practice is expected to be effective, and the outcomes affected by the practice are defined. • The practice is implemented with fidelity. • Studies should provide adequate documentation that the practice was implemented with fidelity.

23. Identification of EBP Continued… • Results from single-subject research document the practice to be functionally related to change in dependent measures. • Studies should document a casual, or functional, relationship between use of the practice and change in a socially important dependent variable by controlling for the effects of extraneous variables. • The experimental effects are replicated across a sufficient number of studies, researchers, and participants to allow confidence in the findings. Considered Evidence-based if: • A minimum of 5 single-subject studies that meet minimally acceptable methodological criteria and document experimental control have been published in peer-reviewed journals. • The studies are conducted by at least 3 different researchers across at least 3 different geographical locations. • The 5 or more studies include a total of at least 20 participants.

24. Conclusion • Single-subject research offers powerful and useful methods for improving the practices that benefit individuals with disabilities. • Any systematic policy for promoting the development or dissemination of evidence-based practices in education should include single-subject research as an encouraged methodology.