NTACT Evidence-Based and Promising Practices

1. NTACT Evidence-Based and Promising Practices

2. Agenda • PowerPoint • Why we care about evidence-based practices (EBPs) • Levels of Evidence • Criteria for the different levels of evidence • What this led to • How to use the resources we provide • Website tour

3. Why Do We Care About EBPs? • NTACT’s Purpose: • Assist State Education Agencies, Local Education Agencies, State VR Agencies, and VR service providers to implement evidence-based and promising practices ensuring students with disabilities, including those with significant disabilities, graduate prepared for success in postsecondary education and employment.

4. Why WeReally Care About EBPs? • When educators use practices that research have shown to be effective, student’s perform better • Cook, Tankersley, & Harjusola-Webb (2008)

5. NTACT’s Levels of Evidence • Refers to the amount and quality of research supporting a practice • A “practice” has three major components • What you do (i.e., the intervention/independent variable) • What improves (i.e., the target outcome/dependent variable) • With whom (i.e., population)

6. NTACT’s Levels of Evidence • NTACT has four • Evidence-Based Practices • Research-Based Practices • Promising Practices • Unestablished Practices • Quality and quantity to move up

7. NTACT’s vs. NSTTAC’s Levels of Evidence • Major Difference: • NSTTAC placed the emphasis primarily on the quality of research. • NTACT values quality, but also places a higher value on the amount of research (i.e., number of studies) supporting a practice. • No longer can a practice have a couple high quality studies and be considered an EBP

8. NTACT’s Process to Determine Level of Evidence • Search the literature for studies on the chosen practice • Sort based on the type of study (i.e., group experimental, single-case, correlational, qualitative). • Review studies for adherence to quality indicators • Count how many high quality and acceptable quality studies support the practice • Compare to the detailed criteria to determine the level of evidence

12. Quality Indicators • Set of criteria for different types of research that, if present, indicate that study is high quality • Group Experimental (Gersten et al., 2005) • Single-Case (Horner et al., 2005) • Correlational (Thompson et al., 2005) • Qualitative (Brantlinger et al., 2005; Trainor & Graue, 2014)

13. Example: Single-Case • Description of Participants and Settings • Participants are described with sufficient detail to allow others to select individuals with similar characteristics (e.g., age, gender, disability, diagnosis). • The process for selecting participants is described with replicable precision. • Critical features of the physical setting are described with sufficient precision to allow replication. • Dependent Variable • Dependent variables are described with operational precision. • Each dependent variable is measured with a procedure that generates a quantifiable index. • Measurement of the dependent variable is valid and described with replicable precision. • Dependent variables are measured repeatedly over time. • Data are collected on the reliability or interobserver agreement associated with each dependent variable, and IOA levels meet minimal standards (e.g., IOA = 80%; Kappa = 60%). • Independent Variable • Independent variable is described with replicable precision. • Independent variable is systematically manipulated and under the control of the experimenter. • Overt measurement of the fidelity of implementation for the independent variable is highly desirable.

14. Example: Single-Case • Baseline • The majority of single-subject research studies will include a baseline phase that provides repeated measurement of a dependent variable and establishes a pattern of responding that can be used to predict the pattern of future performance, if introduction or manipulation of the independent variable did not occur. • Baseline conditions are described with replicable precision. • Experimental Control/Internal Validity • The design provides at least three demonstrations of experimental effect at three different points in time. • The design controls for common threats to internal validity (e.g., permits elimination of rival hypotheses). • The results document a pattern that demonstrates experimental control. • External Validity • Experimental effects are replicated across participants, settings, or materials to establish external validity. • Social Validity • The dependent variable is socially important. • The magnitude of change in the dependent variable resulting from the intervention is socially important. • Implementation of the independent variable is practical and cost effective. • Social validity is enhanced by implementation of the independent variable over extended time periods, by typical intervention agents, in typical physical and social contexts.

15. What Did This Lead To? • 127 practices and 19 predictors (at last count) • Varying levels of evidence • Outcomes organized by relevant transition area • Education • Employment • Independent Living • Population • Stated in practice descriptions

16. Example From Our Effective Practices Matrix E

17. Where to Find these EBPs • Practices and predictors can be found on our website • Dawn will take you through that

18. How to Use • Our suggestions • Decide the outcome you are trying to affect • Look for interventions relevant to that outcome • Like on the NTACT website! • Start with those that have the highest level of evidence with your population • EBP  RBP  PP  Unestablished • Monitor the effects and change course if needed

19. Data Based Decision Making Still Matters • NTACT has identified a lot of practices and predictors • These still don’t cover everything educators are responsible for • In these cases it is critical to use data to evaluate effectiveness and to drive future educational decisions

20. Data Based Decision Making Still Matters • Also, practices demonstrated effective by research are more likely to be effective • But they are not guaranteedto be effective • In any big group (i.e., secondary students with disabilities), no matter how tightly you define the population, there’s always a few non-responders. • So even if there is an EBP for the outcome and population you are working with, data still needs to be used to ensure effectiveness and guide intervention

21. Questions? And contact info: NTACT • www.transitionta.org • ntactmail@uncc.edu Bradley Stevenson, bsteve23@uncc.edu Dawn Rowe, Ph.D., drowe3@uoregon.edu Valerie Mazzotti, Ph.D., vmazzott@uoregon.edu