Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act

1. Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES 2013

2. The Logic of Single Subject Designs

3. Repeated measures • Individuals are their own controls • Baseline phase • Obtains a profile of variation absent the intervention • Allows identification of systematic patterns indicative of maturational effects, seasonality, history • Provides a basis of comparison for treatment phase • Treatment phase • Measurements taken during time the treatment is applied • Phases are compared to make inferences about treatment effect The Basics

4. Consistent measures over time • Intervention that can be described fully and implemented with fidelity • Systematic introduction of the intervention • Replication (looking for a functional relationship not an isolated incident of change) Overall Requirements

5. Observable, quantifiable target behavior (dependent variable) • Can be measured repeatedly • Can be measured with a high degree of inter-observer agreement • For training • Training could be expected to impact it • Impact would be relatively immediate Requirements for Dependent Variables

6. Increased identification of children subject to ICWA • Increase in timely/accurate completion of risk and safety assessment tools • Increase in use of SMART objectives in case plans • Increased presence of concurrent plans in case files Examples of CW Training related Dependent variables

7. History-another event occurring at the same time as the intervention that could affect the dependent variable • Maturation-normal developmental processes occurring over time that could explain the results • Are others but these are particularly relevant Threats to Validity with Repeated Measures Designs

8. Do the data represent a stable pattern or are they unpredictable? • Minimum of 3 separate, consecutive observations required per phase (TankersleyNHSTES 2012) • The more variable the data, the more data points are needed • Are several methods for representing background variability (e.g. putting a confidence interval around a phase mean) Stability of Data

9. Length of Baseline and Stability

10. Common Types of Single Subject Designs

11. Monitors the dependent variable during treatment • Shows trend but can’t make causal inference Types of Designs: “B Design”

12. Shows change from baseline to treatment • May allow causal inference, but doesn’t control for history Types of Designs: “AB Design”

13. Allows causal inference but only works where treatment can realistically be withdrawn Types of Designs: “ABAB” Design

14. Staged start for the intervention for different groups • Allows causal inference where intervention cannot be withdrawn Types of Designs: “Multiple Baseline”

15. Project Background

16. Tribally-driven collaborative effort between WDCF, MCWIC, and the 11 Wisconsin Tribes “Best Outcomes for Indian Children”

17. The Project is focused on the state-wide implementation of the Wisconsin Indian Child Welfare Act (WICWA), which became law in December 2009 • The WICWA is a codification into state law of the Federal Indian Child Welfare Act (ICWA), which became law in 1978. WICWAlaw

18. SHORT TERM • Train CW agencies on tribal child welfare practices • Modify DCF Tribal child welfare approaches • Incorporate WICWA requirements into court procedures and the legal process • Update Adm. Rules and program standards to integrate WICWA • Improve Tribal/State child welfare relationships • LONG TERM • Strengthen relationships b/w state, county, adoption agencies, state and tribal courts • Increase state wide understanding of the history and purposes of the acts in child welfare system • Increase identification of ICWA eligible children • Increase formal notice to tribes • Increase adherence to WICWA placement preferences Goals of project

19. Variety of disciplines involved in the child welfare continuum • Recommends policy and practice changes based on stakeholder input • Three working subcommittees: • Curriculum • Qualified Expert Witness • Active Efforts State Advisory Board

20. Legislative Branch – Codification • Judicial Branch • State Court Office - Children’s court Improvement Program • On going judicial training • Revised ICWA Court Forms • Wisconsin Public Defenders Association • Executive Branch • Department of Children and Families • Specific Programs and internal Departments Cross-systems integration

21. Specific drivers to effect system change in this project: • Leadership • Training • Coaching • Systems Intervention • Facilitative Administration • Decision Support Data Systems National Implementation Research Network (NIRN) Key Implementation Drivers(NIRN)

22. Advisory Board and stakeholder workgroups • New training on WICWA offered to all child welfare staff, supervisors, central office • Specialized legal training for attorneys, briefings for judges • Revised WDCF policies • Desk aids for case workers • Changes to eWiSACWIS system • Revisions to CQI system for review of ICWA cases “Pulling multiple levers”

23. Examples of outcomes: ICWA records generated

24. Examples of outcomes: ICWA identifications

25. Training is one of multiple systemic interventions, with overlapping implementation periods • Strong evidence that trainees are satisfied, are learning a lot, and are motivated/plan to use what they’ve learned on the job when they have opportunity • Emerging evidence of improved outcomes in eWiSACWISdata • Can we determine the impact of training over and above that of the new WICWA tab in eWiSACWIS? Summary of Project Overview

26. Analysis of Single Subject Data: Our findings

27. Visual • Look for obvious contrast between phases in • Level • Trend • Overlap • Variability • More and larger contrasts are evidence of importance of change • Immediacy is evidence of importance of change • Statistical • Apply statistical tests of significance to patterns Types of Analysis for Single Case Designs

28. Visual Analysis Statistical Analysis More complex; e.g. regression discontinuity models Higher power Less prone to human error and biases Statistical significance ≠ practical significance Require a long time series Must meet assumptions about independent distribution of residuals (autocorrelation) Advantages and Disadvantages • Simple; e.g. graphs and descriptive statistics • Differences that are obvious are more likely to be meaningful • Low power • Danger of confirmatory bias and over-interpretation of random variation • Low inter-rater reliability

29. Pros Cons Still suffer from risk of over-interpreting random fluctuations in a short time series Combined Graphical and Statistical Analysis • Graphical aids like trend lines and means can aid in interpretation and improve inter-rater agreement about change • Recommendations (Nugent 2010) • Use both mean referenced and and trend referenced representations of background variability to supplement interpretation

30. AB Design Visual Analysis Immediacy of Change

31. AB Design Level of Change: Visual Analysis Phase Means and Medians

32. AB Design Combined Method: Background Variability Relative to Mean; 2 SD method Nourbakhshand Ottenbacher (1994)

33. AB Design Combined Method: Percentage of Data Points Exceeding the Median (PEM) Ma (2006)

34. AB Design: Visual Analysis Trend Based

35. AB Design: Combined Method Trend Based

36. WICWA Training Evaluation: What we hoped to see

37. Combination Design A AB1 AB1B2 B1 B2

38. Data Considerations, Lessons Learned, and Recommendations

39. Need repeated measures over time • Need sufficient numbers at each time period • Definitions of the measures need to remain constant • Administrative data sources are both promising and challenging • Offer access to measurements of child and family outcomes over time • Not designed for research • Extracts needed for analysis and how they are drawn matters • Getting to the right variables can be like peeling an onion! Data Considerations

40. Unique Child Most Recent Completed Episode of Care Unique Child Earliest Removal in Study Period The Unit of Analysis: Unique Child

41. The Unit of Analysis: Snapshot (duplicated count)

42. What types of training applications might this work for? • How do we separate the contribution of training from other factors affecting implementation? • Is it even possible, feasible, or necessary to tease out the effects of multiple contemporaneous interventions? Questions for Discussion

43. Nugent, William R., (2010). Analyzing Single System Design Data. New York : Oxford University Press • Tankersley, Harjusola-Webb, and Landrum (2012) Using single-subject research to establish the evidence base of special education. Intervention in School and Clinic. 44(2). Pp. 83-90 Further Reading

44. Exercise

45. Discuss a training evaluation situation where single case methods might be appropriate • On your worksheet list • Your evaluation question • The type of single case design you would use • Data sources • Potential issues/pitfalls • Report out In groups