Enhancing State Support: Effective Evaluation Strategies

1. A3 – Improving State Level Supports and Stakeholder Engagement through Effective EvaluationKim Gulbrandson, Justyn Poulos – Wisconsin RtI CenterKey Words: Applied Evaluation, Assessment

2. Objectives Understand connection between implementation and student outcomes. Identify considerations for deciding which implementation and outcome data to evaluate and report. Leave with examples of how states can make implementation and outcome connections and how to report to stakeholders. Gather examples of how to use the same evaluation data for state level continuous improvement.

3. What Are You Trying to Accomplish? Wisconsin RtI Center: Our mission is to build the capacity of Wisconsin schools to develop and sustain an equitable multi-level system of supports to ensure the success for all students.

4. The Fidelity - Outcome Relationship ”Higher Fidelity is correlated with better outcomes across a wide range of programs and practices.” (Fixsen/Blase) https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-NASHMPD15thConference-02-2005.pdf

5. Logic Model – What Data We Collect and Use and Why

6. Align Data Collection and Evaluation Across the Project Same data and logic used for both internal continuous improvement and external evaluation.

7. Evaluation Plan - Calendar Internal Continuous Improvement Monthly Evaluation Brief April Share Outcomes with DPI May-June Share Outcomes with SLT August Annual Report October Share Annual Report with SLT November Evaluation Brief Dec-Jan SLT = State Leadership Team DPI = Department of Public Instruction

8. Evaluation Briefs Align to Logic Audience: Schools and Districts

9. Annual Report Aligns to Logic Professional Learning–Assessing-Fidelity–Sustaining- Student Outcomes Audience: https://www.wisconsinrticenter.org/assets/files/resources/1513016537_2016-17%20Annual%20Report.pdf

10. Since 2009… Professional Learning • Of the 2214public schools in Wisconsin, 1803 (81%) have participated in professional learning offered by the Center • and 1547 (70%) have completed a full training in behavior, reading, and/or mathematics • Of trained schools, 1427 (92%) have self-assessed to measure their implementation. • Of assessing schools, 1072 (75%) reached fidelity or full implementation at any one level. Statewide Overview 2214 public schools

11. PBIS – Trained to Fidelity

12. Math – Trained to Fidelity

13. Reading – Trained to Fidelity

15. Overall Student Outcomesfrom DPI Report - Follows Same Logic

16. Overall Student Outcomes

17. Student Outcomes – Students with IEPs

18. Student Outcomes – Students with IEPs

19. Student Outcomes – English Learners

20. Student Outcomes – Students of Color

21. Student Outcomes – Students of Color

22. Lesson Learned When Looking at Outcomes – Factor in Length of Implementation

23. Make the Data Relatable for Stakeholders • ConnectEd

24. How We Engage Stakeholders in our Logic and Evaluation

25. The Data Has a Dual Purpose 1. Informs our support to schools/districts (our continuous improvement) - INTERNAL 2. Provides our data for evaluation, which impacts funding and political support for the center - EXTERNAL

26. Use Same Logic for Internal Continuous Improvement

27. Lessons Learned - Goal Establish buy-in during goal development and start with realistic goals.

28. Lessons Learned in Continuous Improvement - Assessing Smaller, more rapid indicators of progress for assessing versus an annual review.

29. Lesson Learned - Operationalize the differences between how schools and state differ in problem solving from this data.

30. Shared Practice PDSA Example – to Increase Assessing

31. Key Takeaways Link Internal and External Evaluation through your Logic Have an Evaluation Plan and Map it Out (Calendar) Start with Doable and Reasonable Goals