Using COS Data to Inform Program Improvement at All Levels

1. Using COS Data to Inform Program Improvement at All Levels Every day, we are honored to take action that inspires the world to discover, love and nurture the greatness in all children.

2. Overview of modules • Module 1: Setting the Stage: Global Child Outcomes and the SFP • Module 2: Learning More About the SFP and COS Rating • Module 3: Essential Knowledge for Completing the SFP and COS Rating • Module 4: Guidance for Completing the SFP and COS Rating Within the Team Process • Module 5: Understanding Age-expected Child Development, Developmental Trajectories, and Progress • Module 6: Using COS Data to Inform Program Improvement: Part 1–Essentials for Providers Part 2–Essentials for Administrators Kids' Potential, Our Purpose

3. Module 6 – Part I Using COS Data to Inform Program Improvement: Essentials for Providers Kids' Potential, Our Purpose

4. The focus of this module: • Using COS data for program improvement • Revisit how the COS data from the SFP fits into continuous program improvement cycle • Describe questions and approaches for looking at the COS data that the SFP produces • Examine different types of data and some basic reports in the DMS so we can consider how to use this information for: • quality assurance and • program improvement Kids' Potential, Our Purpose

5. Why gather child outcomes information? • To provide information for families who as consumers have the right to know if their child’s program is effective • To identify strengths and weaknesses of programs and use the information to improve programs • To provide information for policymakers to justify future funding • Because the federal government requires it Kids' Potential, Our Purpose

6. SFP Information • SFP gathered at the individual level • Process is valuable • Quantifies the impact of early intervention services • Use information mostly at aggregate level • Review combined information from many children to inform the program, region, state, and national levels • Developmental gains achieved by children at the Local Lead Agency level are reflected in the state data Kids' Potential, Our Purpose

7. Everyone has a role in using COS data Kids' Potential, Our Purpose

8. Uses for SFP Information • Individual Program or LLA level: • Look at patterns of child progress within a program or LLA • Provides a sense of program or LLA’s effectiveness and strengths and weaknesses for improvement • Across LLA’s Statewide: • Look at patterns of progress across LLA’s • May inform policy makers, technical assistance providers, and to justify future funding • Overall Statewide: • Describe overall pattern of progress for all children in ESIT in WA • Provides required data about the difference ESIT is making for the federal government Kids' Potential, Our Purpose

9. Reports for state use • Quality checks and quality assurance • Information for program improvement COS Data from the SFP Informs Kids' Potential, Our Purpose

10. Continuous Program Improvement Reflect Are we where we want to be? What do these data tell us? Is there a problem? Why is it happening? Is it working? What should be done? Check (Collect and analyze data) Plan (vision) What actions will improve the program and help it best support all children and families? Implement Is it being done?

11. Knowledge Check 1. True or False: Checking COS data quality across team members is enough because these data are mainly used by the child’s IFSP team and family for their reference. 2. True or False: Working with COS data is mainly something that administrators do.

12. Knowledge Check 1. True or False: Checking COS data quality across team members is enough because these data are mainly used by the child’s IFSP team and family for their reference. Ensuring high quality COS data matters because COS ratings are the best way to look at how children are functioning at entry and exit from ESIT within the whole group of children in particular programs, regions, or statewide in ESIT. 2. True or False: Working with COS data is mainly something that administrators do. Everyone has a role in working with COS data – from gathering it, to checking it for quality to asking questions, to interpreting the information produced and thinking through appropriate action steps for program improvement. What your role is depends on your involvement within the organization and your program’s needs and focus.

13. Using data for program improvement = EIA 13 Evidence Inference Action

14. Evidence • Evidence refers to the numbers. For example, in the area of social-emotional skills: • 43% of children statewide entered ESIT fully at age expectations (with an entry rating of 7) • 58% of children exiting ESIT statewide showed substantial increases in their rate of growth after participating in the program. • The numbers are not debatable 14

15. Inferences What can you conclude from these numbers? How do you interpret the numbers? What other information will help us understand these numbers? Is this good news? Bad news? Hard to interpret? Why do the data look this way? What does it mean? 15

16. Inference 16 Inference is debatable Reasonable people can reach different conclusions Stakeholders can help with putting meaning on the numbers

17. Discussing Inferences • It is critical to engage stakeholders at all levels in discussions • Examples of stakeholders and opportunities to incorporate data Kids' Potential, Our Purpose

18. Reflection: Discussing Inferences from Data Take a moment to jot some notes to yourself. • How comfortable are you with reviewing data and making inferences about the data you have? • What would you need to increase your comfort making and discussing inferences with others? • Who else might be a part of these discussions? People in your program/LLA? Other stakeholders? • What kinds of questions might you bring to these discussions? Kids' Potential, Our Purpose

19. Discussing Inferences • Data-driven discussions can take many forms: • incorporate snipets of data as background in a bigger conversation, or • focus a whole meeting on in-depth review of data • Questions and steps: • Have someone fully explain the data. Do you understand what the numbers are? • React: Is it surprising? Expected? Does it make sense? Why? Or, why not? • What could account for the data presented? • What else do you want to know?

20. Kinds of Inferences 20 • Two kinds of inferences: • Early on, inference about the quality or consistency of the data • Later, inferences about root causes or conclusions showing how well a specific practice is being implemented or how effective it is with a group

21. Questions to ask about data • Why does the data look this way? What caused this? • Does it make sense? • Am I surprised? • Do I believe the data? • Believe some of the data? • All of the data? • Are these data reasonable? • What do these data tell us about the program? • Do I need more information or to hear from other perspectives to figure out what these data mean?

22. Action 22 Given the inference from the numbers…. What should be done? Identify specific recommendations or action steps Action can be debatable – and often is… Another role for stakeholders

23. Action 23 • Two kinds of actions: • Early on, focus on improving the quality of the data • Later, as the quality of data improves, actions focus on program improvement strategies

24. Take Home Message Evidence Inferences about program improvement issues Inferences about data quality issues Actions: If data quality is believable, then target actions for improvement areas identified Actions: Focus on quality assurance and improving data quality Kids' Potential, Our Purpose

25. Aside about Quality Assurance Kids' Potential, Our Purpose

26. Starting point… USE your data…. Kids' Potential, Our Purpose

27. Looking at Data on Groups of Children

28. Kids' Potential, Our Purpose

29. Kids' Potential, Our Purpose

30. Kids' Potential, Our Purpose

31. Example Entry COS ratings for • all children entering ESIT • served by all agencies • in the “Strong County” Service Area Kids' Potential, Our Purpose

32. Numbers = evidence • How well do the numbers shown represent the group? • What else do you want to know? Kids' Potential, Our Purpose

33. Kids' Potential, Our Purpose

34. Entry COS rating percentages“Strong County” Service Area Kids' Potential, Our Purpose

35. Entry COS Ratings • Evidence… the numbers, the patterns • Inference… the reason, the explanation • Action…what to do or change given the inference Kids' Potential, Our Purpose

36. Exit COS Ratings for a Group • Reactions? Surprising? • Why do they look this way? What might be causing it? • What do the data mean to you? • What implications does this have for action? Kids' Potential, Our Purpose

37. Practice using evidence, inference, action steps Kids' Potential, Our Purpose

38. Examples of ways providers might use outcomes data… • Review data in the DMS for children on your caseload. Does it match your expectations? Why or why not? • Bring data to share at your next assessment team or IFSP team meeting. Spend a few moments reviewing results, reacting to the data, and inferring what it means. • Bring data to your next supervision meeting and consider possible implications for new things to try or changes in practice. Kids' Potential, Our Purpose

39. Reflection • What data are you most interested in beginning to use? • What supports do you need to be able to use your data more effectively? Who can you talk with to access that support? Kids' Potential, Our Purpose

40. Knowledge Check • Multiple Choice: When thinking about evidence, inference, action steps for working with data, the evidence represents: a. Articles or literature supporting specific practices. b. The numbers themselves, the data being reviewed. c. The section of the law related to the issue. d. The formula used to compute the data. 2. True or False: Because inferences are based on evidence, only one inference is appropriate or valid.

41. Knowledge Check • Multiple Choice: When thinking about evidence, inference, action steps for working with data, the evidence represents: a. Articles or literature supporting specific practices. b. The numbers themselves, the data being reviewed. c. The section of the law related to the issue. d. The formula used to compute the data. 2. True or False: Because inferences are based on evidence, only one inference is appropriate or valid.

42. Resources and Support Check out the following websites: • http://ectacenter.org/eco/pages/quality_assurance.asp • http://ectacenter.org/eco/pages/training_resources.asp • http://ectacenter.org/eco/assets/pdfs/TrainingandTAResourcesMarch2014.pdf • http://ectacenter.org/eco/pages/selflearning.asp • http://ectacenter.org/eco/pages/training_activities.asp Kids' Potential, Our Purpose

43. Additional information – ESIT Website under“Information for Providers” link on the ESIT homepage Child and Family Outcomes heading at: http://www.del.wa.gov/development/esit/training.aspx Kids' Potential, Our Purpose

44. Overview of modules • Module 1: Setting the Stage: Global Child Outcomes and the SFP • Module 2: Learning More About the SFP and COS Rating • Module 3: Essential Knowledge for Completing the SFP and COS Rating • Module 4: Guidance for Completing the SFP and COS Rating Within the Team Process • Module 5: Understanding Age-expected Child Development, Developmental Trajectories, and Progress • Module 6: Using COS Data to Inform Program Improvement: Part 1–Essentials for Providers Part 2–Essentials for Administrators Kids' Potential, Our Purpose

45. For more information contact your LLA or Program Consultant http://www.del.wa.gov/publications/esit/docs/ContactsDirectory.pdf