Session Four Data as a Tool

1. Measuring Child and Family Outcomes Course Session Four Data as a Tool Anne Brager, MS, RN Program Supervisor Frederick County Infants and Toddlers Program \ Adapted from NECTAC Conference August 2007

2. Benefits of Using Data at Many Levels • States are currently undertaking the task of developing systems for collecting and reporting data on child and family outcomes. • Many states are planning to use this information for other purposes in addition to providing data to the federal government.

3. The Need for Good Data There are many reasons why a state would want good data on child and family outcomes. • Programs for young children with disabilities are supported by state funds as well as federal funds, and state policy-makers have the same responsibilities to ensure that limited resources are being used effectively. • In addition to addressing the need for accountability, state-level data on outcomes can be used to improve programs.

4. Program Improvement • States will be able to identify programs or regions that may not be doing as well as other programs and may need extra support to achieve better outcomes for children and families. • Data on outcomes can be used to identify an outcome area in which the state as a whole may be weak, for example, in helping children establish more positive social relationships. • Data on outcomes can be used to identify subgroups of children or families who might not be doing as well as other groups.

5. Program Improvement • With outcome data available to them, states will have the ability to identify where and for whom programs are the most successful and, conversely, where and for which families programs need to be improved. • Because many states plan to use outcome data for their own purposes, they are developing systems that will provide much more information or information at more frequent intervals than the limited information necessary to meet federal requirements.

6. The Need for Good Data What is the ultimate goal?

7. Goal for Children Active and successful participants now and in the future across a variety of settings and situations: • Have positive social relationships • Acquire and use knowledge and skills • Take appropriate actions to meet their needs

8. Goal for Families Enable families to provide appropriate care for their child and have resources they need to participate in community activities • Understand their child’s strengths, abilities, and special needs • Know their rights and advocate effectively for their children • Help their children develop and learn • Have support systems • Access desired services, programs, activities in their community

9. How does a state achieve these goals?

10. System for Producing Good Child and Family Outcomes Adequate funding Good outcomes for children and families High quality services and supports for children 0-5 and their families Good Federal policies and programs Good State policies and programs Good Local policies and programs Strong Leadership • Prof’l Development • Preservice • Inservice

11. Proposition: A healthy system of services and supports produces good outcomes for children and families

12. What Constitutes a Healthy System? Services and supports provided: • By qualified personnel • In a timely manner • Consistent with recommended practices, evidence-based practices, etc. • Transdisciplinary • Family-centered • Build on natural learning opportunities and everyday routines • Some parts of a healthy system are required by law; some parts constitute best practice

13. What else constitutes a healthy system? Creating a culture of accountability and shared responsibility • Willingness (even an eagerness) to regularly use data to examine how the system is functioning • Data are available to people who need them analyzed the way they are needed when they are needed. • Who needs access to data reports? • Local administrators? Practitioners? • You! • Data regularly are used to make decisions and take action • Data isn’t any good if it isn’t in a form that is helpful!

14. The Power of Outcome DataExtends to the Local Level Local program directors can answer exactly the same questions as the state with outcome data: • In what outcome areas is the program doing a good job? • Where does it need to improve? • For which subgroups of children or families is the program effective, and for which groups does it need to be more effective? • What topics should be emphasized in professional development? • Does the program need a new curriculum, or does it need to better implement the curriculum it is using?

15. Program Improvement • Many EI and ECSE programs are doing outstanding jobs, but some may not be. • Even good programs may not be equally good for all the families and children with whom they are working. Information on outcomes is a tool to identify program strengths and weaknesses. • The intent of looking at outcome data should not be to punish programs but to engage in an ongoing reflective cycle of program improvement with the goal of achieving the best possible outcomes for all children and families receiving services.

16. Outcome Data for Individualized Planning • Outcome data originate at the level of the individual child and family with, for example, an assessment of how a child is doing in one or more outcome areas. • Whereas state and local levels tend to want less detailed information, the rich information available through a good assessment process are extremely valuable for working with individual children and their families. • These data may inform individualized program planning, i.e., setting goals and selecting intervention strategies, followed by a review of progress some time later, with possible adjustments to the goals and strategies.

17. Accountability Outcomes are Different than IFSP Outcomes • The intent of IDEA was (and continues to be) that teams write plans individualized to the unique needs and strengths of the child and family.

18. Accountability Outcomes are Different than IFSP Outcomes • Providers who have been collecting data on the OSEP outcomes, however, report that the child outcomes provide a useful framework for discussing what should be included in the individualized plans. • The discussion of how the child is doing in each of the outcome areas can provide an introduction to the development of any individualized outcomes in the general area of Outcome 1, 2, or 3. • Similarly, providers have reported using the three family outcomes as a framework for discussing how the program might assist the family in enhancing the child’s development.

19. Aggregated Data • The distinction between individual data and aggregated data is important for understanding the use of outcome data. • Aggregated data refers to data for a group of children; these data have been summarized with a statistic such as a percentage or a mean score. • When the program director looks at the percentage of children who have made progress in a given outcome area, the director is looking at aggregated data.

20. Using Data • In some states, these data will be available at multiple levels (with proper safeguards to protect confidentiality) so that families, providers, directors, local administrators, and state administrators can ask questions about outcomes being achieved and act accordingly on what they are learning, given their level of responsibility in the system. • This means that a teacher or therapist can access data for children on their caseload, a program director can access data for the children in the program, and a state administrator receives reports with the statewide data aggregated in various ways.

21. FMA Findings Meanings Action

22. Understanding FMA • In order to effectively use data, one must have a clear understanding of the difference between findings, meaning and action.

23. Findings Can’t debate the numbers, the challenge is how you interpret them! • Findings are the numbers • 10% of families responded …… • 45% of children in OSEP category b • The numbers are not debatable! • Data need to be analyzed in “interpretable” ways. • There are choices with regard to how to analyze and present findings. • Some choices are better than others.

24. Meaning The numbers have no meaning by themselves. Then more people involved in interpreting the numbers the better! • The meaning is the interpretation put on the numbers: • Is this finding good news? Bad news? News we can’t interpret? • Meaning is debatable and people can reach different conclusions from the same set of numbers

25. Make meaning of the Data • A by-product of accountability is an “abundance of riches” with respect to data. • The challenge is to analyze the data that directly addresses the specific questions set forth in the inquiry process. • It may be helpful to summarize results according to the question they are addressing.

26. Putting Meaning on the Data • What are alternative explanations for the finding? • Are there other ways of looking at the data that might provide insight into a possible explanation? • Programs needs help with service provision vs. • Programs serving different kinds of children or families

27. Action Why are we engaged in this activity anyways? • Given the meaning put on the findings, what should be done? (A problem may have been identified by the data??) • Recommendations or action steps • Action is always debatable – and often is

28. Take Action • This part of the process makes use of the information gained from the analysis of the data. • Depending on the results of the analyses, programs could decide to change teaching practices, curriculum, or professional development offerings. • States could decide to reallocate resources or create new programs to address a need revealed by the analyses.

29. Building the Capacity to Use Data- Is Our System Healthy? Administrators and practitioners need to be able to ask questions about the system and the outcomes being achieved, get the data to address those questions, and make decisions based on what was learned. • Capacity building: • Ability to ask good questions • Have good data to answer the questions • Have the data analyzed in meaningful ways • Be able to interpret what the data mean • Be able to decide on appropriate actions

30. To Do’s • Create a culture that values accountability and using data for decision-making • Be patient – this is a long, evolving process • Be proud – celebrate the incredible accomplishments of the last 2 years!!