Measuring Child and Family Outcomes

1. Measuring Child and Family Outcomes Session Two Gathering Child Outcome Results

2. A Reminder of Why OSEP’s Focus on Early Childhood Outcomes? • The ultimate goal of an outcomes measurement system is to improve results for young children with disabilities and their families! • To address the Government Performance and Results Act (GPRA), Program Assessment Rating Tool (PART), and IDEA 2004 requirements

3. Goal of Early Intervention • “…To enable young children to be active and successful participants during the early childhood years and in the future in a variety of settings – in their homes with their families, in child care, in preschool or school programs, and in the community.” (from Early Childhood Outcomes Center, http://www.fpg.unc.edu/~eco/pdfs/eco_outcomes_4-13-05.pdf)

4. What Were the OSEP Reporting Requirements For Part C and Preschool Child Outcomes? Data is to be reported on the following outcomes: The percent of children who demonstrate improved: • Positive social emotional skills (including positive social relationships) • Acquisition and use of knowledge and skills (including early language/ communication [and early literacy]) • Use of appropriate behaviors to meet their needs

5. OSEP Reporting Categories Data from individual children will be aggregated and grouped into categories for reporting to the Office for Special Education Programs (OSEP): a. % of children who did not improve functioning b. % of children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers c. % of children who improved functioning to a level nearer to same-aged peers but did not reach it d. % of children who improved functioning to reach a level comparable to same-aged peers e. % of children who maintained functioning at a level comparable to same-aged peers

6. Why Measure Child Outcomes? We need to know how young children with disabilities are benefiting from early intervention and preschool special education: • Federal Reporting Requirements All States must report data to OSEP Impact on federal funding • Program Effectiveness Evidence for State and local policymakers, sponsors See program as good investment See program as important with some challenges that must be addressed • Program Improvement Find meaning in the data to make systemic changes Use data to improve IFSP services to individual/groups of children

7. State Approaches States had many things to consider when developing their plan to gather the needed Child Outcomes data: • Who will provide the data? • What assessments will be used? • How often will data be collected? • When is data collected? • When is it reported? • Dealing with multiple sources? • Dealing with different assessments?

8. State Approaches Related to Assessment Tools • There is no ONE assessment tool that assesses the three OSEP Child Outcomes areas directly • Some states are choosing one assessment selected by state • Some states have developed a list of assessments developed by the state that programs can pick from • While other states are allowing programs to use whatever they have been using • Maryland has provided a list of the five most commonly used assessments

9. Dilemma: How do States Aggregate Data from Different Assessments? Remember states are trying to use domains based data programs to report on the three outcomes: • ECO developed the COSF (Child Outcomes Summary Form) to provide a common metric • Form also provides a way to summarize multiple sources of information on a single child • The majority of states are using the COSF to produce state level child outcomes data

10. Frequency of Data Collection • Some states are measuring outcomes only at entry and exit (OSEP’s minimal requirement) • This is the approach Maryland is taking! • Some states are measuring more often (allows for more meaningful information) • The current status of a state’s data system is a huge factor in frequency of data collection

11. Trends in Approaches to Measurement for Part C Child Outcomes • 40 states are currently using the ECO Child Outcome Summary Form • A 7 point rating scale based on multiple sources of data, often including assessment tools, observation, family report • 8 states using 1 assessment tool statewide • BDI--2: 3 states • State developed tools: 3 states • AEPS: 2 states • 3 states using on-line assessment systems with the capacity to report OSEP data reports • 5 states using other unique approaches- Maryland is one of those states!

13. Rationale for Maryland’s Approach • Desire to align outcome process with the IFSP process • Focus on improving evaluation and assessment practices • Focus on ensuring data is collected in all domains • Have a data system that collects Present Levels of Development (PLOD) • Can get started right away by generating electronic reports from data entered into PLOD • Response to local input

14. How does Maryland Aggregate Data? Child enters LITP at 20 months Data is extracted from the information that has been entered into the Online IFSP-- Present Levels of Development Child exits LITP at 36 months Data is extracted from the information that has been entered into the Online IFSP-- “New Section” (similar to current Present Levels of Development) and Compared to Entry Data To determine “progress.” Progress At Exit Data Status At Entry Data

15. How Do We Get The Data? (for Status At Entry and Progress at Exit)Data Extracted from Present Levels of Development and Electronically Linked to 3 Outcomes to Produce Answers Alignment of broad outcomes to Present Levels of Development Is the child’s acquisition and use of knowledge and skills (including earlylanguage/communication) at the level expected for his or her age?  ___Yes _____No Are child’s social-emotional skills (including social relationships) at the level expected for his or her age?  ___Yes _____No Based on assessment and other information, does the child use appropriate behavior to meet his or her needs at the level expectedfor his or her age?___Yes _____No

16. Protocols for Linking Age Levels/Age Ranges with Outcomes • For the outcome “acquisition and use of knowledge and skills (including early language/communication),” two domain categories (cognitive and communication) are used. If both domains have quantitative data, the category that has the lowest range of data is used. • When an age range has been entered, the midpoint of the range is used.

17. How does MD’s Local Programs Measure and Report Status at Exit Data? • Assessment at exit results are entered on the IFSP form, Section II: Present Levels of Development (PLOD). There is a box to check off if the PLOD are from an entry, interim or exit assessment. • The name of the assessment(s) that were used are documented on the IFSP form. This helps narrow list and determine variability of data. • The results of the assessments at exit are entered into the IFSP database.

18. Who/When/What Does Maryland Rate • Entry ratings are needed in all outcome areas even if a child has delays in one or two outcomes, but not in all three outcomes • Entry ratings on each of the three outcomes should be reported for EVERY child who qualifies for services as soon as possible after the initial eligibility assessment is completed • Exit ratings should be reported on each of the three outcomes for every child who had a COSF completed at entry and who has been in the program for at least 6 months

19. When are Other States Measuring Data? • It is important to know that all states are approaching this question differently. Each state has established their point in time for capturing data. • The definitions of entry and exit on the previous slide are specific to Maryland and provide specific uniform points in time for capturing data. • Maryland will measure children at entry into and exit from of the Maryland Infants and Toddlers system.

20. Maryland’s Approach Converting Assessment Data to OSEP Outcome Categories

21. Measuring Progress Based on the Rate of Growth Between Entry and Exit • Maryland is working with an Evaluation and Assessment Consultant to identify a methodology for measuring developmental gains during participation in early intervention • Currently, they have tested child data using an existing index: Intervention Efficacy Index (IEI) • Going by percent of delay isn’t an accurate picture of progress because children who come in at say 25% delayed and leave the program at 25% have still made progress.

22. Intervention Efficacy Index • The intervention efficacy index relates changes in child capabilities to time spent in program • It describes individual and group progress in terms of developmental gains within and across domains for each month in an intervention program. (Bagnato & Neisworth)

23. Intervention Efficacy Index (IEI) IEI = Developmental gain in months Time in intervention in months IEI = Exit Developmental Age – Entry Develop. Age Time in intervention IEI = 34 months – 20 months 12 months IEI = 1.17 Average developmental gain for each month in intervention; an average of 1.17 months of gain for each month or participation in the intervention.

24. Maryland’s Conversion Formula • While there are typical stages of growth, babies and toddlers develop at different paces, and may develop more quickly in one area than another. • A child’s developmental age (DA) may be lower than the child’s chronological age (CA), but still be considered at age level because children develop typical skills over a range of time. • Maryland selected a cut point of 19% difference between DA and CA to report status at entry data in February 2007 - % of children who entered at age level in each of three outcomes. • To report progress at exit data, we will also be using a 19% allowance for the formulas for each of the progress reporting categories.

25. Conclusion of Test • Conclusion: The IEI must be anchored to the child data to be meaningful.

26. Linking Results to OSEP Categories To make the data meaningful, Maryland had to: • Test the IEI index with real-child data And then: • Determine numerical ranges that would provide the most accurate linkage to OSEP categories

27. The Big Question: Will Maryland’s Approach Work? • Can the currently collected data about present levels of development be used to provide valid information about the 3 functional outcomes?

28. Answering the Question • In order to effectively answer this question, we need to verify the process by comparing the outcomes data generated from Present Levels of Development (PLOD) with an outcomes judgment derived directly from those who know the child using the COSF (Child Outcomes Summary Form)

29. We will explore this question more next week! Looking forward: • The role of the Child Outcomes Summary Form (COSF) in Maryland • Practice using the COSF • Explore how a child’s exit testing scores (PLOD) compare to the COSF ratings we give comparing his function to typical peers