Measuring Child and Family Outcomes

1. Measuring Child and Family Outcomes Week Three Measuring Child Outcomes Child Outcome Summary Form

2. The Role of the COSF in Maryland • Last week we explored the various approaches states are taking in measuring child outcomes. • We learned that Maryland is taking a very unique approach that uses the present levels of development (PLOD) and the online database. • The Child Outcomes Summary Form (COSF) plays a vital role in determining whether Maryland’s approach of converting PLOD scores align with the level of functioning a child displays compared to their typical aged peers.

3. What Were the OSEP Reporting Requirements For Part C and Preschool Child Outcomes? Data is to be reported on the following outcomes: The percent of children who demonstrate improved: • Positive social emotional skills (including positive social relationships) • Acquisition and use of knowledge and skills (including early language/ communication [and early literacy]) • Use of appropriate behaviors to meet their needs

4. OSEP Reporting Categories Data from individual children will be aggregated and grouped into categories for reporting to the Office for Special Education Programs (OSEP): a. % of children who did not improve functioning b. % of children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers c. % of children who improved functioning to a level nearer to same-aged peers but did not reach it d. % of children who improved functioning to reach a level comparable to same-aged peers e. % of children who maintained functioning at a level comparable to same-aged peers

5. Dilemma: How do States Aggregate Data from Different Assessments? Remember states are trying to use domains based data programs to report on the three outcomes: • ECO developed the COSF (Child Outcomes Summary Form) to provide a common metric • The COSF also provides a way to summarize multiple sources of information on a single child • The majority of states are using the COSF to produce state level child outcomes data

6. Trends in Approaches to Measurement for Part C Child Outcomes • 40 states are currently using the ECO Child Outcome Summary Form • A 7 point rating scale based on multiple sources of data, often including assessment tools, observation, family report • 8 states using 1 assessment tool statewide • BDI--2: 3 states • State developed tools: 3 states • AEPS: 2 states • 3 states using on-line assessment systems with the capacity to report OSEP data reports • 5 states using other unique approaches- Maryland is one of those states!

8. Measuring Progress Based on the Rate of Growth Between Entry and Exit • Maryland is working with an Evaluation and Assessment Consultant to identify a methodology for measuring developmental gains during participation in early intervention • Currently, they have tested child data using an existing index: Intervention Efficacy Index (IEI) • Going by percent of delay isn’t an accurate picture of progress because children who come in at say 25% delayed and leave the program at 25% have still made progress.

9. Maryland’s Conversion Formula • While there are typical stages of growth, babies and toddlers develop at different paces, and may develop more quickly in one area than another. • A child’s developmental age (DA) may be lower than the child’s chronological age (CA), but still be considered at age level because children develop typical skills over a range of time. • Maryland selected a cut point of 19% difference between DA and CA to report status at entry data in February 2007 - % of children who entered at age level in each of three outcomes. • To report progress at exit data, we will also be using a 19% allowance for the formulas for each of the progress reporting categories.

10. The Big Question: Will Maryland’s Approach Work? • Can the currently collected data about present levels of development be used to provide valid information about the 3 functional outcomes?

11. Answering the Question • In order to effectively answer this question, we need to verify the process by comparing the outcomes data generated from Present Levels of Development (PLOD) with an outcomes judgment derived directly from those who know the child using the COSF (Child Outcomes Summary Form)

12. We will explore this question more this week! Week 3 Objectives: • Understand the role of the Child Outcomes Summary Form (COSF) in Maryland • Practice using the COSF • Explore how a child’s exit testing scores (PLOD) compare to the COSF ratings we give comparing his function to typical peers

13. How will Maryland Validate the Results? • In December 2006, local ITP’s began completing the Child Outcomes Summary Form (COSF) as soon as possible following initial evaluation and assessment. • Local programs are completing COSF’s at exit for children: • Who were referred since December 2006 • Who received services for at least six months, and • For whom a COSF was completed at entry • COSF results will be entered into the IFSP database.

14. Child Outcomes—ValidationWHY? Maryland has elected to use the information from domain-specificassessment results to determine the results of functional child outcomes, therefore it is important that the results be verified. In other words, we are comparing the following: “Are the responses derived from the electronically-extracted domain data consistent with direct responses from providers about a child’s functioning in the three outcomes?”

15. Why is There a Need for the Child Outcomes Summary Form? • The Child Summary process utilizes information from multiple sources to arrive at a single rating or score • This process allows data from different tools and sources to be comparable • Different programs will be using different assessment instruments • Outcomes data will need to be aggregated across programs within and across states

16. Features of the Child Outcomes Summary Form • The summary form allows for a variety of different kinds of information to be brought together with input from families and professionals • It is not an assessment tool • It uses information from assessment tools and observations to get a global sense of how the child is doing at one point in time • Ratings are based on the child’s functioning: • What the child does across settings and situations • Compared with what is expected given the child’s age • The resulting rating reflects a level of functioning that compares the child to typically developing peers. The process also includes a statement of child progress • 7-point rating scale

17. Another Benefit • Another benefit of the COSF is that it allows us to track progress over time • We do this by comparing the child’s functional status rating at entry and at exit to see how much progress has been made in those three outcome areas

18. A Decision Making Process • Using this process, the local program gathers together all information on each child. It is not an assessment tool. Instead it is a decision making process: • Used to transform information of many types and from multiple sources into the same three OSEP outcomes • Using information gathered from the parent and professional judgment to develop a consensus on a rating for each child • Based on different types of age-referenced tools that can compare child to same age peers • Based on information about child in natural contexts • This information is then summarized into a common scale, using a rating process. The rating provides a way to reduce many different kinds of data to a common score

19. This Requires taking a Global View of Children by… • Looking at child outcomes—what comes out of children’s participation in Part C supports and services—requires each team member to adopt a more global view of how the child is developing relative to both same-age peers and performance of lifelong skills in future situations and settings • That means taking a more panoramic view of the child’s function in the family’s home and community routines and activities with the standard of how same-age peers would perform in those situations and settings.

20. Using Multiple Measures & Sources in Your Decision Making Process • In addition to an assessment tool, you will want to consider other sources of information including: • Screening information • Family/caregiver conversation and interviews • Observations across a variety of settings • Anecdotal records • The information gathered to document progress toward goals and outcomes on IFSPs can be part of the information related to child outcomes

21. Family Input • There are several reasons why we want to use information from conversations with the family when considering our ratings for the COSF. The most important reason is the family will naturally have the most information about how their child takes part in everyday routines and activities in their home and usual settings.

22. Understanding the COSF

23. Why use Summary Ratings (1-7)? • The summary ratings reduce rich information from assessment and observation into a rating to allow a summary of progress across children • The ratings do not provide information for planning for the individual child. Information at the rich, detailed level will be more helpful for intervention planning purposes

24. Understanding the COSF Summary 1-7 Rating Metric • The rating provides a way to reduce many different kinds of data to a common rating • The rating is based on a 7-point scale that is anchored to typical functioning • A 6 and 7 Rating represent functioning at the same level as a typically developing peer • 1 to 5 ratings reflect below typical functioning

25. Definitions of Ratings • Each number between 1 and 7 contains a specific description to guide the decision making process considering the following: • How typical the behavior is in everyday situations • In comparison to expectations for age-matched peers • In terms of conditions or behaviors that interfere with the child’s ability to achieve age-expected behaviors and skills

26. Completing the COSF • For each of the three outcome areas, you will need to decide the extent to which the child displays behaviors and skills expected for his or her age • Ratings should reflect the child’s level of functioning using whatever assistive technology or special accommodations are present in the child’s typical settings

27. The Two COSF Questions that need to be Answered • To what extent does this child show age- appropriate functioning, across a variety of settings and situations, on this outcome? (Rating: 1-7) • Has the child shown any new skills or behaviors related to [this outcome] since the last outcomes summary? (Yes-No)

28. The Two COSF Questions that need to be Answered • Question a: To what extent does this child show age- appropriate functioning, across a variety of settings and situations, on this outcome? (Rating: 1-7) • Needs to be answered for all three outcomes at both entry and exit • Complete a new COSF form at exit answering question a for all three outcomes

29. The Two COSF Questions that need to be Answered • Question B: Has the child shown any new skills or behaviors related to [this outcome] since the last outcomes summary? (Yes-No) • Answer question B at exit for each of the three outcomes. • Remember question B looks at progress the child has made since entering the program. It is not about whether the child has changed COSF ratings. • If a child comes in at a COSF rating of 6 compared to his typical peers and leaves at a COSF rating of 6 when compared to his typical peers that child made progress!

30. The Summary Ratings (1-7): • Provide an overall sense of the child’s current functioning in 3 outcomes They do not provide: • Information on the services provided themselves • The family’s satisfaction with services or • An explanation of why the child’s functioning is at that level

31. Summary Ratings Reflect Global Functioning • Ratings on each outcome are a snapshot of • The whole child • Functioning • Across settings and situations • Ratings are not • Skill by skill • Split by domains

32. Essential Knowledge for Completing the COSF Between them, team members must: • Know about the child’s functioning across settings and situations • Understand age-expected child development • Understand the content of the three child outcomes • Know how to use the rating scale • Understand age expectations for child functioning within the child’s culture

33. What Happens to the COSF Form and Ratings • The COSF for each child must be maintained in their early intervention record • The ratings for entry and exit are entered into the MSDE Part B/C database where it is converted to OSEP ratings

34. A Reminder About Functional Outcomes • Functional refers to things that are meaningful to the child in the context of everyday living • Refers to an integrated series of behaviors or skills that allow the child to achieve the outcomes • Refers to skills and behaviors shown across a variety of settings and situations • They are not • a single behavior, nor are they • the sum of a series of discrete behaviors

35. Pros, Cons and Challenges to Maryland’s Approach! Pros: • Uses existing IFSP data and statewide database • Use of COSF for validation has generated good discussions of functional performance • Possibility of more meaningful results using factors such as time in intervention and relationship of chronological age to developmental age at entry

36. Pros, Cons and Challenges to Maryland’s Approach! Cons: • Using domain-based assessment results to measure developmental progress in functional outcomes • Using multiple assessment tools, rather than single or limited number of tools Challenges: • Making decisions thoughtfully with regard to impact on local programs and families, but quickly enough to ensure meaningful data and analysis

37. What if the “electronic results” and “validation results” are different? • During 2007-2008, we began to conduct a validation study by comparing electronic results (PLODs) and Child Outcome Summary Form (COSF) results, conducting focus groups with local programs, and discussing discrepancies in results with local provider teams. • Based on the validation study results, we will determine how our approach should be modified or changed, whether we will continue our progress reporting by using PLOD data or through the COSF.

38. What We’re Learning • Providers lack knowledge about typical development • Many providers question the expectation that they should know and/or assess child development beyond their discipline specific domain • Even in Universities with faculty knowledgeable in Part C, development and intervention is taught from a discipline specific approach • Concerns about validity and reliability

39. Implications for Personnel Preparation • Understanding functional outcomes • Moving beyond domains • Moving beyond disciplines • Understanding age expected behavior

40. Positive Implications seen so far… • Families report that developmental explanations make more sense • Providers report that it is easier to develop individualized functional goals • LITP directors report that the individual child outcomes and goals are more functional and relevant