Using the ECO Child Outcomes Summary Tool

1. Using the ECO Child Outcomes Summary Tool Kathy Hebbeler ECO Center at SRI International Prepared for the State TA Call, October 20, 2005 Early Childhood Outcomes Center

2. Important Information • We will not be going over the material on slides 4 to 37 on the call. It is for your information and will be discussed at the OSEP Early Childhood Meeting post-season on outcomes in December. • State options are indicated with a . Early Childhood Outcomes Center

3. Go to Slide 38  Early Childhood Outcomes Center

4. ECO Framework for Developing an Outcomes System 1. Identify purposes 2. Identify outcomes 3. Specify evidence statements 4. Identify measurement approaches (including how to transform the data) • Implement procedures for data transmission And other steps….. Early Childhood Outcomes Center

5. Decision: Purpose of the outcomes system • Produce data for OSEP • Provide evidence of effectiveness of program (state legislature, ICC, etc.) • Provide information for program improvement • Etc. Early Childhood Outcomes Center

6. Decision: Identify child outcomes areas • Given: OSEP child outcome areas • Positive social emotional skills (including positive social relationships) • Acquisition and use of knowledge and skills (including early language/communication) • Use of appropriate behaviors to meet their needs • State decision: Other outcome areas? Sub-outcomes? Early Childhood Outcomes Center

7. Decision: Identify family outcomes areas • Given: OSEP family indicators • Percent of families participating in Part C who report that EI services have helped the family • Know their rights • Effectively communicate their children’s needs • Help their children develop and learn • State decision: Other outcome areas? Early Childhood Outcomes Center

8. Child outcome areas: What states are doing • Adopting the 3 as is • Adding sub-indicators under the outcomes • ? Early Childhood Outcomes Center

9. Family outcome areas: What states are doing • Add to/expand the OSEP indicators • Add to/expand the ECO family outcomes • E.g., quality of life • ? Early Childhood Outcomes Center

10. Evidence Statements: OSEP only or More? Early Childhood Outcomes Center

11. Decision: What kind of evidence statements does state want to be able to produce? Evidence Statement - a statement that incorporates a statistic and provides evidence as to whether not an outcome has been achieved Early Childhood Outcomes Center

12. Decision: What kind of evidence statements to produce? • Given = OSEP indicators/measures (evidence statements): a. % of infants and toddlers/preschoolers who reach or maintain functioning at a level comparable to same-age peers b. % of infants and toddlers/preschoolers who improve functioning but are not in “a” c. % of infants and toddlers/preschool children who did not improve functioning Early Childhood Outcomes Center

13. OSEP Indicators and Measurement Categories 5 year old level Group a: maintained or reached typical 2 year old level Entry Exit Early Childhood Outcomes Center

14. Indicators and Measurement Categories 5 year old level Group b: made progress but… 2 year old level Entry Exit Early Childhood Outcomes Center

15. OSEP Indicators and Measurement Categories 5 year old level Group c: did not make progress 2 year old level Entry Exit Early Childhood Outcomes Center

16. OSEP Indicators and Measurement Categories Group a: maintained or reached typical 5 year old level Group b: made progress but.. Group c: did not make progress 2 year old level Entry Exit Early Childhood Outcomes Center

17. Decision: Does the state want to be able to produce any other evidence statements? • Note: this does not necessarily mean collecting more data. It could mean recording or analyzing the same data in a different way. • If the state wants outcome data for anything other than to meet the federal reporting requirements, the answer to this question is almost certainly “Yes.” Early Childhood Outcomes Center

18. Decision: Does the state want to be able to produce any other evidence statements? • The investment in building the outcomes system is in collecting good assessment data. Rolling up or analyzing the data differently adds almost nothing to the cost. Early Childhood Outcomes Center

19. Good news, bad news + OSEP evidence statements are relatively straightforward - OSEP evidence statements are of limited usefulness if state wants data for demonstrating effectiveness or for program improvement Early Childhood Outcomes Center

20. Who is in OSEP Group a? 5 year old level Group a: maintained or reached typical = children who “closed the gap” 2 year old level Entry Exit Early Childhood Outcomes Center

21. Who is in OSEP Group b? 5 year old level Group b: made progress but… = children who made progress toward “closing the gap” 2 year old level Entry Exit Early Childhood Outcomes Center

22. a2+ b1

23. Changing developmental trajectories • The strongest evidence for the effectiveness of EI/ECSE is the percentage of children whose developmental trajectories changed after they received services • “Gap closers” = a2 + b1 Early Childhood Outcomes Center

24. cat Illustration: Getting more powerful evidence (OSEP only)

25. Illustration: Getting more powerful evidence (same data, more information) 72 74 79

26. Measurement: Good assessment procedures are the foundation for the child outcome system Early Childhood Outcomes Center

27. What is assessment? • “Assessment is a generic term that refers to the process of gathering information for decision-making.” (McLean, 2004) Early Childhood Outcomes Center

28. What is assessment? • “Early childhood assessment is flexible, collaborative decision-making process in which teams of parents and professionals repeatedly revise their judgments and reach consensus about the changing developmental, educational, medical, and mental health services needs of young children and their families.” Bagnato and Neisworth, 1991 Quoted in DEC Recommended Practices, 2005 Early Childhood Outcomes Center

29. DEC Recommended Practices for Assessment • Involves multiple sources (families, professional team members, service providers, caregivers, etc.) • Involves multiple measures (observations, criterion-curriculum-based instruments, interviews, curriculum-compatible norm-referenced scales, informed clinical opinion, work samples) Early Childhood Outcomes Center

30. State decisions related to assessment • What assessment information to include in the outcome system? • Single measure statewide • List of approved measures • Whatever measures programs are using Early Childhood Outcomes Center

31. State decisions related to assessment • What assessment information to include in the outcome system? • Collect new assessment information • Use existing assessment information • States now gathering information to learn what measures are being used in the state Early Childhood Outcomes Center

32. State decisions related to building on existing assessment information • Options: • Build upon data from the eligibility determination process for outcomes • Build upon data from ongoing progress monitoring for outcomes • Combination Early Childhood Outcomes Center

33. Building on Eligibility Measures • States use standardized, norm-referenced tests as part of eligibility + Easy to anchor to typical development • Not necessarily reflective of young children’s skills and behaviors in regular contexts • Challenges in aligning domains to functional outcomes • Challenges with cultural relevance for diverse populations • States add other information sources (informed clinical opinion) for eligibility determination Early Childhood Outcomes Center

34. Building on Ongoing Progress Monitoring • States use curriculum based/ criterion referenced measures for ongoing progress monitoring + Based on observation in everyday contexts by those who know children well + Many c-b measures are a closer match than norm-referenced measures for the 3 functional outcomes • Some have age expectations but for others a challenge in establishing age anchors • Training is needed to insure measures is administered reliably, especially if have large numbers of “assessors” with varying levels of experience and expertise Early Childhood Outcomes Center

35. State decisions related to assessment • What kind of measures to include? • Norm-referenced • Curriculum-based • Combination Early Childhood Outcomes Center

36. State decision related to assessment • Parent input is critical • Consistent with best practice • Impossible to determine how child is doing across a variety of settings without information from caregivers • How to incorporate parent input? • As part of the item scoring within a measure • As a separate source of information • Both Early Childhood Outcomes Center

37. State decisions related to assessment • Frequency of data for outcome system? • Given for OSEP: (Near) entry and (near) exit • Anything in between? • every 6 months • annually • at a fixed point (e.g., every March) Early Childhood Outcomes Center

38. Transforming the assessment data Early Childhood Outcomes Center

39. Need to transform the data • Under any assessment option, state needs a way to transform the assessment information into the indicators/evidence statements. • No assessment provides information directly on the 3 outcomes areas • No assessment provides information directly on the 3 categories (a,b,c) in the OSEP indicators Early Childhood Outcomes Center

40. Need to transform the data • …if assessment process involves multiple sources of information • …if more than one assessment is being used in the state Early Childhood Outcomes Center

41. DesiredOutput Input ≠ OSEP Indicators Social % a, b, c Knowledge &… % a, b, c Meet Needs % a, b, c Time 1 Scores Cognitive Communication Social Adaptive Motor Time 2 Scores Cognitive Communication Social Adaptive Motor (for 10,000 children!) Early Childhood Outcomes Center

42. Decisions related to transforming the assessment data • What is the process by which the data gets transformed? • What is the rubric used to “roll up the data? • Where does the transformation occur? • State • Local program Early Childhood Outcomes Center

43. What is the rubric? • Depends on the kind of evidence statements desired • If OSEP only, 2-point scale is sufficient • If more is desired, need a scale with more points • ECO has drafted a rubric with a 7-point scale Early Childhood Outcomes Center

44. If state only wants 3 OSEP indicators • Transform data for each child into 2 categories at entry: Q1. [Based on assessment and other information…] Are child’s skills and behavior related to taking action to meet needs at the level expected for his or her age? 1. yes 2. no Early Childhood Outcomes Center

45. If state only wants 3 OSEP indicators • And 2 items, 2 categories each, at exit: Q2a. [Based on assessment and other information…] Are child’s skills and behavior related to taking action to meeting his or her needs at the level expected for his or her age? 1. yes 2. no Q2b. [Based on assessment and other information…] Has the child acquired new skills and behaviors related to taking action to meet needs since entering the program? 1. yes 2. no Early Childhood Outcomes Center

46. If state only wants 3 OSEP indicators • Using the answers to those 3 questions, the state can classify every child as a, b, or c. • However, transforming the data differently allows the state to do a lot more with the data. Early Childhood Outcomes Center

47. If states want stronger evidence statements: ECO Child Outcomes Summary Form (under development) • Rubric that summarizes child status on each outcome on a 7 point scale • NOT an assessment; rather a summary of information on child functioning • Has gone through several revisions and is still being revised • In an ideal world, would be extensively researched before being released Early Childhood Outcomes Center

48. Age-expected skills and behavior Movement away from age-expected Movement toward age-expected A way to think about how children are doing with regard to each outcome

49. Child Outcomes Summary Form

50. Definitions for Outcome Ratings