1 / 50

The Results are In:  Using Early Childhood Outcome Data

The Results are In:  Using Early Childhood Outcome Data. Kathy Hebbeler and Lynne Kahn ECO at SRI International and ECO at UNC. August, 2011. What we will cover. Quick review of the reporting requirement and state approaches Share the national data

justine-chen
Download Presentation

The Results are In:  Using Early Childhood Outcome Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Results are In:  Using Early Childhood Outcome Data • Kathy Hebbeler and Lynne Kahn • ECO at SRI International and ECO at UNC August, 2011

  2. What we will cover • Quick review of the reporting requirement and state approaches • Share the national data • Describe how the national data were computed • Discuss the quality of the national data • Discuss the meaning of the numbers Early Childhood Outcomes Center

  3. OSEP Reporting Requirements: Child Outcomes • Positive social emotional skills (including positive social relationships) • Acquisition and use of knowledge and skills (including early language/ communication [and early literacy]) • Use of appropriate behaviors to meet their needs Early Childhood Outcomes Center

  4. OSEP Reporting Categories Percentage of children who: a. Did not improve functioning b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c. Improved functioning to a level nearer to same-aged peers but did not reach it d. Improved functioning to reach a level comparable to same-aged peers e. Maintained functioning at a level comparable to same-aged peers 3 outcomes x 5 “measures” = 15 numbers Early Childhood Outcomes Center

  5. The Summary Statements • Of those children who entered the program below age expectations in each outcome, the percent who substantially increased their rate of growth by the time they turned 3 years of age or exited the program. • The percent of children who were functioning within age expectations in each outcome by the time they turned 3 years of age or exited the program. Early Childhood Outcomes Center

  6. State Approaches to Outcomes Data

  7. Note: Based on 29 States with highest quality data Early Childhood Outcomes Center

  8. Note: Based on 29 States with highest quality data Early Childhood Outcomes Center

  9. Note: Based on 33 States with highest quality data Early Childhood Outcomes Center

  10. Note: Based on 33 States with highest quality data Early Childhood Outcomes Center

  11. Criteria for States with Quality Data • Low percentage of missing data • No odd patterns in “a” or “e” categories • Did not use questionable data collection methods Early Childhood Outcomes Center

  12. Calculating Missing Data for Part C Proxy for missing data = Number with data for C3/ Exiting Data (618) • Do not expect this number to be 100% • ..but we don’t expect it to be 10% either Early Childhood Outcomes Center

  13. Part C: Percent of Exiters included in Outcomes Data 08-09 <10% = 10* 10- 20% = 4 20- 30% = 8 30- 40% = 11 40- 50% = 8 50- 60% = 8 60- 70% = 4 70- 80% = 2 >80% = 1 09-10 <10% = 5* 10- 20% = 4 20- 30% = 6 30- 40% = 8 40- 50% = 5 50- 60% = 11 60- 70% = 9 70- 80% = 1 >80% = 0 *3 states are sampling for Part C. Excluded states with <28%.

  14. Calculating Missing Data for 619 Proxy for missing data = Number with data for B7/ Child count • Do not expect this number to be 100% • ..but we don’t expect it to be 10% either

  15. Percent of Child Count included in Outcomes Data for ECSE 08-09 <10= 11* 10- 20%= 15 20- 30%= 12 30- 40%= 12 40-50% =1 >50% = 2 09-10 <10= 6* 10- 20%= 11 20- 30%= 12 30- 40%= 16 40-50% =4 >50%= 0 *4 States are sampling for 619 Excluded states with <12% of child count

  16. Problem with Missing Data • We don’t know how well the data the state has represent the entire state. • If the data are representative, the percentages for the a to e Progress Categories and the Summary Statements won’t change as data are added on more children. Early Childhood Outcomes Center

  17. Exclusion criteria: a<10% in any outcomes e<65% in any outcomes Early Childhood Outcomes Center

  18. Can we trust these data? Early Childhood Outcomes Center

  19. Pattern checking for validity • Checking across years • How do the 2009-10 compare to the data for 2008-09? • Checking across methods • How do the data for all states compare to states with highest quality data? Early Childhood Outcomes Center

  20. Part C, Outcome A: Social Relationships

  21. Part C, Outcome B: Knowledge and Skills

  22. Part C, Outcome C: Meets Needs

  23. Part B Preschool: Social Relationships

  24. Part B Preschool: Knowledge and Skills

  25. Part B Preschool: Meets Needs

  26. Part C

  27. Part C

  28. What to these data tell us? • Nationally, a high proportion of children who receive Part C and ECSE services are showing greater than expected progress • Nationally, many (over half) are exiting the program functioning like same age peers in at least one of the outcomes. Early Childhood Outcomes Center

  29. Should your state data look like the national data? • Probably not • More important that each state continue to focus on the quality of its own data • Getting outcomes data on all children who exit • Working with programs whose data look unusual to address possible data quality issues Early Childhood Outcomes Center

  30. Supporting States in Building a Child Outcomes Measurement System

  31. Two Frameworks • Child Outcomes Measurement System • Family Experiences and Outcomes Measurement System Early Childhood Outcomes Center

  32. Child Outcomes Measurement System • The set of components a state needs to have in place to make full use of child outcomes data. • NOT just a data system or a data collection method. Early Childhood Outcomes Center

  33. Purpose of the Framework • Provide a common language for ECO and other TA providers to use in discussing COMSs with states. • Provide a organizing structure of categorizing resources and state examples related to implementation of a COMS. • Serve as the organizing structure for the state self assessment Early Childhood Outcomes Center

  34. Framework and Self-Assessment • Framework • Set of components and quality indicators • Provides the structure for the self-assessment • Self-assessment • Scale that provides criteria for levels of implementation within each quality indicator • Rating assigned based on level of implementation within each indicator Early Childhood Outcomes Center

  35. Process for Framework Development • Built off what we had learned from ECO work with states • Literature review • Repeated discussion and review internally and with 7 Partner States Early Childhood Outcomes Center

  36. Framework Partner States Early Childhood Outcomes Center

  37. COMSFramework Components Purpose Analysis Data Collection and Trans-mission Reporting Using Data Cross-system Coordination Evaluation Early Childhood Outcomes Center

  38. Quality Indicator • Provides additional detail as to what constitutes quality implementation of the component. • 18 quality indicators across the 7 components Early Childhood Outcomes Center

  39. Quality Indicators Components Elements Purpose Data Collection and Transmission Analysis Reporting Using Data Evaluation Cross-System Coordination a. State has… b. State has… c. State agency.. d. Representative.. e. State agency… f. State …… g. State provides… h. State has.. Early Childhood Outcomes Center

  40. Early Childhood Outcomes Center

  41. Purpose of the Self Assessment • Assist states in setting priorities for improving their COMS • Provide information to assist states in advocating for resources for systems development • Provide guidance to states on what constitutes a high quality child outcomes measurement system. Early Childhood Outcomes Center

  42. Each QI has multiple elements. Evidence for the extent of implementation for each element is provided. Each element is rated as NY = Not Yet or Don’t know IP = In Process, or IF = Fully Implemented The QI is given a rating based on the ratings of the elements. Early Childhood Outcomes Center

  43. Back-up for Each Element Live link from the element Describes the element Describes what “fully implemented looks like” Provides examples of what “In process” might look like Provides examples of how states are addressing the element Provides additional resources related to the element Early Childhood Outcomes Center

  44. The Scale for the Quality Indicators Early Childhood Outcomes Center

  45. Early Childhood Outcomes Center

  46. Working with the Self Assessment Early Childhood Outcomes Center

  47. Possible Process • State identifies the highest priority QI(s). • Stakeholders convened • Priority QIs reviewed and rated • Plan developed to address elements not yet fully implemented. • Progress reviewed with stakeholders at regular intervals. • State identifies next set of QI(s), etc. Early Childhood Outcomes Center

  48. Additional information For information on improving data quality and using data for program improvement www.the-eco-center.org Measuring and Improving Child and Family Outcomes Conference, September 18-21, 2011 in New Orleans, LA Early Childhood Outcomes Center

More Related