1 / 39

ENHANCE Update

ENHANCE Update. Research Underway on the Validity of the Child Outcomes Summary (COS) Process . ECO Center Advisory Board Meeting March 8, 2012 Arlington, VA. ENHANCE is funded by grant R324A090171 from the the U.S. Dept. of Education, Institute for Educational Sciences. Topics.

herbst
Download Presentation

ENHANCE Update

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington, VA ENHANCE is funded by grant R324A090171 from the the U.S. Dept. of Education, Institute for Educational Sciences

  2. Topics Project design - review Data collection progress What we are learning Implementation Child assessment study State data study Next steps

  3. Project Design

  4. ENHANCE Project Objectives • Conduct a program of research to examine the validity of ratings generated by COS and identify conditions that lessen validity. • Revise the COS and supporting materials based on study findings. • Identify a series of validity analyses that can feasibly be conducted in states to allow each state to examine the validity of its own COS data on an ongoing basis.

  5. Validity – What Are We Trying to Demonstrate? • Validity is NOT a characteristic of an assessment or measurement device. • Validity is a characteristic of the data produced by the tool and how these data are used. • Are data valid for the purpose of….. • Implications: • State A’s COS data could be valid; • State B’s COS data could not be. Standards for Educational and Psychological Testing(1999) by American Educational Research Association, American Psychological Association, National Council on Measurement in Education

  6. Validation Process • Validation process: • Develop propositions (validity argument) - If data were valid for this use, then we would see…. • Collect evidence to examine each of those propositions

  7. Examples of propositions in the COS Validity Argument 3. Children differ from one another with regard to level of functioning in the 3 outcome areas as reflected in COS ratings. 7. Functioning (COS ratings) in an outcome area at time 1 is related to functioning in that area at a later point in time. 9. COS ratings will be related to the nature and severity of the child’s disability.

  8. Design Real World Not controlled Conditions

  9. Design:37 Project Data Collection Sites 19 Programs Part C • Illinois • Maine • Minnesota • New Mexico • North Carolina • Texas • Virginia 18 Districts Part B Preschool • Illinois • Maine • Minnesota • New Mexico • South Carolina • Texas

  10. Studies and Data Collection Progress

  11. ENHANCE Studies • Provider Survey • Team Decision-Making • Comparison with Child Assessments • State Data Study

  12. Goals Learn about COS implementation – processes in use Identify providers’ knowledge and training experiences Describe perceptions about if COS produces an accurate rating and influences on that Understand impact of COS on practice Provider Survey Process & Sample • Online survey responses • All providers in program who • participate in COS Status • Survey underway, continues • through April

  13. Goals Examine understanding and application of outcomes and rating criteria Describe team process Identify if ratings are consistent with evidence discussed Process & Sample Video teams discussing COS ratings 210 children’s teams Status Data collection underway Code videos this summer & fall Team Decision-Making Study

  14. What Are We Learning?

  15. Implementation Considerable variability across states and even across programs, within a state • Training • Ongoing staff support and quality assurance • Teaming (not just for COS) • Parent involvement • Timing and Process Implication: Results will tell us about COS validity under real-world conditions

  16. Number of Providers in COS Ratings - Preliminary Percentage of COS forms Number of providers

  17. Comparison with Child Assessments Study Goals • Compare entry and exit COS ratings to BDI-2 and Vineland-II scores • Compare conclusions from COS and assessments Process & Sample • Longitudinal, external assessor at program entry & program exit • 216 children Study Status • Local, trained assessors in place • Recruiting families since Aug. 2010 • Sample shows expected variability, including initial COS ratings, tool scores

  18. Comparison with Child Assessments Study – Preliminary…

  19. Validity argument claims 3. There is variability in children’s functioning in the three outcome areas and that variability is reflected in the COS ratings.

  20. Distributions of Preliminary COS Ratings (1-7) EI (n=71) ECSE (n=49) Ratings

  21. Validity argument claims 3. There is variability in children’s functioning in the three outcome areas and that variability is reflected in the COS ratings. 10. COS ratings in the corresponding outcome areas are moderately correlated with: • the social-emotional (Outcome 1), • cognitive (Outcome 2), • communication (Outcome 2), and • adaptive (Outcome 3) domain scores of assessment tools.

  22. Methods • Methods table

  23. Methods • Preliminary correlations between COS Ratings and assessment tools • What expect to see? 1.0 .00 .70 .42

  24. Preliminary Correlations: COS Ratings & Assessment Scores ECSE larger COS-Assessment Correlations than EI

  25. Correlations: BDI-2 and Vineland-II Domains ECSE larger BDI-Vineland Correlations than EI

  26. Methods • COS Group 1 – Ratings of 1, 2, 3 • COS Group 2 – Ratings of 4, 5 • COS Group 3 – Ratings of 6, 7

  27. Outcome 1: Positive Social Relationships EI (n=71) ECSE (n=49)

  28. Outcome 2: Acquiring and Using Knowledge and Skills EI (n=71) ECSE (n=49)

  29. Outcome 3: Taking Appropriate Action to Meet Needs EI (n=71) ECSE (n=49)

  30. Summary of preliminary findings Patterns • Means for groups generally follow expected directions on assessment tools • Group comparisons showed expected differences • Effect sizes were nearly all larger for ECSE than EI • For COS – assessment tool comparisons • For comparisons between assessment tools by the same external assessor More data are needed for final conclusions

  31. State Data Study Goals • Examine characteristics of COS data and relationships to other variables • Look for consistency in patterns across states to test claims Sample • All valid COS data within the state for a reporting year • 15-18 states conducting all analyses • Additional states sharing select analyses Status • Piloted procedures with 3 Part C, 3 Part B Preschool states • Now working with 4 Part C, 6 Part B Preschool states • Recruiting more states, requesting data

  32. Validity argument claims 7. Functioning, as reflected in the COS rating, in an outcome area at time 1 is related to functioning in that area at a later point in time.

  33. Correlations:Entry and Exit Ratings Part B 619 Preliminary state data

  34. Validity argument claims 3. There is variability in children’s functioning in the three outcome areas and that variability is reflected in the COS ratings. 14. COS rating distributions at entry will be related to the disability-related characteristics of the population served by states.

  35. Part C 08-09 entry ratings across states Taking appropriate action to meet needs 2 3 4 5 6 7 COS Ratings

  36. Next Steps and Reactions

  37. Next steps • Gather more state data • Complete data collection involving local programs/districts • Analyze provider survey results • Code videos

  38. Questions? Reactions? • Questions? Reactions? • Implications for the national data? • Implications for ECO?

  39. Find out more ENHANCE Website http://ENHANCE.sri.com ECO Center Website http://www.the-ECO-center.org Contact ENHANCE staff E-mail: ENHANCE@sri.com

More Related