1 / 65

SSIP Phase I: Data Analysis

SSIP Phase I: Data Analysis. Part C/619 State Accountability Priority Area April 8, 2014. Disclaimer. This SSIP presentation and supplemental materials were developed prior to OSEP ’ s publication of the final SPP/APR package. Webinar Goals.

alissa
Download Presentation

SSIP Phase I: Data Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SSIP Phase I: Data Analysis Part C/619 State Accountability Priority Area April 8, 2014

  2. Disclaimer This SSIP presentation and supplemental materials were developed prior to OSEP’s publication of the final SPP/APR package

  3. Webinar Goals • Participants will leave the webinar with a basic understanding of: • Phase I: Data Analysis process • Resources and strategies that can support states in the Data Analysis process • Considerations for engaging stakeholders in the Data Analysis process

  4. Data Analysis Requirements A description of how the state analyzed key data to: • select the State-identified Measurable Result. • identify root causes contributing to low performance. The description must include information about: • how the data were disaggregated by multiple variables. • if applicable, any concerns about the quality of the data and how the state will address these concerns. • If applicable, methods and timelines related to any additional data to be collected and analyzed.

  5. Measurable Result Requirements • May, but need not, be an SPP/APR indicator or a component of an SPP/APR indicator. • Must be clearly based on the data and state infrastructure analyses. • Must be a child-level outcome in contrast to a process outcome. • May be a single result or a cluster of related results.

  6. Baseline and Targets Requirements • States must establish baseline (expressed as a percentage) with the Measurable Result (FFY 2013 data) • States must set targets (expressed as percentages) for each of five years FFY 2014 – FFY 2018 • FFY 2018 target must be higher than FFY 2013 baseline

  7. Fundamentals of Data Analysis • EIA • Evidence • Inference • Action • Starting with questions

  8. Fundamentals of Data Analysis • EIA • Evidence: just the #s • Inference: interpretation • Action: steps to be taken

  9. Starting with Question(s) • Where are areas of lower performance? (analysis by variables) • Geographic areas of the state • Child/family characteristics • Program characteristics

  10. Starting with Question(s) • Where are areas of lower performance? (analysis by child characteristics) • Does our program serve some children more effectively than others? • Do outcomes vary for children with different racial/ethnic backgrounds? • Are outcomes different for Dual Language Learners as compared to mono-language learners?

  11. Starting Points Potentially starting with: • An Issue (e.g. shifting demographic) • An Initiative • Child outcomes data

  12. Questions and Existing Initiatives • What is the state performance in social emotional development compared to other outcome areas? • What have our trends been in the area of social emotional development for young children? • Are there certain areas of the state that have lower performance in the area of social emotional development for young children? • Do social emotional outcomes differ by child characteristics (race/ethnicity, socioeconomic status, geographic region of the state)?

  13. Broad Data and Infrastructure Analysis • Purpose – Explore child results (and potentially the related family results) and practices that would be justifiable targets for improvement • Goal – Assemble evidence to substantiate to leadership and stakeholders why you selected a particular result • Strategies – Analysis of child results and related data to identify areas of lower performance

  14. Types of Broad Data Analysis Analysis of child outcomes data • By summary statement • State data compared to national data • Local data comparisons across the state • State trend data • Analysis by race/ethnicity, disability, income Analysis of related family outcomes data • State data compared to national data • Local data comparisons across the state • State trend data • Analysis by race/ethnicity, income, length of time in program • Linked to child outcomes data

  15. Resource: Broad Data Analysis Template • Purpose: to look at how children in the state are performing relative to national data, across years, within the state and by comparisons across programs http://ectacenter.org/eco/assets/docs/SSIP_child_outcomes_broad_data_analysis_template_FINAL.docx

  16. Resource: Broad Data Analysis Template

  17. Resource: Broad Data Analysis Template

  18. Meaningful differences calculator • Purpose: to look at statistical significance of change in state SS data from year to year; and allow comparison of local to state http://ectacenter.org/eco/pages/summary.asp#meaningfuldiffcalc

  19. Resource: Analyzing Child Outcomes Data for Program Improvement • Quick reference tool • Consider key issues, questions, and approaches for analyzing and interpreting child outcomes data. http://www.ectacenter.org/~pdfs/eco/AnalyzingChildOutcomesData-GuidanceTable.pdf

  20. Guidance Table

  21. What is the likely child result that will be the focus of your SSIP? • Social Relationships • Knowledge and Skills • Action to Meet needs • All three of the above • Something other than above

  22. In-Depth Data and Infrastructure Analysis • Purpose – Conduct further analysis exploring the link between the practices and infrastructure and the child result. • Goal – Gather sufficient evidence to link specific practices and infrastructure to child results (to inform needed improvement strategies). • Strategies - Subgroup analysis, comparisons of programs, “root cause analysis,” local data drill-down, narrative summary of analysis

  23. Identify Root Causes Contributing to Low Performance • Analyze data at the local level • Identify factors contributing to low performance (including infrastructure) • Contributing factors: • Explain why you have the problem • Point to how the problem can be addressed

  24. Identify Root Causes Contributing to Low Performance • Identify barriers for each contributing factor • What is standing in the way of addressing this contributing factor? • Why hasn’t it been addressed to date?

  25. Resource: Subgroup Analysis Template • Purpose: to provide states with table shells for subgroup analyses that have proven useful in understanding predictors of child outcomes. http://ectacenter.org/eco/pages/usingdata.asp

  26. Subgroup Analysis Example

  27. Local Contributing Factor Tool http://ectacenter.org/~docs/eco/ECO-C3-B7-LCFT.docx http://ectacenter.org/~docs/topics/gensup/14-ContributingFactor-Results_Final_28Mar12.doc

  28. LCFT: Question Categories System/ Infrastructure Practitioner/ Practices Policies/ procedures Competencies of staff Funding Implementation of effective practices Training/TA Time Supervision Resources Data Supports Personnel

  29. Data Quality • Not the focus of the SSIP • But must be addressed in the SSIP • Describe data quality issues • Describe data quality improvement efforts

  30. Data Quality: Pattern Checking • Checking predictable patterns to help determine ‘red flags’ to be investigated in the data. http://ectacenter.org/eco/assets/pdfs/Pattern_Checking_Table.pdf

  31. Data Quality Profiles • The profiles include information about: • State vs. national • Data quality criteria used for national analysis • Completeness of data • Progress categories patterns • Trends over time Contact: Abby Winer abby.winer@sri.com

  32. Data Analysis State ExampleVirginia Part C Beth Tolley Kyla Patterson

  33. Starting Point • Child and family outcome data • Stakeholder Input • State Interagency Coordinating Council (VICC) • Local System Managers

  34. Preparation • Child Outcomes Broad Data Analysis Template • Data Quality Profile • Support from ECTA and DaSy

  35. Process • Overview of SSIP • Powerpoint presentations and handouts with data and analysis questions • Large and small group discussion and input

  36. Broad Analysis Questions • Does our state’s data look different than the national data? • Are our state outcomes trends stable over time? • Is the data trending upwards? • Is the data trending downwards? • Is our state performing more poorly in some outcomes than others? • Are the outcomes similar across programs? • What about data quality? Can we be confident in our data?

  37. Child Outcomes: National vs. State FFY11 and State FFY12

  38. National Vs. State Meaningful Differences

  39. Virginia Trends

  40. Virginia Trends

  41. Child Outcomes: Local vs. State

  42. Data Quality Elements • Completeness of data • number of children reported for the outcome/number who exited • Virginia’s results: average= 65%; range for Local Systems = 17% - 100% • Expected Patterns for Progress Categories • Virginia’s state date is within these parameters for all three outcomes • Child Outcomes State Trends Over Time • As noted on previous slides, Virginia’s results do not show wide variations which would trigger concerns about data quality

  43. Family Outcomes: State Trends over Time

  44. Family Outcomes: Local vs. State 2012-20134C: EI has helped the family help their child develop and learn

  45. Identification of Area of Concern • Based on broad data analysis of child and family outcome data • VICC and LSM identified same area of concern: • Outcome 3C – Use of appropriate behaviors (taking action) to meet needs

  46. In-Depth Data Analysis Purpose: • To identify the specific measurable result • To identify root causes and contributing factors – why is it happening?

  47. Stakeholder Questions • Does the child’s reason for eligibility impact results on this outcome? • Does age at entry or length of time in EI impact results? • Does use of Part B entry ratings as Part C exit ratings have an impact? • Is there consistent understanding of the developmental areas involved in determining a rating on this indicator?

More Related