Other Uses for Part C Child and Family Outcome Data

1. Other Uses for Part C Child and Family Outcome Data Linda Goodman Connecticut Birth to Three System Early Childhood Outcomes Conference July 31, 2010

2. Use #1 Results and Performance Accountability, Decision-making and Budgeting Mark Friedman Fiscal Policy Studies Institute Santa Fe, New Mexico www.resultsaccountability.comwww.raguide.org

3. “All Performance Measures that have ever existed in the history of the universe involve answering two sets of interlocking questions.”

4. Quantity Quality HowMuch did we do? ( # ) HowWell did we do it? ( % )

5. Effort HowWell HowMuch Effect

6. Quality Quantity How welldid we do it? How much did we do? Effect Effort Is anyonebetter off? # %

7. Quality Quantity How welldid we deliver it? How much service did we deliver? Effect Effort Output Input How much change/effect did we produce? What quality of change/ effect did we produce?

8. Education Percent of 9th graders whoenter college oremployment after graduation Number of 9th graders whoenter college oremployment after graduation Quantity Quality How well did we do it? How much did we do? Student-teacherratio Number ofstudents Effect Effort Is anyone better off?

9. Number of children exiting whose development was accelerated Percent of children exiting whose development was accelerated Part C Quantity Quality How well did we do it? How much did we do? Number of Eligible Children Percentage of children under 12 months Effect Effort Is anyone better off?

10. Not All Performance Measures Are Created Equal Quantity Quality How much did we do? How well did we do it? LeastImportant 2nd MostImportant Effect Effort Is anyone better off? 3rd MostImportant MostImportant

11. The Matter of Control Quantity Quality How much did we do? How well did we do it? MoreControl Effect Effort Is anyone better off? LessControl Partnerships needed to improve performance

12. “Turning the curve” “We haven’tgot the money, so we’ve gotto think.” Lord Rutherford1871 - 1937

13. THE LINKAGE Between POPULATION and PERFORMANCE POPULATION ACCOUNTABILITY Healthy BirthsRate of low birth-weight babiesStable FamiliesRate of child abuse and neglectChildren Succeeding in SchoolPercent graduating from high school on time PERFORMANCE ACCOUNTABILITY Child Welfare Program POPULATIONRESULTS Contributionrelationship Alignmentof measures # Foster ChildrenServed % withMultiplePlacements CUSTOMERRESULTS Appropriateresponsibility # RepeatAbuse/Neglect % RepeatAbuse/Neglect

14. Connecticut Part C Example Quality of Life Result: All Connecticut children are healthy and ready for school success at age 5, contributing to a reduction over time in Connecticut’s achievement gap at Grade 4. Program Contribution to Result: By providing family-centered early intervention services, the program strengthens the capacity of Connecticut’s families to meet the developmental and health-related needs of their infants and toddlers who have delays or disabilities, thereby ensuring that more of these children are ready for Kindergarten. Activities include coaching caregivers to embed intervention in the child’s daily natural routines and learning opportunities. Partners: 44 contracted providers programs; local school districts; pediatricians and family medicine practitioners, hospital NICU staff, Office of Policy and Management; Department of Public Health; Department of Insurance; State Department of Education; Department of Children and Families; Board of Education and Services for the Blind; Children’s Trust Fund, UCEDD, Early Childhood Cabinet; HMO Association Members and Anthem; United Way; State ICC; Local ICCs;

15. Performance Measure 1 Percentage of infants and toddlers with disabilities or developmental delays who improve so that at exit, they function at age level in three skill areas Story Behind the baseline: The three skill areas on the graph are: (1) social/emotional; (2) acquisition and use of knowledge (including early language and literacy); and (3) use of appropriate behavior to meet their needs. Each bar represents the percentage of children that performed at age level in each skill area at the time they exited the Birth to Three System. Data began in FY07 and some variation is due to the fact that it has taken three years for the children with the most significant disabilities, identified at birth, to exit the system. Children that received services for at least six months prior to exit Skill Areas

16. Proposed actions to turn the curve: Since Birth to Three only enrolls children with significant delays or disabilities, it is not possible that 100% of children will exit the program at age-level. However, we do expect these percentages to increase from their current levels. The research shows that improvement is most highly correlated with the ability of each provider to ensure that parents and childcare givers know how to incorporate intervention techniques into daily routines so that each child gets the maximum amount of practice, all day, every day. This is a different approach than out-patient rehabilitation services, in which short sessions in therapy rooms are expected to lead to generalization and carryover into other environments. That approach does not work well for infants and toddlers. Since each local program is responsible for overseeing the delivery of early intervention services, the next step will be to rank each program on its child outcome data and then focus on improving the low-performing programs through technical assistance and additional training for their personnel.

17. Performance Measure 2: Percentage of families who report that, as a result of receiving services from the Birth to Three System, they are better able to help their children develop and learn. Story behind the baseline: The blue bars on the graph show the percentage of families who strongly or very strongly agreed with the statement: “Birth to Three has helped me to help my child develop and learn.” The trend in this positive family response has increased from 73% to 78% in the past three years, which shows the programs’ effectiveness in achieving their mission of helping families to facilitate their children’s development. The percentage of families indicating “agree, strongly agree, or very strongly agree” for this measure was 87.5%

18. Proposed actions to turn the curve: This data, like the child outcome data, will be publicly posted by program and used to monitor any low-performing programs and drive improvement in how well staff work with families. Additional training for service coordinators will be offered both through Birth to Three and through the UCEDD.

19. Focused Monitoring Use #2 National Center for Special Education Accountability Monitoring Alan Coulter Jane Nell Luster

20. There is something very sad in the disparity between our passion for figures and our ability to make use of them once they are in our hands. -M.J. Moroney, 1951

21. Why? Monitoring resources are in shorter and shorter supply yet there is a greater emphasis on general supervision It is not possible for EI programs or districts to concentrate on the hundreds of requirements in IDEA at all times Programs cannot work to improve hundreds of things at the same time Focusing on important result-oriented priorities can more readily lead to improved results for young children

22. Six Components of Focused Monitoring Stakeholder participation and decision-making Data availability, analysis, and use Integration with the system of general supervision Off-site and on-site monitoring processes Corrective actions, incentives, and sanctions Evaluation

23. Principles of Focused Monitoring A limited number of priorities are chosen by a diverse group of stakeholders A limited number of indicators are identified within each priority area The system is data and information-based and is verifiable Data-based information is used to allocate monitoring resources in the direction of most need

24. Principles of Focused Monitoring The monitoring agency provides supports and imposes sanctions upon programs in order to achieve corrective actions Standard, uniform benchmarks are used for inquiry when making monitoring decisions There is a relationship between monitoring and corrective actions – solutions are linked to identified problems. The system includes clear, known triggers for interventions and sanctions

25. Principles of Focused Monitoring Limited resources are allocated to the areas of greatest need, which are determined by identifying what is most likely to lead to improvements in child performance Available information is used to select priorities that will prove child and family outcomes Monitoring strategy is systemic Corrective actions ensure a change in behavior that results in improved child and family outcomes

26. Principles of Focused Monitoring Monitoring staff are well trained and engage in continuous professional development There is third party evaluation of the monitoring/enforcement system

27. Selecting Priorities & Key Performance Indicators Priority areas should align with SPP Best not to select compliance indicators if possible Indicators must be measureable Data must be available (preferably at least quarterly) otherwise indicator must be deferred until the required data is available

28. Determining priority indicators in 2004 Many priorities... • Family involvement • define • direct/indirect • opportunity to be involved • knowledgeable to help their child • information and decision-making • advocacy • Child Find • identification by diagnosed condition by age • correct assessment, all needs identified • referral age • timely referral new money • families not accepting services • Service provision • timelines of objectives met • withdrawals • competent providers • appropriate services and support • resources/other families • measurable progress • services delivered match IFSP • Transition • date of transition plans • date of referral to LEA • date of transition conference with LEA • Inservice training on Part B requirements • the timely preparation of families for all transitions • training personnel and parents about advocacy for Part B and C • adequate information shared from Part C to Part B • IFSP • child and family outcomes met • resources • objective timelines met • planning

29. Determining priority indicators in 2004 Little data…. • Family involvement • define • direct/indirect • opportunity to be involved • knowledgeable to help their child • information and decision-making • advocacy • Child Find • identification by diagnosed condition by age • correct assessment, all needs identified • referral age • timely referral new money • families not accepting services • Service provision • timelines of objectives met • withdrawals • competent providers • appropriate services and support • resources/other families • measurable progress • services delivered match IFSP • Transition • date of transition plans • date of referral to LEA • date of transition conference with LEA • Inservice training on Part B requirements • the timely preparation of families for all transitions • training personnel and parents about advocacy for Part B and C • adequate information shared from Part C to Part B • IFSP • child and family outcomes met • resources • objective timelines met • planning

30. But in 2009… things changed Priority: Child & Family Outcomes Indicator: Families are more confident and gain new skills which help their children develop and learn. DATA FOR INDICATOR • responses to the family survey measure #32 (Over the past year Birth to Three services have helped me and/or my family figure out solutions to problems as they come up) • child outcome data (C3b) showing the percent of children who closed or reduced the “gap” in their acquisition and use of knowledge and skills (including early language/communication).

31. But in 2009… things changed Priority: Child & Family Outcomes Indicator: Families are more confident and gain new skills which help their children develop and learn. Selection criteria • Programs that are more than 2 SD below the statewide mean are listed as “ very low” • Programs that are 1 – 2SD below the statewide mean are listed as “low”. 0 – 1SD below the statewide mean is “mid” • The first programs selected will be those that are very low in at least one measure. The next programs selected will be those that are low in at least one measure.

32. How might that look?

33. What’s the Monitoring Process? Program is notified that they’ve been selected for focused monitoring At desk audit, monitoring team studies data, looks deeper into the data and related data and forms a hypothesis. (e.g. High rate of staff turnover is why children in this program are not making as much progress as children in similar programs.) Discusses with program administrator Monitoring team goes on-site, follows established protocol that includes looking at records, talking to families, talking to staff, and sometimes talking to LEAs.

34. Protocol excerpts Where to look

35. Protocol excerpts Where to look

36. What’s the Monitoring Process? Monitoring team members keep administrator informed Monitoring team meets at the end of each day to discuss. When all data is gathered, monitoring team meets to discuss hypothesis and any other issues they have discovered Monitoring team meets with administrator(s) for exit interview. Nothing said should be a surprise. TA offered Final report is issued, including findings

37. What’s the Process? Improvement plan (or compliance agreement) written to correct any non-c0mpliance and improve performance. Strategies may include training and TA. Accountability and Monitoring staff track improvement plan timelines and data until data indicates that non-compliance is corrected and performance has improved.