Session Three Gathering Family Outcome Results in Maryland

1. Measuring Child and Family Outcomes Session ThreeGathering Family Outcome Results in Maryland Anne Brager, MS, RN Program Supervisor Frederick County Infants and Toddlers Program

2. Supporting Young Children and Their Families Early intervention and early childhood special education support young children with disabilities and their families. • For children, the ultimate goal of this support is to enable young children to be active and successful participants during the early childhood years and in the future in a variety of settings – in their homes with their families, in child care, preschool or school programs, and in the community. • For families, the ultimate goal is to enable families to provide care for their child and have the resources they need to participate in their own desired family and community activities.

3. Outcomes According to the Bailey and Bruder reading this week: • An “outcome” is defined as a benefit experienced by families as a result of services and supports received. • An outcome is not about whether services were received or how satisfied families are with services • An outcome is about what happens as a result of services provided to children and families. • Child and family outcomes are interdependent in that positive outcomes experienced by the family serve to promote the child outcomes and outcomes achieved by the child benefit the family.

4. A Shift from Satisfaction to Benefit • Many states and their local programs have been gathering some kind of information from families participating in early intervention for years, but to date, much of that information has focused on satisfaction with services rather than outcomes achieved by families. • The federal government’s requirement for accountability is helping to encourage this shift from focusing on satisfaction with services to benefit for children and families.

5. Family Outcomes for Accountability • As outcomes in an accountability framework, these statements should be taken as desirable accomplishments of the system. • It is understood that a service system cannot guarantee the achievement of any outcome involving families or children. • The achievement of an outcome is the result of a variety of factors, only one of which is early intervention or preschool special education.

6. Strive to Achieve • Even in the best system, it is likely that not all families or children will achieve all of the desired outcomes. • Nevertheless, early intervention and early childhood special education should strive to achieve the outcomes for all the families and children they serve.

7. The Background • The Early Childhood Outcomes (ECO) Center was funded by the Office of Special Education Program to develop an approach for collecting data on child and family outcomes for the Part C early intervention and Part B preschool programs of IDEA (Individuals with Disabilities Education Act). • The resulting data are to be used for federal and state accountability purposes and to improve programs.

8. ECO Center’s Process for Developing Family Outcomes • ECO embarked on a year-long consensus building process that involved input from and review by numerous stakeholders including federal, state, and local policy-makers and administrators, local providers, family members of children with disabilities, and researchers.

9. ECO Considerations: Stakeholders identified two major issues for consideration when developing the family outcomes: • How can a national system to assess family outcomes accommodate the wide range of individualized outcomes? • Should a similar set of family outcomes be expected of early intervention and preschool programs?

10. ECO Process Technical Workgroup on Family Outcomes • Developed priority outcome statements based on commonalities across the literature ECO Constituent Workgroups & Public Comment • Reacted to and tweaked the resulting list of outcome statements

11. ECO Stakeholders As a result of the year-long process, ECO made the following recommendations to OSEP: • Recommended 3 child and 5 family outcomes (Feb 2005, revised April 2005) • Recommended that the same set of family outcomes be used birth through five • Recommended family report of perceptions as the measurement strategy

12. OSEP Picks 3 Family Outcomes ECO • Families understand their children’s strengths, abilities and special needs. • Families know their rights and advocate effectively for their children. • Families help their children develop and learn. • Families have support systems. • Families are able to gain access to desired services, programs and activities in their community. • OSEP • Know their rights • Effectively communicate their children's needs • Help their children develop and learn

13. OSEP Selects 3 Family Outcomes Family Outcomes are Indicator #4 for the State Performance Plans: Percent of families participating in Part C who report that early intervention services have helped the family: A. Know their rights B. Effectively communicate their children's needs C. Help their children develop and learn

14. Family Outcomes Process • ECO recommends outcomes • OSEP finalizes outcomes • States determine measurement system • ECO provides guidance on measurement • Technical assistance • Family ECO survey tool developed

15. State Approaches to Gathering Family Outcomes Data to Report to OSEP There have been three different approaches taken by states to gather the needed data: • National Center for Special Education Monitoring (NCSEAM) Survey –measures family involvement; also produces data for federal reporting • ECO Family Outcomes Survey –measures family outcomes; also produces data for federal reporting • State developed surveys –many had been and continue to use satisfaction surveys

17. State Decisions • ECO Survey: 14 states • State developed: 7 states • NCSEAM survey: 26 states • Undecided: 3 states

18. How to Choose an Approach • This week’s reading by Bailey, Bruder & Hebbler, provides Guidance for States in Documenting Family Outcomes for Early Intervention and Early Childhood Special Education • Challenges to measuring family outcomes are described and recommended solutions are offered.

19. Maryland Chooses theNCSEAM Survey (with modifications) The National Center for Special Education Accountability Monitoring was funded by OSEP to assist states, local agencies, and OSEP in the implementation of focused monitoring and evidenced-based compliance with federal law for improved results for children with disabilities and their families.

20. NCSEAM Task • NCSEAM developed a set of survey instruments that will yield valid, reliable and useful measures of families’ perceptions and family involvement in early intervention and special education.

21. The NCSEAM Family Survey –Early Intervention The family survey for Part C includes 2 scales: • Impact: Impact of Early Intervention Services on the Family – 22 items • Family-Centered Services: – 25 items

22. The NCSEAM Parent Survey –Preschool Special Education The preschool parent survey includes 3 scales: • Efforts/Quality: Preschool Special Education Partnership Efforts and Quality of Services – 50 items • Impact: Impact of Preschool Special Education Services on the Family – 22 items • Participation: Parent Participation – 20 items

23. The NCSEAM Parent Survey –Special Education (school-age) The Part B school-age parent survey includes 4 scales: • Efforts: Schools’ Efforts to Partner with Parents – 25 items • Quality: Quality of Services – 25 items • Impact: Impact of Special EducationServices on Your Family – 22 items • Participation: Parent Participation – 23items

24. Use of a Single Scale vs. bothPart C Scales States using the NCSEAM survey for Part C data collection need to decide whether to use a single scale or both scales: • Families’ responses to items in the scale called Impact of Early Intervention Services on the Family are those needed to address the SPP/APR family outcomes indicator. • However, administration of the additional Part C scale, Family-Centered Services offers two benefits: • continuity of measurement from birth to five • opportunities to examine the association between family-relevant aspects of service delivery and family outcomes

25. How the Two NCSEAM ScalesWork Together • The Family-Centered Services scale is a process measure. It measures the extent to which programs reach out to families and provide high-quality services to children and families, from the family’s perspective. • The Impact on Family scale is an outcome measure. It measures the extent to which families perceive that they have achieved positive outcomes as a result of their participation in early intervention.

26. Measurement and ProgramImprovement • The NCSEAM data suggest that as early intervention programs provide more family-centered services - as indicated by families’ higher measures on the Family-Centered Services scale - there is likely to be a significant increase in positive outcomes for families – as indicated by families’ higher measures on the Impact on Family scale.

27. How the Impact on Family ScaleMeasures Family Outcomes • Data from the NCSEAM National Item Validation Study showed that the relative likelihood of families’ reporting that early intervention helped them achieve different outcomes was very similar across all groups of families.

28. Consistency of Responses to the Items • The consistency held up across families that differed by: • State of residence • ethnicity • age of child • language in which the survey was administered (English vs. Spanish) • method of administration (selfadministered or items read by a facilitator)

29. Items Families Agree with Most Over the past year, early intervention services have helped me and/or my family: • Do things with and for my child that are good for my child's development. • Understand my child's special needs. • Be more optimistic about my child's future.

30. Items Families Agree with Slightly Less Over the past year, early intervention services have helped me and/or my family: • Communicate more effectively with the people who work with my child and my family. • Understand the roles and responsibilities of the people who work with my child and family.

31. Items Families Agree with A Lot Less Over the past year, early intervention services have helped me and/or my family: • Know about my child's and family's rights concerning special education services. • Know where to go for help or support to meet my child's needs.

32. Items Families Agree with Least Over the past year, early intervention services have helped me and/or my family: • Participate in typical activities for children and families in my community. • Know about services in the community.

33. The Importance of a ConsistentPattern of Responses • The consistency of people’s responses to items is what allows us to use these responses as a true measure of the thing we want to evaluate, in particular, for Part C, whether early intervention helped families achieve certain outcomes.

34. Calculation of the PercentReportable to OSEP • We calculate a measure (score) for each completed survey, based on the family’s responses to all the items in the relevant scale. • For the report of state-level performance, we combine the measures of all participating families in the state. • The percent we report to OSEP (and the public) is the percent of families with measures at or above an established standard.

35. How NCSEAM Established a Recommended Standard for eachSub-indicator • NCSEAM convened a stakeholder group with broad representation of families, state and local agencies, advocates, and researchers. • Participants were provided with a list of the items in an order reflecting families’ likelihood of agreeing with each item. (Note: This is not the order in which items appear on the actual survey.)

36. How NCSEAM Established aRecommended Standard for each Subindicator The question we asked was: • What is the highest item with which you would require an “agree” response in order to have confidence that the meaning of the indicator (e.g., early intervention services helped families know their rights) is being achieved?

37. “Threshold” items for Part C Over the past year, early intervention services have helped me or my family: (a) know about my child's and family's rights concerning early intervention services. (b) communicate more effectively with people who work with my child and my family. (c) understand my child's special needs/do things with and for my child that are good for my child's development.

38. What the “standard” Means • The standard is not about agreement with a single item. Given the consistent pattern in families’ responses to the items, a high likelihood of agreement with the threshold item implies the same or greater likelihood of agreement with items located “below” this one on the scale. • The next slide shows the items grouped by threshold indicators

39. PART C IMPACT ON FAMILY Over the past year, Early Intervention services have helped me and/or my family: • participate in typical activities for children and families in my community • understand how the Early Intervention system works. • keep up friendships for my child and family. • know about services in the community. • know where to go for support to meet my family's needs. • be more effective in managing my child's behavior. • know about my child's and family's rights concerning early intervention services. • improve my family's quality of life. • know where to go for support to meet my child's needs. • make changes in family routines that will benefit my child with special needs. • do activities that are good for my child even in times of stress. • get the services that my child and family need. • feel that my family will be accepted and welcomed in the community. • communicate more effectively with people who work with my child and my family. • feel that my child will be accepted and welcomed in the community. • be able to evaluate how much progress my child is making. • feel more confident in my skills as a parent. • understand the roles of the people who work with my child and family. • feel that I can get the services and supports that my child and family need. • understand my child's special needs. • do things with and for my child that are good for my child's development. • feel that my efforts are helping my child.

40. Positive Outcomes Resulting fromFamilies’ Participation in EarlyIntervention Families and children participate fully in the community Parents know their rights and can access services for their child and family Parents work effectively with the professionals who provide services to their child Parents have increased knowledge and skills to help their child develop and learn Parents have greater confidence in themselves and are more optimistic about the future

41. Percent of Families who Report that Early Intervention Helped Them Know Their Rights Far above threshold 75% At or just above threshold Below threshold 25% Far below threshold

42. Implementation Issues Once states chose their instrument, they had more decisions to make: • Will all parents have an opportunity to participate, or will the state use a sample of parents? • If a sample is used, how will the sample be selected? • How will the surveys be disseminated and the data collected?

43. Implementation Issues • What accommodations will be made for parents who are speakers of languages other than English, who have difficulty reading the items, or who have other access issues? • How will the results be reported?

44. All Families or a Sampling? Will all parents have an opportunity to participate, or will the state use a sample of parents? • In June 2006, Maryland Infants and Toddlers Program (MITP) disseminated a baseline survey (the NCSEAM survey customized for Maryland based on stakeholder recommendations) to all families actively receiving early intervention services. • In June 2007, the survey was disseminated to all families actively receiving services.

45. How will the Surveys be Disseminated and the Data Collected? • Maryland contracted with Avatar Inc. to assist with and oversee the process for gathering family outcomes data. • The survey was mailed with a customized, family-friendly cover letter indicating local support contacts for questions or assistance. • A self addressed stamped envelop was included for survey return directly to Avatar, Inc.

46. Accommodations Made? What accommodations will be made for parents who are speakers of languages other than English, who have difficulty reading the items, or who have other access issues? • A Spanish version of the letter and survey was provided to local programs to disseminated as needed • Local contact information for assistance was included in the letter

47. How will the Results be Reported? • The survey is anonymous other than 3 identifying markers: • Ethnicity • Child’s age at time of survey completion • Child’s age when first referred to EI • The individual survey results are combined and reported to OSEP as a representative of the state

48. Maryland’s Baseline Results • The pilot year resulted in a response rate of 16.6%, with 18.8% giving measureable data. • The results indicated that: • 74% of respondents reported that Outcome 1 was achieved (EI services helped their family know their rights) • 72% reported Outcome 2 was achieved (EI helped their family effectively communicate their child’s needs) • 82% reported Outcome 3 was achieved (EI helped their family to help their child develop and learn)

49. What Determines Achieved? • An outcome was determined to be “achieved” if the family marked agree, strongly agree or very strongly agree in response to the targeted survey questions

50. Maryland’s Interesting Facts from the First Year Survey • 1275 out of 6508 families returned the survey • The number of responding families with male children was much higher than those who had female children • Most respondents had entered when their child was between 1-2 years of age and the survey was completed when their child was between 2-3 years • Increasing the response rate is a requirement for each local program in this years Consolidated Local Implementation Grant (CLIG)