Understanding and Using the Results from the NCSEAM Family Survey

1. Understanding and Using the Results from the NCSEAM Family Survey Batya Elbaum, Ph.D. NCSEAM Measuring Child and Family Outcomes NECTAC National TA Meeting Baltimore, MD, August 2007

2. Part C Indicator #4 “Percent of families participating in Part C who report that early intervention services have helped the family a) know their rights, b) effectively communicate their children’s needs, and c) help their children develop and learn.”

3. Sample results • Mean = 594, S.D. = 147, S.E. = 10 • Percent of families who report that early intervention services have helped the family • a) know their rights = 59% • b) effectively communicate their children’s needs = 54% • c) help their children develop and learn = 65%

4. What do the numbers mean? Interpreting the measures and percentages

5. Measuring the 3 subindicators • To measure something means to locate it on a continuum. • Do we need three different measurement rulers for the three specific OSEP outcomes, or can we use a single measurement ruler?

6. Measuring the 3 subindicators • NCSEAM analyses of responses from thousands of families confirmed that the three outcomes selected by OSEP as family outcome indicators are all part of a single hierarchy of positive family outcomes.

7. Measuring the 3 subindicators • Families appear to achieve these outcomes in a very consistent order. • Families who report that EI helped them know their rights almost invariably report that EI also helped them help their child develop and learn. • Families who report that EI helped them effectively communicate their children’s needs almost invariably report that EI also helped them know their rights and help their child develop and learn.

8. Locating families on the pathway • The data tell us that there is a continuum of “Impact of EI Services on Families”, such that the impact can range from very low to very high. ----------------------------------------------------- Very Low Very High • Different families experience different levels of positive impact as a result of their participation in early intervention, and so different families will be located at different points along this continuum.

9. Locating families on the pathway • When we put numbers on the continuum, we create a “measurement ruler” measuring the impact of EI for each family Fam-1 Fam-2 Fam-3 ↓ ↓ ↓ -----|----------|----------|----------|----- 400 500 600 700

10. Item locations on the ruler

11. Location of key items related to Indicator C4 Indicator 4b: Effectively communicate their children’s needs. IFS Item: “Communicate more effectively with the people who work with my child and family.” [556] 556 Indicator 4a: Know their rights. IFS Item: “Know about my child's and family's rights concerning Early Intervention services.” [539] 539 Indicator 4c: Help their children develop and learn. IFS items: “Understand my child's special needs.” [516] “Do things with and for my child that are good for my child's development.” [498] 516

12. Measurement of family outcomes at different levels • Just as we can locate individual families on the ruler, we can take the average measure for all families in the state and locate the state on the ruler. • Similarly, we can take the average of all families in a particular program and locate that program on the ruler.

13. What is the meaning of the percentage? • The percentage reported to OSEP is the percent of families with measures that are at, or above, a certain point on the measurement ruler.

14. Percent of measures above the adopted standards standards

15. Sample descriptive interpretation • Approximately 90-95% of families agreed, with approximately two-thirds of families expressing strong or very strong agreement, that early intervention helped them: • Understand their child’s special needs. • Do things with and for their child that are good for their child’s development. • Feel that they can handle the challenges of parenting a child with special needs.

16. Sample descriptive interpretation • Approximately 80-90% of families agreed, with approximately 50-60% expressing strong or very strong agreement, that early intervention helped them: • Know about their child’s and family’s rights concerning early intervention services. • Know where to go for support to meet their child’s needs. • Be more effective in managing their child’s behavior.

17. Sample descriptive interpretation • Approximately 70-75% of families agreed, with about 45% expressing strong or very strong agreement, that early intervention helped them: • Communicate effectively with the people who work with their child and family. • Know about services in the community.

18. Sample descriptive interpretation • Approximately 65-70% of families agreed, with about 40% expressing strong or very strong agreement, that early intervention helped them: • Participate in typical activities for children and families in the community.

19. The meaning of “agreement” • The raw percentages of families that agree (in any category of agreement) with the key items related to the indicators is higher than the percentages reported on the indicator. Why? • Because the standard that most states adopted for the indicators is a higher standard than a simple “agree.”

20. Evaluating the percentages on the indicator • Against what standard is the percentage being evaluated? • What percent of families do we want to be at or above the measure that represents our idea of where families should be on the continuum of positive family outcomes?

21. The APR Report on C4

22. What to cover in relation to your data • Sample • Method • Results • Interpretation

23. Sample • Describe the sampling plan • Describe the extent to which the obtained sample is representative of the population • State your survey response rate

24. Method: Instrument • Describe the instrument • Provide a copy of the instrument • Report reliability estimates for the instrument (based on NCSEAM pilot, .90 or above using 22 items)

25. Method: Procedure • Describe how the survey was distributed/administered (face-to-face, mail survey, online) • State whether the survey was available in languages other than English • Describe efforts to ensure the participation of low-literacy families

26. Results: Data analysis • Describe how the data were analyzed • Describe how the percent on the subindicators was calculated

27. Results: Reliability • Report the reliability of the data • Report a confidence interval for the percent on each subindicator (this confidence interval will depend on the size of your sample)

28. Results: Validity • Face validity: Does the instrument appear to address what we are supposed to be measuring? • Convergent validity: Do the results accord with results obtained from other sources (e.g., monitoring data) that address the same thing?

29. Interpretation • Report progress (or slippage) in relation to target • Consider the results in relation to: • Effectiveness of improvement activities • Extent of implementation of improvement activities

30. Improving Performance on the Indicator

31. Short-term goals • Increase response rates • Ensure equal access to participation • Build a broad base of understanding of the indicator and how it is measured

32. Longer-term goals • Improve performance on the indicator • Document implementation of improvement activities • Examine the relation between implementation and improvement so as to better understand “what works” • Improve performance on the indicator for all subgroups

33. Further information • www.monitoringcenter.lsuhsc.edu • elbaum@miami.edu