Deborah T. Carran, Jacqueline Nunn, Sara Hooks Johns Hopkins University Stacey N. Dammann

1. Uses of a Statewide Longitudinal Data System to Evaluate and Inform Programs, Policies, and Resource AllocationsPresented February 13, 2013 Deborah T. Carran, Jacqueline Nunn, Sara Hooks Johns Hopkins University Stacey N. Dammann York College 26th Annual Management Information Systems [MIS] Conference

2. Background: Linking Data Sets Part C Early Intervention Services data (Birth – 3) Part B Special Education Services data (4, 5 – 21) General Education Services data K Children assessed for K Readiness WSS-K, scaled then scores at 3 levels Readiness (Developing, Approaching, Fully Ready) Maryland State Assessment (MSA), scaled then scored at 3 levels (Advance, Proficient, Basic) • Math and Reading assessments administered annually in grades 3 through 8 2013 MIS Conference

3. 2013 MIS Conference

4. Benefits of Using the Maryland IDEA Scorecard • Data at the State, District, and School levels that drills down to the student level • Data that allows users to identify students in need of targeted interventions in the alert categories (attendance, academics, suspension, and mobility) • Data to create an action plan to monitor student progress within targeted interventions • Reporting functions to support monitoring of progress towards targets of interventions 2013 MIS Conference

5. How to use these data at the state level? • Descriptive • Who do we serve? • Longitudinal • What is the educational placement of children served in EIS by K and Grade 3? • Is Fall K WSS is a successful predictor of later standardized test performance? • Comparative • Is there a difference in standardized achievement performance within educational services (Gen Ed and Sp Ed) for children served in EIS? 2013 MIS Conference

6. Tracking Plan 2013 MIS Conference

7. Method • PARTICIPANTS • 42% tracking match • Missing data • Outcomes • WSS-K (2006-07) • RSAA and MSAA (Spring 2011) • Procedure • Three studies presented 2013 MIS Conference

8. Missing Data • N = 5,328 participants received EIS • 42.1% (n = 2,245) matched • 57.9% (n = 3,083) unmatched (missing data) • Is there a significant difference between matched and unmatched participants on characteristics: gender, Part C eligibility category, Part C MA, Race, age of entry, age of exit, and/or months in EIS? 2013 MIS Conference

9. Gender Results of the Chi-Square statistic assuming equal probabilities indicated no significant difference between Unmatched and Matched participants for gender [X2 (1, N= 5,328) = 0.93, p> .05]. Inspection of cell counts and percentages indicated that the distribution for gender was similar for both groups. 2013 MIS Conference

10. Part C Eligibility Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Part C Eligibility [X2 (2, N = 5,317) = 47.23, p< .001]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were eligible due to a Condition with a High Probability than Matched participants. Conversely, a greater proportion of Matched participants were eligible based on 25% Delay than Unmatched participants. 2013 MIS Conference

11. Part C Eligibility cont. 2013 MIS Conference

12. Part C Medical Assistance Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Part C Medical Assistance [X2 (1, N = 5,278) = 17.49, p = .001]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were eligible for Medical Assistance while receiving EIS. 2013 MIS Conference

13. Race Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Race [X2 (5, N = 5,328) = 16.76, p< .05]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were Black or African American while a smaller proportion of participants in the Unmatched group were Hispanic than in the Matched group. 2013 MIS Conference

14. Entry/Exit Age in Months 2013 MIS Conference

15. Summary – Missing Data • Gender – the two groups are similar • Eligibility – significantly greater proportion of Unmatched participants in High Probability Category • Medical Assistance – significantly greater proportion of Unmatched participants received MA • Race – significantly greater proportion of Black or African American participants in Unmatched group with significantly greater proportion of Hispanic participants in Matched group • Age – Matched participants were significantly younger at age of entry than Unmatched participants. Matched participants received EIS longer than Unmatched participants. Age of exit was similar for both groups. 2013 MIS Conference

16. Outcome Instruments • Work Sampling System-Kindergarten (WSS-K) Assesses 7 Domains, scaled then scored at 3 levels (Proficient, In Process, Needs Development) • Personal and Social Development • Language and Literacy • Mathematical Thinking • Scientific Thinking • Social Studies • The Arts • Physical Development • Reading State Accountability Assessment (RSAA) • scaled then scored at 3 levels (Basic, Proficient, Advanced) • Math State Accountability Assessment (MSAA) • scaled then scored at 3 levels (Basic, Proficient, Advanced) 2013 MIS Conference

17. Procedure • Student data tracking system links records of students in General Ed, Special Ed (Part B) and EIS (Part C) • Identify children in 3rd Grade 2010-11 school year with birth dates between Sept. 1, 2001 and Aug. 31, 2002 • At the state level, student identifiers entered in student data tracking system to obtain student service level (Gen Ed or Sp Ed), student outcomes, and demographic info • Deidentified data provided to researchers 2013 MIS Conference

18. Studies and RQ for this Presentation • Impact of Early Intervention on K Readiness • Who was served by EIS? • Does EIS impact later K Readiness Scores? • Tracking Children Receiving Early Intervention Services (Part C Services Birth to 3) into Elementary School to Grade 3 • What is the educational placement at Grade 3 for children who received EIS? • What is the SAA performance at Grade 3 for children who received EIS? 2013 MIS Conference

19. Different Data Set 3. K Readiness and Grade 3 SAA Performance • Is there a relationship between K Readiness and Grade 3 RSAA and MSAA scoring? • What subscales of the WSS predict RSAAand MSAA scoring? 2013 MIS Conference

20. Study 1: Impact of Early Intervention Services on K Readiness • Does level of service provided to children (Birth – 3) enrolled in early intervention services (EIS) programs enhance their later performance on the Kindergarten Work Sampling System (WSS-K)? 2013 MIS Conference

21. K Readiness: Method • 2,245 children • Who were eligible and received EIS services in MD linked with MD MMSR scores • Born between Sept 1, 2001 and Aug 31, 2002 • EIS services • WSS-K • Summary scaled composite score • Hierarchical Linear Regression 2013 MIS Conference

22. Participants 2013 MIS Conference

23. EIS Services & WSS-K Average Scores Max WSS-K score of 90 2013 MIS Conference

24. Variables Outcome: WSS-K Predictors: • Demographics: FaRMs, Gender, Minority • Earliest Age of Child that Part C Services Begin (-) receive services earlier better prepares child to enter K • Time in Program (+) indicates longer time in program better prepares child to enter K • Total Minutes Services (+) indicates longer time in program better prepares child to enter K 2013 MIS Conference

25. Hierarchical Regression Results 2013 MIS Conference

26. EIS – WSS Conclusions • Demographic Controls: WSS-K was higher for students not economically disadvantaged, higher for girls, and for White students • Age Svc Began (-): For every month earlier a child starts receiving services, he/she is expected to score .017 SD increase on the WSS-K • Supports previous findings • Time in Program (-):inconclusive; possible that children who are in the program for longer times have more severe disabilities • Total Minutes Services (-): inconclusive, possible that children with more severe disabilities will have more/longer services • Correlation Time in Program & Total Min Svc r = .37, p < .001 2013 MIS Conference

27. Study 2 Tracking Children Receiving Early Intervention Services (Birth to 3) into Elementary School by Service Level: How do they Compare with their Peers? 2013 MIS Conference

28. Background • Part C Services (IFSP) • Part b/B Services (IEP) • Little research examining outcomes of children receiving Part C services, mostly Part B services (Cole, Dale, Mills & Jenkins, 1993; Daley & Carlson, 2009; Peterson et al., 2004; Walker et al., 1988) • Focused on developmental progress over short term; not longitudinal or growth trajectories • Enrollment in special education changes for children as they move through elementary school • Limitations (disability, small samples, covariates) 2013 MIS Conference

29. First Steps: Descriptive Analyses • Children in 3rd grade (within birth cohort) • What is their Education Service (Gen Ed, Sp Ed)? • Who were the children receiving EIS? 2013 MIS Conference

30. Participants 2013 MIS Conference

31. Total Sample, Gen Ed and HI Sp Ed *Autism, Deaf, Deaf-Blindness, Developmental Delay, Hearing Impaired, Intellectual Disability, Multiple Disabilities, Orthopedic Impairment, Traumatic Brain Injury, Visual Impairment 2013 MIS Conference

32. 2013 MIS Conference

33. Students in Grade 3 Historically Tracked to EIS 2013 MIS Conference

34. Students in Grade 3 Tracked to EIS, EIS Characteristics 65.5%, n = 1,628 enrolled in general education at grade three 2013 MIS Conference

35. Average State Assessment Scores by Educational Service at K and Grade 3 2013 MIS Conference

36. Hierarchical Regression Analyses Outcome: RSAA Grade 3, MSAA Grade 3 Three Models: Model 1, 2, 3 Demographics: FaRMs, Gender, Minority Model 2, 3 Part C EIS Model 3 EIS x FaRMs EIS x Gender EIS x Minority 2013 MIS Conference

37. Hierarchical Regression Results: WSS-K *p ≤ .05, **p ≤ .01 2013 MIS Conference

38. WSS-K Results General Ed • WSS-K higher for students notFaRMs, for girls, and for White students • EIS students scored lower (.7 M diff) • No interaction effects Special Ed • WSS-K higher for students not FaRMs, for girls, and for White students • EIS students scored lower (.8 M diff) • Sig interaction EIS x Gender • F > M NO EIS (2.9 M diff) • M > F EIS (1.7 M diff) 2013 MIS Conference

39. Hierarchical Regression Results: RSAA *p ≤ .05, **p ≤ .01 2013 MIS Conference

40. RSAA Results General Ed • RSAA higher for students notFaRMs, for girls, and for White students • EIS students scored lower (3.1 M diff) • No interaction effects Special Ed • RSAA higher for students notFaRMs and for White students • EIS students scored lower (22.3 M diff) • No interaction effects 2013 MIS Conference

41. Hierarchical Regression Results: MSAA *p ≤ .05, **p ≤ .01 2013 MIS Conference

42. MSAA Results General Ed • WSS-K higher for students notFaRMs, for girls, and for White students • EIS students scored lower (1.3 M diff) • Sig interaction EIS x Gender • F > M No EIS (1.9 M diff) • M > F EIS (4.2 M diff) Special Ed • WSS-K higher for students not FaRMs, for girls, and for White students • EIS students scored lower (16.2 M diff) • Sig interaction EIS x Gender • M > F No EIS (10.4 M diff) • M > F EIS (42.9 M diff) 2013 MIS Conference

43. General Ed EIS SAA Conclusions • Demographic Controls: WSS-K, RSAA, & MSAA was higher for students not economically disadvantaged, higher for girls, and for White students • EIS: WSS-K, RSAA, & MSAAwhen FaRMs, Gender, and Minority are controlled, students who received EIS scored lower than their Gen Ed peers. • EIS Interactions: WSS-K & RSAA no significant interactions; MSAA EIS x Gender (F EIS) 2013 MIS Conference

44. Special Ed EIS SAA Conclusions • Demographic Controls: WSS-K, RSAA, & MSAA was higher for students not economically disadvantaged and for White students; females scored higher on WSS-K & MSAA • EIS: WSS-K, RSAA, & MSAAwhen FaRMs, Gender, and Minority are controlled, students who received EIS scored lower than their Sp Ed peers • EIS Interactions: WSS-K & MSAA significant EIS x Gender (F EIS lower in K and more at Grade 3) 2013 MIS Conference

45. Youth in EIS have lower average scores than their peers in Gen Ed and Sp Ed 2013 MIS Conference

46. Female EIS in Sp Ed (HI) Services have lower average scores than their peers 2013 MIS Conference

47. Study 3: K Readiness and Grade 3 MSA Performance Is performance on the WSS-K predictive of Grade 3 high stakes testing (Reading MSA and Math MSA)? 2013 MIS Conference

48. Background Maryland Model for School Readiness (MMSR) Pearson Work Sampling System Assessment Assesses 7 Domains, scaled then scored at 3 levels (Proficient, In Process, Needs Development) • Personal and Social Development • Language and Literacy • Mathematical Thinking • Scientific Thinking • Social Studies • The Arts • Physical Development 2013 MIS Conference

49. Method • Participants/Procedure • General Ed Fall K Work Sampling System student scores (2003, 2004, 2005) N = 152,105 Matched to • Grade 3 Spring MSA Math, MSA Read student scores (2006, 2007, 2008) N = 100,958 Match Rate 66% for Reading & Math • Instruments • WSS is a 30 item instrument with scaled items coded 1 - 3 • MSA is a high stakes test coded 1 - 3 2013 MIS Conference

50. Kindergarten MMSR Composite Score Distribution by Grade 3 MSA Performance Level 2013 MIS Conference