Best and Emerging Practices in Data Driven Decision-Making

1. Best and Emerging Practices in Data Driven Decision-Making Philip A. Streifer, Associate Professor (The University of CT) and President (EDsmart division of PCG, Inc.) Sue Gamm, Educational Consultant, Public Consulting Group

2. Topics: • Critical Issues – Disproportionality • What is Data Driven Decision-Making • How does it work? • Advances in analytic techniques

3. Disproportionality: Background Inappropriate identification and overrepresentation of African American students is not a new problem Promise of P.L. 94-142 30 years later

4. Issues Disproportionate representation of minorities in: • Special Education • Various disability eligibility areas • More restrictive placements • Suspensions and expulsions Variance depends on impact of disability

5. National Research Council • Minority Students in Special and Gifted Education 2001 • http://books.nap.edu/catalog/10128.html

6. Impact: Poverty “We know that minority children are disproportionately poor, and poverty is associated with higher rates of exposure to harmful toxins, including lead, alcohol, and tobacco, in early stages of development. Poor children are also more likely to be born with low birthweight, to have poorer nutrition, and to have home and child care environments that are less supportive of early cognitive and emotional development than their majority counterparts. When poverty is deep and persistent, the number of risk factors rises, seriously jeopardizing development.”

7. Impact: Instruction “While children come to school from disadvantaged backgrounds, as a disproportionate number of minority students do, high-quality instruction that carefully puts the prerequisites for learning in place, combined with effective classroom management that minimizes chaos, can put students on a path to academic success. There are currently no assurances that children will be exposed to effective instruction or classroom management before they are placed in special education programs…”

8. Impact: Wait to Fail • Substantial evidence that early identification and intervention more effectively impacts behavior and achievement • “Wait-to-fail” principle increases likelihood children will fail because they don’t receive early supports and decreases effectiveness of supports once received • Early identification and intervention for children at risk for later failure helps all children who need additional supports. Disproportionately large number of those students expected to be from disadvantaged backgrounds

9. Impact: IDEA 2004 The State has in effect, consistent with the purposes of this title and with section 618(d), policies and procedures designed to prevent the inappropriate over identification or disproportionate representation by race and ethnicity of children as children with disabilities, including children with disabilities with a particular impairment described in section 602.

10. Impact: IDEA 2004 State reviews data to determine if significant disproportionality based on race and ethnicity is occurring regarding – • Identification of children with disabilities, including any particular area • Placement in restrictive settings • Incidence, duration, and type of disciplinary actions, including suspensions and expulsions. .

11. Consequence Require any LEA identified to reserve 15% of IDEA funds to provide comprehensive coordinated early intervening services to serve children particularly - but not exclusively - in significantly over identified groups.

12. Data Driven Decision-Making • What is it? • How do you do it? • What tools are needed? • What can you learn? • Advances in analytic procedures

13. An Exploration…Who is Likely to Do Well? What are Contributing Factors? - B Disaggregating Disaggregating Grades in class? Lower Academic Level? Who are They? Organizational What Special Poor Performance Courses How Many Ed? are they Students? Curriculum taking? C Relationship to Prior Tests CMT PSAT Reading CAPT Verbal DRP Literature scores Scores Lit Scale A Scores Longitudinal

14. Before A Data Warehouse

15. With A Data Warehouse

16. Cohort: Class of 2001 Transcript SAT PSAT Attendance G12 00-01 00-01 N/A 00-01 G11 99-00 99-00 99-00 99-00 G10 98-99 98-99 N/A N/A G9 97-98 97-98 N/A N/A

17. Attendance SAT PSAT Transcript Class of 1997 Class of 1998 Class of 1999 Class of 2000 Class of 2001 Can we determine relationships between data? Are district or school test scores improving over time? G12 G11 G10 G9 Are students’ individual grades or test scores improving over time?

18. Data Warehouses are BIG and COMPLEX

19. EDanalyzer™ -- Quick and Easy Access to Data • Identify students ‘in need’ under NCLB/AYP • Across subject areas and learning strands • Chart progress • Student progress can be disaggregated by a host of variables, such as attendance and class grades, showing the relationship between school variables and test scores

21. How would you perform this analysis? • Target: 10th Grade Mastery Test • Predictors: courses, teachers, demographics, parental information, special program participation, previous test scores (CMT, TASA, Teranova, DRA, etc.), class grades, teacher comments. • Variables are a variety of score types: scale, ordinal, nominal, text • Goal: use all variables in one analysis

22. How would you perform this analysis? • What are the indicators of student success? • For those below standard in 4th grade but above standard in 10th grade? • Can we isolate program participation as an indicator of student success? • What sequence of teachers and/or courses are the best indicators of student success?

23. Shift in Purpose of Analysis • From – Hypothesis testing • To – Exploratory data analysis • Where is this headed?................

24. Can we Predict the Future (to Identify Interventions that Work)? • Develop a predictive model on 10th graders from 2004 and apply that ‘model’ to current sixth graders to determine how well they perform if no intervention. • Prior to AI tools; not possible • With Artificial Intelligence Predictive Analytics – Yes; Research to Practice

25. Examples in Other Sectors • Richmond, VA Police Department • Nassau County Medicaid Administration • US Government: Sandia National Labs • Credit Card Companies • Text Mining – Accident Reports • Other Applications in the Future: Medical Research

26. A New Look at Value Added Teacher Effectiveness Analysis Applying Artificial Intelligence/Data Mining Techniques

27. Barriers… • structure and quality of data held in data warehouses • VAM analyses do not include school process variables such as course/program information and the like (RAND 2003) • Missing variables (RAND 2003) • tools to perform the proper analyses

28. AI Data Mining Tools • Purpose: for exploratory data analysis (not hypothesis driven analysis) • Handles multiple variable types in one analysis • Can predict missing variables from dataset • Resolves barriers noted by RAND (2003) • Classification & Regression Trees

29. Classification & Regression Trees • Uses recursive partitioning to split the records into segments with similar outcome values • Examines the input fields to find the best split, measured by the reduction in an impurity index that results from the split • The extent to which subgroups have a wide range of output field values within each group. • Goal: create subgroups that tend to have the same or similar output values – minimize the impurity of the tree • Split defines two subgroups, each of which is subsequently split into two more subgroups, and so on, until one of the stopping criteria is triggered

30. Education Data • Inputs – race, SES, etc. • Process – programs, courses, teachers • Outcomes – test scores and grades, etc. By using Process variables we can begin to limit the impact of confounding variables – a problem noted by RAND

31. Problem • Does teacher emerge as the strongest predictor among all in the dataset? • N = 121 Students • Kindergarten through end of Second Grade

32. Variables • Target: Gates-MacGinitie reading comprehension test NCE score (spring test) • Predictors: • gender (nominal) • date of birth (ordinal) • ethnic code (nominal) • kindergarten experience (nominal – yes or no) • readiness (nominal – yes or no) • parent/guardian relationship (text – mother, father, parents, guardian) • absent days for 2002, 2003 and 2004 (separate variables by year – ratio scale) • teacher-ID-school (separate variables by year 2002, 2003 and 2004 – teacher name and school) • second grade writing score (text-ordinal such as basic, proficient, etc. – a locally developed and scored writing assessment – a teacher test) • and reading program that children were assigned to in 2002, 2003 and 2004 (text – by name of reading program)

33. DTREG Output (selected) • Percent Variance Explained • Relative Importance of Predictors • Tree – (for rule induction)

34. G2_2004

35. All Variables: 96.2 % of variance explained

36. Remove date-of-birth: 93.8 % of variance explained

37. Remove date-of-birth and reading program: 92 % variance explained

38. AI Data Mining and Disproportionality • If there are clues in school data as to what interventions work, these techniques could identify them • Teacher effects can be isolated but with difficulty • These analyses will always be imperfect; DDDM is inherently imperfect • DDDM is a compass heading…no more…but when lost in a sea of data, a compass can be very helpful