Fixing a Broken Model

1. Fixing a Broken Model How to Use Significance Testing to Assess for Disproportional Representation

2. Presenters • Clifford V. Hatt, Ed.D., ABPP • Coordinator, Psychological Services, Virginia Beach City Public Schools, Virginia • Jon C. Thompson, Psy.D., HSPP • Pediatric Neuropsychologist St. Vincent’s Hospital, Indianapolis, Indiana

3. What’s “Broken”? • No way to tell if two different percentages are really different • Using the “white” student population as the comparison group • Using “adjustment” amounts to determine if the percentages are meaningful • Considering a disproportion instead of incidence of a disorder

4. The Questions and Issues The What and The How of Disproportionate Representation

5. Proportional Concept • Assumes same ethnic proportions will/should be reflected in all educational situations • Special Education Classifications • Referrals for educational assistance • Discipline referrals • AP classes • Drop out rates • Seems simple and self-evident

6. Disproportion vs. Incidence • Do disabilities and illnesses occur in the general population and in subpopulations at the same rate? • If the incidence of mental retardation occurs in 1% of the general school age population (based on USDOE data), can we expect it to occur at that same rate in specific school-aged ethnic groups?

7. 2004: % of Students Age 6-21 Served Under IDEA, Part B

8. Incidence Rates Reported in DSM-IV-TR Learning Disability: Approximately 5% of students in public schools are diagnosed with a learning disability (reading, written language, math) Mental Retardation: Approximately 1% of general population is diagnosed with mental retardation

9. Is Special Education Data Consistent?

10. Consistency of Comparative Data • If Special Education Data is consistent, which incidence rate should be used for comparison? • United States Data • Virginia Data • DSM-IV-TR

11. For every complex problem there is an answer that is clear, simple, and wrong. H. L. Mencken (1880-1956)

12. Problems with the Methods • Risk Index • Number of students from a given ethnic background identified with a disability/Total number of that ethnicity in the general student population • Similar to incidence rate • Only descriptive analysis

13. Problems with the Methods • Odds Ratio (Relative Risk Index) • Uses the Risk Index of an ethnic minority as the nominator and the Risk Index of whites as the denominator • Gives simple comparison of groups with whites serving as the control group • Can determine whether a particular ethnic group will be identified as having a disability more or less often compared to whites

14. Problems with the Methods • Odds Ratio (Relative Risk Index) • No way to evaluate for significance • Assumes whites are the majority and are representative of a disability group as a whole • Assumes all disabilities are equally represented across ethnic groups

15. Problems with the Methods • Composition Index • Calculated by dividing the # of students of a racial or ethnic group in a category by the total # of students in that ethnic or racial group in the total student population. • Provides composition for a pre-determined population so that when all compositions of the different ethnic groups are added together, total equals 100%

16. Problems with the Methods • Composition Index • Composition of the identified category is then compared to composition of the population as a whole • No way to evaluate for significance • Highly susceptible to interpretation flaws when population under consideration is small • Assumes all disabilities occur with equal frequency across all populations

17. Arbitrary Numbers • It may be recognized that composition ratios are not very useful when comparing subgroups. • Some type of “statistical” adjustment is often suggested without any clear rationale. • Some states may use “+/- 20 percentage points” to suggest significant disproportionality (Coutinho & Oswald, 2004).

18. A “Fix” for the Problems of a Broken Model • What if you could determine if two percentages (proportions) were really different? • What if you could compare local with state data and state data with federal data using the same method? • What if you could track progress made after interventions were implemented?

19. A “Fix” for the Problems of a Broken Model • What if you actually saw measurable differences in what you did instead of always never quite measuring up? • What if this all made sense?!!!

20. One Solution • Determine the significance of the difference between two proportions (percentages are just a special type of proportion) • Determine if real statistical differences are found • Use a z-score as the common measure

21. Standard normal distribution -2 -1 0 1 2

22. Within 2* SDs (* really 1.96) 95% -2 -1 0 1 2

23. Formula Key

24. Formula for z

25. Formula in Special Ed Terms • z = (% min spec ed - % min students)/ sqrt[(x * 1-x)*(1/n1 + 1/n2)] • x = (# min students + # spec ed min students)/(n1 + n2) • n1 = total # minority students • n2 = total # spec ed students

26. A Simple Way to Do It • Excel spreadsheet with the formula imbedded Excel compared to state population • Enter the number of minority students in special ed and the number of minority students in the local student population • Select a comparison group: state data or national data • Enter the respective information

27. Results for Virginia

28. Implications • Different choices in the use of the comparison group make a difference! • The method you chose to compare data makes a difference! • Disproportionality is a problem regardless of method. • More specific methodology allows for more appropriate, targeted remedies. • Can be easily calculated in an Excel spreadsheet format.

29. Implications • Modified z-score method provides a way to determine whether the differences between predicted and obtained proportions are meaningful. • More appropriately identifies districts that need to have corrective action. • Also provides a way to determine progress or change following interventions. • Accounts for differences in size or number of students under consideration.

30. A Little Comic Wisdom • Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong remedies. Groucho Marx (1890-1977)