Dis-proportionality and Best practices in Un-biased Assessment

1. Dis-proportionality and Best practices in Un-biased Assessment • Steve Hirsch, PhD, NCSP WSU –EWU School Psychology program coordinator East Valley, School Psychologist smhirsch@wsu.edu

2. Best & Realistic Practice in Non-Discriminatory Assessment • Completely unbiased assessment is an illusion. • “Non-discriminatory assessment is not a search for an unbiased test but rather a process that ensures every individual, not just those who are deficit in some way, is evaluated in the least discriminatory manner possible.” Ortiz, 2004

3. Best & Realistic Practice in Non-Discriminatory Assessment • What is needed is a comprehensive, systematic framework comprising a broad range of methods and procedures. • But… just as I am ready to dismiss standardized testing as invalid, I will present an intriguing model that is logical and allows for the use of standardized tests of cognitive functioning

4. Workshop Itinerary • The concept of dis-proportionality • The etiology of the concept • Cultural and Linguistic competency • Inadequacies of standardized tests • Value of formative assessment • Assessment of the Learning Ecology-FAAB • Our traditional attempts to reduce bias • The matrix of cultural/linguistic factor loading in cognitive assessment

5. Dis-proportionality only exists in the ‘soft’ handicapping categories (hard) Deafness Blindness Orthopedic impairment Severe mental retardation (soft) Specific learning disability Mental retardation Emotional disturbance

6. In Washington… Enrollment % of pop. in SpEd Child Count 2.7% 4.0% 16.4% Am Ind /AN 8.7% 4.7% 6% Asian/PI 5.7% 7.7% 14.8% Black 14.6% 14.6% 10.6% Hispanic 65.8% 68.9% 10.7% White

9. State Totals

10. Washington State Summary Adequate Yearly Progress 2007-08

11. Class of 2008 Washington State Dropout Rates

12. Doubt is uncomfortable, certainty is ridiculous. -Voltaire (François-Marie Arouet)

13. Where are we going wrong? From whence comes dis-proportionality? • Assessment/identification process? • Inaccessibility of the general curriculum to some ethnic minorities? • Is it a result of pre-conceived stereotypes? • Could it be that some populations are actually more prone to the deficits in the psychological processes that define a learning disability (i.e. memory, attention, processing speed, language, motor)?

14. Debate: How is dis-proportionality in the category SLD occurring? • Given • The most dis-proportionate populations have FS average scores significantly below whites on IQ testing (the tests are biased that way) • To be identified as LD you must have a significant discrepancy between IQ and achievement on standardized test • We all know that it is nearly impossible to qualify a child with borderline to low average IQ How then are we qualifying a disproportionate number of blacks and Native Americans as LD? “Professional Judgment”?

15. How did we get to this point? Could dis-proportionality be a result of how we view special education? • From WAC, special education represents ‘brokenness’ of the nervous system. “Keep ‘em out of sped” • To the ELL parent, special education may represent what it was in the previous culture. “keep ‘em out” • To an advocacy group, special education often represents a label, stigma but protection in form of IEP “keep ‘em out unless they qualify” • To the team, special education often represents additional help usually in a small group or individual setting. E.g. tutoring. “put them in”

16. Could dis-proportionality be a result of how we view special education? • If special education is viewed as a way of helping a child using resources not available to the general education community, or is viewed as protection in form of IEP, then how could we not expect certain minority populations to be over-represented? • They don’t test academically as well on our standardized achievement tests and therefore are more likely to look as if they need extra help.

17. Best & Realistic Practice in Non-Discriminatory Assessmentala APA • Consider influence of language and culture on behavior • Consider validity of methods and procedures • Make interpretations of data within context of a student’s linguistic & cultural characteristics When in doubt, hide behind vagueness

18. Needed: Cultural & Linguistic Competency • Cultural competence- knowledge base of, or direct experience (with values, attitude, beliefs and customs of particular culture) that will guide data collection & analysis. • Doesn’t come from a book or taking a trip or even looking out your kitchen window at another country. Look for professional development or experiences with community or advocates

19. Willingness To see and Accept help Family/school Acculturation indicators Perception of Disability Discipline

20. Needed: Cultural & Linguistic Competency • Linguistic competence: Ability to communicate effectively in individual’s native language (no interpreter)---i.e. multi-lingual Typically not reality so next best thing: Possession of a knowledge base related to first and second language development and instructional methodology as it relates to ELL

21. And if you are going to use standardized tests-skills you need to know: • Adequacy of sampling norms • Full range of abilities being measured in subtests • Linguistic demands and cultural loading • Reliability and validity of “cultural-fair tests” • Sources of potential bias

22. And if you are going to use standardized tests-keep in mind that • Published tests of cognitive ability are often viewed as psychometrically non-biased due to sampling norm that mirrors census data. • Acculturation impacts appropriateness of tests Decisions must be made on individual basis

23. Debate: Is a particular cultural group ever adequately represented in a normative sample? • If 3% of test sample represents a group that constitutes 3% of the population, does that mean that the test is appropriate for your particular student? • The 3% in test sample-to what extent are they representative of their cultural population? • To what extent is your student representative of their cultural/linguistic population? • Just because a culture is represented in a sample doesn’t mean that the test is fair to the culture

24. Adding insult to injury • If you look at sampling norm data in test manuals, you typically find the following (TONI-3, 1997): • Carefully stratified across US geographic regions (northeast; midwest; south; west-census matched) • Ethnicity categories- African American; Hispanic; Asian; Native American. • While they are careful not to conclude that a mid-westerner is similar to a westerner, they assume that Chinese, Vietnamese, Korean, Hmong, Indonesian, Filippino, Samoan, Indian, Japanese and Hawaiian among hundreds of other cultures are all identical

25. Other important issues • Degree to which the test is linguistically loaded • Limited verbal load (WISC perceptual organization) • Non-verbal Minimal verbal interaction (KABC) • Language-free (UNIT) • Degree to which a test, even if language-free, is culturally loaded • Bottom line: we know very little how different bilingual populations perform on tests normed on children from monolingual single-culture environment

26. What assessment tools are available, and what could be assessed: Cognitive • Reduced Language Load (WISC-PO index) • Would you want the world to believe that your IQ is solely a function of non-verbal subtests? Not me! • Non-verbal assessments (KABC-II; DAS) • If instructions frame problem-solving strategy but instructions are in English… • Language-free assessment (UNIT) • But cognitive functioning is in large part, verbal

27. What assessment tools are available, and what could be assessed: Academic • Translated assessments e.g. Woodcock-Munoz • Problem is that bilingual children may never received formal instruction in their native language. Academic achievement must be viewed carefully • Curriculum Based Measurement (ORF) when used with ELL students: • Has been shown to have good reliability • Has been shown to have good validity • Has been shown to be a reliable progress monitor

28. Academic Assessment of ELL: Life is good-its almost time for Gonzaga basketball epiphany • Global norm-referenced tests invalid • Translated versions of tests-questionable • ORF susceptible to word-calling • What if we looked at more than the correct number of words read per minute during DIBELS? What if we focused on a composite of speed, accuracy and prosody? True Oral Reading Fluency. Will it correlate with overall reading or WASL reading? Do ELL students improve in prosody? Need research!

29. Academic Assessment of ELL: fear of shoveling 10 ft of snow epiphany • We have all sorts of benchmarks for ORF but none reflect the ELL student. Are there unique learning trajectories in ORF for ELL students based on their English language proficiency? But how does acculturation degree factor into this?

30. What assessment tools are available, and what could be assessed: Instructional Environment • Do students have sufficient background to understand content of instructional materials? • Can students relate to content of instructional materials from cultural perspective? • Is curriculum being adapted to needs of bilingual students?

31. What assessment tools are available, and what could be assessed: Instructional Environment • A difficult task: Evaluate teacher expectations of bilingual students and how it might impact instruction? • Do we have adequate benchmarks to assess the progress of ELL students against? • Interesting study for someone out there: How are teacher’s expectations of a student influenced by student’s cultural background?

32. Do teachers have different expectations of children of differing cultural backgrounds? • Based on your interactions with children of these cultural backgrounds, rank them in terms of the likelihood of this child returning homework daily ___Russian ___Hmong ___Samoan ___Vietnamese ___Hispanic ___Chinese ___Native American

33. What assessment tools are available, and what could be assessed: Instructional Environment • The Real Danger: Shared 40 yrs ago • “Standardized tests label black children as less educable; place black children in special classes; perpetuate inferior education; assign black children to lower education tracks than whites; deny black children higher educational opportunities and destroy positive intellectual growth and development of black children.”

34. Are standardized tests really that poor for ELL students? • Norms usually limited to small samples of minority children • Norms routinely exclude students with limited English proficiency • Test items tap info that minority children may not be familiar with • Tests typically don’t allow examiners to probe and inquire further • Correct items on tests usually based on majority children’s responses • Standardized testing procedures assume that students have appropriate test-taking skills

35. What about translated tests? • Not really the answer • Sometimes vocabulary and concepts not easily translated (e.g. Yiddish word, ‘mensch’ not easily translated to English) • Difference in dialects result in translation errors • Some concepts change meaning once translated

36. So if not standardized tests, what ? Then we will look at Best Practices in non-discriminatory assessment- A comprehensive framework

37. For every complex problem there is an answer that is clear, simple, and wrong. H. L. Mencken (1880-1956)

38. Comprehensive Framework • Evaluate the learning ecology • Evaluate language proficiency • Evaluate opportunity for learning • Evaluate educationally relevant cultural and linguistic factors • Develop, and revise hypotheses

39. Assessment of learning ecology • Exploration of extrinsic factors that might lead to difficulties • Functional Assessment of Academic Behavior (FAAB) • Interviews w/parent, teacher, child • Observations in multiple settings • Assess home, classroom teacher/student interaction; home/school interaction

40. Assessment of opportunity for learning-specific factors to be examined • Regularity of attendance • Experience with school environment • Language match (native vs. English) • Parent’s ability to support language instruction • Years of instruction in English

41. Assessment of opportunity for learning-specific factors to be examined • Quality of ESL instruction or instruction in native language • Frequency of school change • School attitude & expectations re. dual language learners • Socialization vs. isolation from peers

42. Assessment of educationally relevant cultural and linguistic factors • Learning takes place everywhere, not just school. Many factors outside educational setting can affect the learning process. • Exposure to two or more cultures or languages during early childhood may create circumstances that cause child to have very different experiences than other children.

43. Evaluate and revise hypotheses • Only when enough confidence exists in the belief that there are no plausible external factors that could account for student’s learning difficulties is it time to consider internal factors. • There should also be evidence that efforts to reduce impact of external factors through intervention were unsuccessful.

44. So far • The above steps should all be part of a pre-referral process not part of the special education assessment

45. Reduce bias in traditional testing practice • Bias is due to non-representational sampling- key is acculturation and English proficiency. Even with native language tests, bias remains. Two traditional approaches to reduce bias:

46. Traditional approaches to reduce bias: • Administer tests in standardized way-attempt to evaluate results in non-discriminatory manner • Use of existing locally developed norms-more appropriate comparison group. Standardization then helps for comparison • Deviations from standardization would produce results that are unknown and unpredictable as does translated tests • Test selection based on properties relating to cultural loading

47. Traditional approaches to reduce bias: • Modify testing process in a way that is less discriminating • Any modification will of course violate standardization and impact validity and interpretability of results • You also lose ability to use results for comparison purposes • We typically don’t know impact of any modification • We could look at data qualitatively

48. Traditional approaches to reduce bias: some modifications • Bilingual administration • Use of extended-expanded instructions on sample items • Mediation of concepts to ensure comprehension prior to test administration • Repetition of items to facilitate comprehension

49. Traditional approaches to reduce bias: some modifications • Extension or elimination of time limits • Acceptance of alternate responses e.g. non-verbal gestures • Additional probing and querying of incorrect or partially correct answers

50. Utilize authentic assessment procedures • Whereas standardized norm-referenced tests are driven by questions and needs related to classification, diagnosis and legal eligibility, authentic assessment is geared more towards answering questions regarding instructional needs and interventions.