420 likes | 580 Views
Cultural Validity in Assessment Practices. Guillermo Solano-Flores American Institutes for Research Mid-Atlantic Equity Center Annual Regional Conference, Washington, DC, March 12, 2004. Traditional: Classifies individuals into large categories Focuses on differences between groups
E N D
Cultural Validity in Assessment Practices Guillermo Solano-Flores American Institutes for Research Mid-Atlantic Equity Center Annual Regional Conference, Washington, DC, March 12, 2004
Traditional: Classifies individuals into large categories Focuses on differences between groups Uses adaptation and accommodation of tests Centralized Deficit model view Alternative: Defines a socio-cultural context Focuses on score dependability Emphasizes the process of test development Community-based Multidisciplinary perspective Paradigms in testing
Overview • The need for interdisciplinary approaches to test design • Understanding how culture and language influence test taking • Implications: new paradigms in testing
First NSF study: Assessing assessment cultural validity • Participants: Students from twelve sociocultural contexts (ethnicity, language background, SES, geographical area, locale, type of school) • Students were given items used in standardized testing that are supposed to be “culturally-sensitive” • Students were asked to explain how they interpreted the items and how they related their content to their personal experience
Second NSF study: Sociolinguistic perspectives in testing • Participants: Students from different geographical areas who are assumed to speak different dialects of the same language; teachers who are familiar with the dialect spoken by their students • Teachers from different sites adapt the same set of items based on their knowledge of their students’ dialects • Students take the tests in both dialect versions
Pressing issues in the testing of linguistic and cultural minorities • Accountability based on standardized test scores • Bilingual and multicultural education under attack
Limitations of current approaches to testing linguistic and cultural minorities • Lack of theories of language and culture • Testing practices driven by erroneous assumptions about language and culture • Erroneous assumptions about the effectiveness of current testing practices • Recent, cognitive approaches to testing overlook the important cultural influences on cognition
What is the least number of $1.00 bills that his mother should give him so he will have enough money to buy lunch for 5 days?
Major (yet preliminary) finding: The syntactical structure of some test items is unnecessarily complex
Socio-cultural activity questions How do you see [this item] as part of... • …what you do when you are not at school? • …what you do for fun when you are at school? • ...your school day in the classroom? • ...any traditions that you have?
World views: a low-income student’s response to the Lunch Money item Interviewer: What is this item about? Student: It’s about Sam, trying to get her lunch, but her mom only has one dollar, and she needs more for five days, so I think she should give her a dollar ninety-five.
Cultural factors relevant to assessment • Epistemologies -- ways of constructing knowledge and making sense of experience • Teaching and learning styles -- ways of transmitting and acquiring knowledge • Discourse styles -- ways of expressing ideas
Cultural influences in assessment Different cultural backgrounds produce different ways in which students: • interpret what a test item is about • use their knowledge and experience to solve problems • demonstrate their knowledge
Patterns of language dominance diversity: A more realistic view
Assessing linguistic minority students: Common mistakes • Underestimating students’ English proficiency • Overestimating students’ proficiency in their native language • Lowering academic standards and expectations
Difficulty reading “$1.00 bills” in “His mother has only $1.00 bills.” Identified by linguists: Yes Anticipated by at least 20% of teachers: No Observed in at least 20% of students: Yes Statistically significant differences between groups: Yes
Difficulty reading “least” in“What is the least numberof $1.00 bills…?” Identified by linguists: Yes Anticipated by at least 20% of teachers: Yes Observed in at least 20% of students: No Statistically significant differences between groups: No
Why we should pay more attention to the process of test development
Validity and high-stakes testing Information on the validity of a test for a given population of students is usually available after decisions affecting those students and based on the scores from that test have been made.
Traditional: Psychometrics Cognitive psychology Other: Cultural anthropology Sociolinguistics Structural linguistics Reading Disciplines relevant to test development
Conclusions Current approaches to testing English language learners do not address effectively the fact that assessments are extremely sensitive to wording
Implications for assessment • New paradigms in testing • Multidisciplinary approaches • Combining quantitative and qualitative methods
Exercise 1: Gumball machine The gum ball machine has 100 gum balls; 20 are yellow, 30 are blue, and 50 are red. The gum balls are well mixed inside the machine. Jenny gets 10 gum balls from this machine. What is your best prediction of the number that will be red?
References (1) • Solano-Flores, G., & Trumbull, E. (2003). Examining language in context: The need for new research and practice paradigms in the testing of English-language learners. Educational Researcher, 32(2), 3-13. • Solano-Flores, G. (2003). The multidimensionality of test review and test design: A conceptual framework for addressing linguistic and cultural diversity in testing. Paper presented at the !0th Biennial Conference of the European Association for Research on Learning and Instruction, Padova, Italy - August 26 – 30, 2003. • Solano-Flores, G., Trumbull, E., & Nelson-Barber, S. (2002). Concurrent Development of Dual Language Assessments: An Alternative to Translating Tests for Linguistic Minorities. International Journal of Testing, 2(2), 107-129.
References (2) • Solano-Flores, G., & Nelson-Barber, S. (2001). On the cultural validity of science assessments. Journal of Research in Science Teaching, 38(5), 553-573. • Solano-Flores, G., Lara., J., Sexton, U., & Navarrete, C. (2001). Testing English language learners: A sampler of student responses to science and mathematics test items. Washington, DC: Council of Chief State School Officers.