Outline

1. Outline • Test bias – definitions • The basic issue: group differences • What causes group differences? • Arguments that tests are not biased • Differential item functioning analysis • Criterion-related sources of bias

2. Outline • Other approaches to testing minority groups • Chitling test • BITCH test • SOMPA • Models of test Bias • Regression • Constant Ratio • Cole/Darlington • Quota

3. A test is biased if it gives a systematically wrong result when used to predict something. So, an intelligence test would be biased if, for example, it underestimated one group’s probability of success in a given endeavor. Test bias – definition

4. Various groups within society differ in their average scores on some psychological tests African-Americans score 1 standard deviation lower than Whites Asian-Americans score slightly higher than Whites Ashkenazi Jews score highest of all Test bias – the basic issue

5. What causes group differences? • We don’t know. Here are some candidate accounts: • Genetics • Socioeconomic factors • Caste • Culture • Stereotype threat

6. Highest IQ scores are for Ashkenazi Jews Cochran et al. (2006): medieval social environment for European Jews selected for verbal & math intelligence (but not spatial) Some relation to disease genes? Genetics

7. Much higher proportion of African-Americans are poor than of Whites, with consequences for nutrition, health care, resources such as books in the home But AA – White difference is not eliminated when groups are equated on SES Socioeconomic factors

8. “Involuntary” minorities all over the world do less well in school and drop out earlier than majority children Ogbu: African-American children lack “effort optimism” – the sense that hard work will be rewarded Caste

9. A. Wade Boykin: African-American culture has a “deep-structure” that conflicts with the demands made by typical American schools Culture

10. When children are ordered to do their own work, arrive at their own individual answers, work only with their own materials, they are being sent cultural messages. When children come to believe that getting up and moving about the classroom is inappropriate, they are being sent powerful cultural messages. When children come to confine their 'learning' to consistently bracketed time periods, when they are consistently prompted to tell what they know and not how they feel, when they are led to believe that they are completely responsible for their own success and failure, when they are required to consistently put forth considerable effort for effort's sake on tedious and personally irrelevant tasks . . , then they are pervasively having cultural lessons imposed on them" (Boykin, 1994, p. 125).

11. Awad (2007) 313 African-American university students at a historically Black university GRE (Verbal) and several psychological tests Racial identity & test scores

12. Cross Racial Identity Scale Cross (1991) Rosenberg Self-Esteem Scale Rosenberg (1965) Academic Self-Concept Scale Reynolds (1988) Awad (2007)

13. Academic self-concept predicted GPA but not GRE test scores Racial identity predicted neither GPA nor GRE scores Self-esteem didn’t predict either Racial identity & test scores

14. Steele & Aronson (1995) A social-psychological threat produced in a situation in which a negative stereotype about your group is made salient You fear you will confirm the stereotype This affects highly able, school-identified African-Americans because they feel the most pressure to do well Stereotype threat

15. Major tests have been subjected to impressive scrutiny for decades Enormous resources are devoted to this purpose Criterion validity has been established very securely for the major intelligence tests – they do predict college and job performance Arguments that tests are not biased

16. It is not appropriate to focus on individual items on a test, which some critics of testing do Items should be drawn from a variety of domains, not all of which will be familiar to anyone Arguments that tests are not biased

17. Test developers evaluate tests on the basis of overall patterns of prediction utility They’re future-oriented, not past-oriented: “How will you do in college or in a job?” Not “have you had the opportunity to learn?” Arguments that tests are not biased

18. Do you think of test score results as “outcomes” or as “information” (predictors)? Test developers say, results are the beginning, not the end – they are information that will guide us Opponents see test results as outcomes Arguments that tests are not biased

19. Systematic studies have asked whether biased items produce group differences on tests such as Stanford-Binet and Wechsler tests These studies found no evidence that group differences disappeared when allegedly biased items were removed Arguments that tests are not biased

20. Group differences just as large on what is considered the most culture fair test, Ravens Progressive Matrices, as on WAIS IQ scores have same utility for prediction regardless of race or socio-economic status. Argument that tests are not biased

21. In this approach to testing for bias, you first form groups for comparison which are equated on overall test score Implication: groups are equivalent in overall ability Then, you look for differences between groups on individual items Where difference is found, you conclude that the item is biased (since groups are not different on ability) Differential item functioning analysis

22. But removing such items does not eliminate group differences E.g., people depicted in test items may typically be White & male But changing this has little effect (McCarty, Noble, & Huntley, 1989) Differential item functioning analysis

23. We evaluate criterion validity by looking at correlation between test scores and criterion scores E.g., SAT scores vs. GPA after 4 years at university Criterion-related sources of bias

24. If correlation is good, we use test scores (e.g., SAT) to predict criterion – and make selection decisions What do we do if the correlation is different for different groups? This would imply that test scores mean different things for different groups Criterion-related sources of bias

25. In this graph, Group B performs better than Group A but the correlation is the same for both Criterion Group B Group A Test score Criterion-related sources of bias

26. In this graph, the slopes of the lines are the same but the intercepts are different Equal slopes means equal correlations – that is, equally good predictions Criterion-related sources of bias Group B Criterion Group A Test score

27. Here, the intercepts are different and the slopes are different, so predictions for Groups A and B would not be equally good Such cases are rare Criterion-related sources of bias Group B Group A X1 X2

28. Major tests, such as SAT and WISC-R, have equal criterion validity for various ethnic groups (e.g, African-American, White, Latino/Latina) Similar results have been found in other multi-ethnic countries, such as Israel Criterion-related sources of bias

29. Other approaches to testing minority groups • The Chitling Test • The BITCH Test • SOMPA

30. Developed to make a point about testing for information a group is unlikely to have acquired Questions require a particular form of “street smarts” to answer correctly No validity data exist for this test If you want to predict college performance for minority students, this test won’t help The Chitling Test (Dove, 1968)

31. Task: define 100 words drawn from the Afro-American Slang Dictionary and Williams' personal experience African-Americans score higher than Whites Williams argues that this test is analogous to the standard IQ tests, which are also culture-bound The BITCH test (Williams, 1974)

32. Problem: there is no reason to accept the claim that this is an intelligence test. There is no validity evidence – no prediction of any performance Does not test reasoning skills May have some value for testing familiarity with African-American culture The BITCH test (Williams, 1974)

33. System of Multi-cultural Pluralistic Assessment Based on idea that what constitutes knowledge is socially-constructed Mercer also suggested that IQ tests are a tool Whites use to keep minority groups “in their place”. SOMPA (Mercer, 1979)

34. Inspired originally in part by over-representation of minority group children in EMR classes in US schools Mercer: this over-representation resulted from both More medical problems Unfamiliar cultural references on tests SOMPA (Mercer, 1979)

35. Fundamental assumption: all cultural groups have the same potential on average On this view, if one cultural group does more poorly than another on a test, that is a fact about the test, not the groups. SOMPA (Mercer, 1979)

36. Combines 3 kinds of evaluation: Medical Health, vision, hearing, etc. Social Entire WISC-R Pluralistic Compare WISC-R scores to those of same community SOMPA (Mercer, 1979)

37. Estimated Learning Potentials: WISC-R scores adjusted for socio-economic background But these ELPs don’t predict school performance as well as the original WISC-R scores Mercer: ELPs are intended to assess who should be in EMR classes SOMPA (Mercer, 1979)

38. A major problem, in my view, is that we don’t know what consequences arise for children who are removed from EMR classes on basis of ELPs Is what we call these children important? It is if the label has an effect, but data do not show that effect SOMPA used much less today than it used to be SOMPA (Mercer, 1979)

39. Models of test Bias • Regression • Constant Ratio • Cole/Darlington • Quota

40. Basis – unqualified individualism: Treat each person as an individual, not as a member of a group Select people with highest scores for job or college place Ignores sex, race, other group characteristics Leads to highest average performance on criterion Regression

41. Basis – choose so that selection ratio for groups = success ratio for groups Select the best candidate but give a boost to minority group members’ scores so that selection probability = success probability Constant Ratio

42. Adjust test scores for minority groups upwards by half the mean difference between groups Leads to somewhat lower average performance on criterion Constant Ratio

43. Basis – If there is special value in selecting minority group members, then a minority score of Y on criterion is equal to a majority score of Y + k on criterion Separate regression equations used for different groups and adjustment made Leads to lower average performance on criterion Cole/Darlington

44. If a value is placed on selection of minority group members, and intercept is lower for that group, then we consider minority test score X1 and majority test score X2 equal X1 X2 Cole/Darlington k

45. Basis – idea that all groups should have equal outcomes Selection based on different regression equations for each group Produces lower average performance on criterion Quota

46. If 10% of population is Asian then 10% of student body should be Asian Another way to look at this: if 10% of population is Jewish then no more than 10% of professors should be Jewish. This puts the quota idea in a different light. Quota