Week 3 of course “Quality and Ethics in Social Science Research” Matthijs Kalmijn

1. Value-free Research in the Social Sciences: Principles and Controversies2017-2018Semester 1, block 1 Week 3 of course “Quality and Ethics in Social Science Research” Matthijs Kalmijn University of Amsterdam, Department of Sociology

2. Content • Principles • Merton’s classic norms • Objectivity • Bias (MacCoun) • Subgroup work & presentations • Controversies • Same-sex parenting: bad for kids? • Racial diversity: good for companies? • Remedies

3. My own approach • Sociology (family, life course, inequality) • Quantitative • Survey based • Statistically analyze data to • Describe reality • Test theories about reality • Primacy of science

4. The norms of scienceRobert K. Merton • Universalism • Communalism • Disinterestedness • Organized skepticism

5. Examples of violations • Bourdieu said it …. • Epidemiological surveys in the Netherlands • Undocumented research data • Theoretical paradigms

6. Objectivity in research • Objective / subjective is a continuum • Don’t throw out the baby with the bathwater • Aim for more rather than less objectivity • Distinguish • Subjectivity in the research questions we choose • Subjectivity in the theories we wish to test • Subjectivity in the use of research findings • Distinguish • Subjectivity of the researcher • Subjectivity of the things/people we study

7. Value free science • Political and moral values may motivate you to do research • Political and moral values may distort your interpretation of research results • The problem of value free science is part of a more general problem of biased evidence interpretation

8. Biased evidence interpretation: MacCoun’s typology

9. Example 1 • Mental contamination

10. Example 2

11. Disorder & prejudice WARNING FRAUDULENT! Stapel, D. A. and S. Lindenberg. 2011. "Coping with Chaos: How Disordered Contexts Promote Stereotyping and Discrimination." Science 332(6026):251-53.

12. DiederikStapel’smethod • PI works out theory and hypotheses in detail with collaborator • Collaborator develops full questionnaires for school-based survey experiment (makes several revisions) • Printing of questionnaires, purchase of incentives and other research material • School-based field stage (PI is only contact) • Questionnaires put in Excel by school pupils • Data back via PI to collaborator, starts analyzing • Hypotheses confirmed (“you have gold there”) • Elaborate co-writing stage and submission (e.g., JPSP)

13. DiederikStapel’smethod • PI works out theory and hypotheses in detail with collaborator • Collaborator develops full questionnaires for school-based survey experiment (makes several revisions) • Printing of questionnaires, purchase of incentives and other research material • School-based field stage (PI is only contact) • Questionnaires put in Excel by school pupils • Data back via PI to collaborator, starts analyzing • Hypotheses confirmed (“you have gold there”) • Elaborate co-writing stage and submission (e.g., JPSP)

14. Other examples of fraud • Biological science (Hwang Woo-Suk, stem cell research) • Medical research (Potti, cancer research) • Anthropology (Mart Bax, wars in Bosnia) • Political science (LaCour & Green, on how contact changes homophobia)

15. Example 3 • Statistical significance • Is b significant? • b has a t-value which has a p-value, e.g. 4% • Meaning: Suppose the real b (effect) is 0, the chance of finding that value of b (or higher) is 4% • Since this chance is low (below 5%), we reject the null hypothesis (0 effect unlikely) • We call the observed b significant

16. Disconfirmation bias Masicampo, E. J. and D. R. Lalande. 2012. "A Peculiar Prevalence of P Values Just Below .05." Quarterly Journal of Experimental Psychology 65(11):2271-79.

17. Summary

18. Motivation for bias • Self-interest • Interests of (external) organization • Political and moral views • Societal effects/consequences • Overcommitted to a theory

19. Subgroup work

20. Subgroup work • Subgroups formed based on discipline and/or substantive research interests • Work out examples of biases that can arise in your own research, at least one for each cell • These examples can be fictitious but must be ‘realistic’ (how could it go wrong….) • Make a Powerpoint and present back to class (one presenter) • Send to matthijskalmijn@gmail.com

21. Controversies 1: Regnerus (2012) Criticized by Amato, Eggebeen and Osborne and others

22. Regenerus ‘motivated’ or not? "As Christians, our lives should reflect our relationship with God and our desire to glorify Him," Regnerus says. "I've noticed that some Christian professors see a disconnect between their faith and their profession. I believe that if your faith matters, it should inform what you teach and what you research.” (Old quote in Trinity Christian Vollege Alumni Magazine)

23. Replication by Cheng and Powell 2015

24. Controversies 2: Herring Criticized by Stojmenovska, Bol & Leopold

25. Herring ‘motivated’ or not? “Yes, diversity is still a good thing, but not just because it is related to business outcomes. Diversity is also a good thing because it reinforces the belief that everyone—no matter their race, ethnicity, gender, sexuality, or religion—deserves an equal opportunity. This message is even more important now than it was two presidential elections ago.” (in Herring’s response to Stojmenovska, Bol & Leopold)

26. Replication by Stojmenovska et al.

27. RACIAL DIVERSITY COMPANY SIZE CONFOUNDING VARIABLE SPURIOUS! COMPANY SALES

28. Some remedies – institutional • Socialize students • Take an oath (?) • Let social scientists compete • Organize peer review • Share data • Conduct meta analyses

29. Some remedies – individual • Engage alternative hypotheses • Strive for accuracy and precision • Look (also) for falsification options • Allow (and do) replication studies • Have respect for the annoying facts

30. Moral and political values are sometimes good motivators but always poor guides