1 / 21

Is There a Credibility Crisis in Sociology? And If Yes, What Can be Done?

This presentation examines the credibility crisis in sociology, specifically in quantitative social research with observational data. It explores the lack of failed replications, doubts regarding data policies, and the presence of questionable research practices. The presentation also suggests measures such as reproduction, open science, and institutional change to address this crisis.

jackbrown
Download Presentation

Is There a Credibility Crisis in Sociology? And If Yes, What Can be Done?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Is There a Credibility Crisis in Sociology?And If Yes, What Can be Done? Katrin Auspurg & Josef Brüderl Presentation at MZES OSSC19 Mannheim, 25. January 2019

  2. Science: In Search for Knowledge Science should produce valid and robust knowledge At least four sources of error: • Bad measurement • Invalid and/or unreliable measurement • Bad research design • “Bad” researchers (errors) • “Biased” researchers (non-objective research) • Fraud: Fabricating/Falsifying data and/or analyses • Questionable research practices (QRPs) Recently it became clear that the 4. source is widespread  credibility crisis (replication crisis) in science

  3. Contents • No Crisis in Sociology! • Really no Crisis in Sociology? • What Can Be Done? Focus: (quantitative) social research (in sociology) with observational data Replication: reproducing results with new data Reproduction: reproducing results with the same data • Identical model specification (pure reproduction) • Extensions of model specification

  4. No Crisis in Sociology! • Program of the ASA meeting 2018 • „Crisis“: 37 • „Replication (crisis)“: 1 • „Publication Bias“: 0 • DGS Kongress 2018 program • „Krise“: 22 • „Replikation(skrise)“ 0 • „Publication Bias“: 1 • Program of the 1st as-Kongress2018 • “Krise”: 1 • „Replikation(skrise)“: 0 • „Publication Bias“: 0

  5. No Bad Research in Sociology! • Retraction Watch Database (Nov. 2018) • 390 retractions w/ subject “sociology” • # retractions by journal (ordered according to impact factor) • American Sociological Review: 1 • American Journal of Sociology: 0 • Sociological Methods & Research: 0 • … • Top 10: 1 • Top 30: 4 #Retractions in all 147 SSCI sociology journals: 11

  6. No Replications in Sociology!What We Don't Know Won't Hurt … Source: OwnCalculations

  7. The Illusion of „Credible Sociology“ • No credibility crisis • No failed replications • No doubts/no data policy • No replications

  8. Really no Crisis in Sociology? • No direct evidence • So far no large-scale replication initiative to our knowledge • We suppose that also in sociology (most?) published research findings are false • There is evidence for QRPs in sociology • Studies on publication bias • Analytical flexibility in connection with „cherry-picking“ • Sociology is full of errors • Measurement errors • Faulty research designs • Researchers working with complex data make errors • Codingerrors • Mis-specified models • ...

  9. Publication Bias German journals of sociology, 50 articles US journals of sociology, 46 articles not sign. not sign. sign. sign. Auspurg/Hinz 2011 Gerber/Malhotra 2008

  10. Analytical Flexibility • Robustness analyses of the effect of job training on earnings • CPS data • Field experiment Source: Munoz/Young 2018 Statistical models are a “garden of forking paths” (Gelman/Loken 2014)

  11. Errors in Sociology • Statistics on the retracted SSCI-papers w/ subject „sociology“ (Retraction Watch, Nov. 2018; N = 11) • Number of retractions overall 11 • Plagiarism / duplication: 5 • Problems w/ data quality: 1 • Errors: 5 • Miscoding of missing values 2 • Coding swichteroo 2 • Other coding error 1 • At least three papers in ASR miscoded missing values: Jasso (1985), Herring (2009), Munsch (2015)

  12. Mis-specified Model Schmidt-Catran, A. and D. Spies. 2016. "Immigration and Welfare Support in Germany." ASR 81: 242-61. Auspurg, K., J. Brüderl and Th. Wöhler. 2018. “Does Immigration Reduce the Support for Welfare Spending?” ASR, forthcoming.

  13. Measurement Error 1st Comment & Reply 2nd Comment Erratum

  14. What Can Be Done? • The main problems in social research are probably • faulty designs • simple errors, and • misuse of analytical flexibility • We need „Reproduction, Reproduction” (c.f. King 1995) • Each original research result should be scrutinized intensely • Reproductions are a powerful tool for reducing the False Positive Rate (FPR) (Diekmann 2011)

  15. We Need a Cultural Revolution (in Sociology) • From horizontal to vertical growth of knowledge (Grunow et al. 2018) • Horizontal growth: research should be innovative • “Knowledge is emerging as a flat and unstructured mass of all kinds of ideas that do not sufficiently relate to each other” • Vertical growth: research should build upon previous findings • First reproduction, then replication/extension • How can we achieve this? • Moral appeals will not do the job • Current incentive structures are such that „new finding“ dominates „replication“ (social dilemma) • We need institutional change (Diekmann 2011, Freese/King 2018)

  16. How to Achieve the Cultural Revolution • Open science • FAIRData, Open Materials, Open Code, Transparency • The first step of each new research project should be the reproduction of at least one main finding in the research field • Replication journal/section: publishes results of reproductions (also if the reproduction has been successful) • New citation rule: original work cited along with its reproductions • A database on all reproductions and their finding is neededE.g., ReplicationWiki at the University of Göttingen  3. and 4. are an “incentive revolution”: (Coffman et al. 2017) Replicators would get the credit that counts in science

  17. BACKUP

  18. Evidence for QRP in Sociology 100 90 80 70 60 % 50 40 30 20 10 0 Strong results(42% of total) Null results(22% of total) Mixed results(36% of total) Source: Mervis 2014; Christensen/Miguel 2018 Null results rarely see the light of day? Publication outcomes of TESS funded experiments (Franco et al. 2004; Mervis 2014)

  19. Evidence for Problems in Qualitative Research • First evidence for publication bias • Petticrew et al (2008): „Publication bias in qualitative research: what becomes of qualitative research presented at conferences?” • 56% of studies not published (2 yrs after conference) • Studies without clear findings less likely to be published • First evidence for analytical flexibility • Toews et al. (2016): “Extent, Awareness and Perception of Dissemination Bias in Qualitative Research” • Survey with about 800 qualitative researchers • 36% reported that important findings were missing in one or more of their published reports • 97% estimated that other researchers omit important findings

  20. Not Everybody in Sociology Sees it That Way „In der … qualitativen Sozialforschung hingegen, … eignet sich Replikation grundsätzlich nicht zur Prüfung der Qualität von Studien“ (J. Strübing, „Problem, Lösung oder Symptom? Zur Forderung nach Replizierbarkeit von Forschungsergebnissen“, in Forschung & Lehre 02/2018) „Entsprechend darf die Sekundärnutzung von Daten nicht als ‚Normalmodell‘ von Forschung gelten …“ (DGS, „Bereitstellung und Nachnutzung von Forschungsdaten in der Soziologie“. Positionspapier vom 08.01.2019)

  21. Examples: Miscoding Missing Values • Jasso (1985) ASR: coital frequency increases with wife‘s age • Kahn/Udry (1986): 4 observations had value 88; Jasso treated these as valid values; treating these as missing, the effect of wife‘s age becomes non-significant • Herring (2009) ASR: diversity increases firm revenue • Stojmenovska/Bol/Leopold (2017): 206 firms had values 88,888,888,888; Herring treated these as valid values; treating these as missing, the effect of diversity becomes non-significant • Munsch (2015) ASR: marital infidelity is at 10 % • Munsch (2018): 246 missings were miscoded as „infidelity“ (Stata .-problem); with correct coding marital infidelity is at 6 %

More Related