1 / 23

Good data practices

Good data practices. Jelte M. Wicherts. Source: Wicherts , J. M. (2011). Psychology must learn a lesson from fraud case . Nature, 480 , 7. Integrity in black and white. ?. Dr. Evil. Good. Interested in prestige Critical of results of others Unreliable and sloppy Secretive and dishonest

eddy
Download Presentation

Good data practices

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Good data practices Jelte M. Wicherts

  2. Source: Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480, 7.

  3. Integrity in black and white ? Dr. Evil Good Interested in prestige Critical of results of others Unreliable and sloppy Secretive and dishonest Interested in quantity Seeks support for own theories Uninterested in prestige Critical of own results Reliable and rigorous Open and honest Interested in quality Seeks “truth”

  4. Integrity in 50 shades of grey Sloppy Science Top Science Source: Wicherts, J. M. & Veldkamp, C.L.S. (2013). De vijftigtintengrijs van wetenschappelijkeintegriteit. De Psycholoog.

  5. A former professor: “I was getting better and better in using techniques to improve poor results. […] What I did was not as white as snow, but it was not pitch-dark either. It was grey and it was common. How else could all the others get all those beautiful results? […] After years of balancing on the cliff, the grey became darker black, and finally I fell all the way down.” Source: D. Stapel, 2012, Ontsporing [Derailment]p. 143-144; mytranslation

  6. Norms vs. Counternorms • Survey among 3,247 US scientists, asking: • Whether they subscribed to norms of “good science” • Whether they behaved according to these norms • Whether their typical colleague behaved according tothese norms Universalism: Scientists evaluate research only on its merit, i.e., according to accepted standards of the field. Particularism: Scientists assess new knowledge […] based on reputation […] of the individual or research group. Commonality: Scientists openly share findings with colleagues. Secrecy: Scientists protect their newest findings to ensure priority in publishing [..] Source: Anderson, M.S., Martinson, B. C., & De Vries, R. (2007). Journal of Empirical Research on Human Research Ethics, 2 (4), 3-14

  7. Governance: Scientists are responsible for the direction and control of science through governance, self-regulation and peer review. Administration:Scientists rely on administrators to direct the scientific enterprise through management decisions. Quality:Scientists judge each others’ contributions to science primarily on the basis of quality. Quantity: Scientists assess each others’ work primarily on the basis of numbers of publications and grants. Disinterestedness: Scientists are motivated by the desire for knowledge and discovery. Self-Interestedness: Scientists compete with others in the same field for funding and recognition of their achievements. Organized Skepticism: Scientists consider all new evidence, hypotheses, theories, and innovations, even those that challenge or contradict their own work. Organized Dogmatism: Scientists invest their careers in promoting their own most important findings, theories, or innovation. Source: Anderson, M.S., Martinson, B. C., & De Vries, R. (2007). Journal of Empirical Research on Human Research Ethics, 2 (4), 3-14

  8. Do researchers regard their colleagues highly? norm>counternorm norm=counternorm norm<counternorm Source: Anderson, M.S., Martinson, B. C., & De Vries, R. (2007). Journal of Empirical Research on Human Research Ethics, 2 (4), 3-14

  9. Do researchers share data upon request? In 2005, we requested the raw data from 141 papers published in four APA journals for use in a study of the effects of outliers on the outcome of data analyses. Source: Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726-728.

  10. Reasons for refusal • This is an ongoing project/ IRB does not allow it • I have no time to do this…I’m up for tenure • My research assistant/postdoc/student left • I recently moved, I have a new computer! • “I am afraid your request is not possible”

  11. Reasons to be patient… • This will take me some time, I’ll get back to you • I’ll send you the data tonight, tomorrow, next week, next month, ASAP • I’ll send you the data within a few days 2925 days and still counting!

  12. Are statistical results checked by (co-)authors and reviewers? Method: a representative sample of 257 papers Recomputed 4720 p-values from NHST and checked for consistency p = .06 Results: 128 papers (50%) contained at least one error 39 papers (15%) contained at least one error related to p = .05 Conclusion: Errors predominantly led to “better” results Source: Bakker, M. & Wicherts, J. M. (2011). (Mis)reporting of statistical results in psychology journals.Behavior Research Methods, 43, 666-678.

  13. Reporting errors in papers from which data were or were not shared DATA NOT SHARED (N=28) DATA SHARED (N=21) Source: Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6, e 26828.

  14. Gross reporting errors (around p=.05) DATA NOT SHARED (N=28) DATA SHARED (N=21) Source: Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6, e 26828.

  15. Errors and data sharing Haphazard data documentation plays a role in reluctance to share and occurrence of errors. Poor data documentation also suggests that authors hardly share data with co-authors.

  16. Shalvi et al., 2011, Organizational Behavior and Human Decision Processes

  17. Willingness to share research data is related to the strength of the evidence Data shared? non-significant significant 10 errors! Source: Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6, e 26828.

  18. Human factors in statistics • Statistical analyses are complex and prone to human error • Our statistical intuitions are poor (e.g., we tend believe in the law of small numbers) • Researchers who conduct these analyses have clear expectations about outcomes

  19. Solution 1: The co-pilot model • Let your co-authors (or colleagues) replicate your analyses • Openness concerning analytic choices • Requires that you document data well • Facilitates sharing andpublication of data Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480, 7. Wicherts, J. M. & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence, 40, 73-76.

  20. Solution 2: Better training

  21. Solution 3: Just publish the data Wicherts, J. M. & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence, 40, 73-76.

  22. …in Journal of Open Psychology Data

  23. Thanks! Marjan Bakker Denny Borsboom Michele Nuijten CoosjeVeldkamp Dylan Molenaar

More Related