1 / 33

Life After P -hacking (APS May 2013, Washington DC) With minor edits for posting

Life After P -hacking (APS May 2013, Washington DC) With minor edits for posting. Photo not necessary. Leif Nelson UC Berkeley. Uri Simonsohn Penn (gave the talk). Joe Simmons Penn also. Definition. p -hacking : exploiting researchers’ degrees-of-freedom seeking p<.05.

rimona
Download Presentation

Life After P -hacking (APS May 2013, Washington DC) With minor edits for posting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Life After P-hacking(APS May 2013, Washington DC) With minor edits for posting Photo not necessary Leif Nelson UC Berkeley Uri Simonsohn Penn (gave the talk) Joe Simmons Penn also

  2. Definition p-hacking: exploiting researchers’ degrees-of-freedom seeking p<.05

  3. Life after p-hacking • n>50 • Direct replications • 21 words • Compromise writing • Who to hire • What about Bayesian?

  4. ~ Median study: n=20 • False-Positive Psych: n>20 • What can you reliably detect with n=20? • Mturk study. • N=674 • Why not published ds?

  5. n=20 is enough for: • Men taller than women n=6 • People above median age closer to retirement n=10 • Women, more shoes than men n=15

  6. n=20 is not enough for: • People who like spicy food are more likely to like Indian food n = 27 • Liberals rate social equality as more important than do conservatives n = 34 • People who like eggs report eating egg salad more often n = 47 • Men weigh more than women n = 47 • Smokers think smoking is less likely to kill someone than do non-smokers n = 146

  7. People who like spicy food are more likely to like Indian food n = 27 • Liberals rate social equality as more important than do conservatives n = 34 • People who like eggs report eating egg salad more often n = 47 • Men weigh more than women n = 47 • Smokers think smoking is less likely to kill someone than do non-smokers n = 146

  8. Are you studying a bigger effect than: • Men weigh more than women? • If not, use n>50

  9. Life after p-hacking • n>50 • Direct replications • 21 words • Compromise writing • Who to hire • What about Bayesian?

  10. Estimates are way off Subjects confused? Big outliers

  11. p < .03 Estimates are way off Subjects confused? Big outliers

  12. Study 1? p < .03

  13. Run calories study again. • Same exclusion rule.

  14. Why not just conceptual replication? • Restart p-hacking clock • Failures do not count

  15. Replications • Conceptual • Rule out confounds • Rule in generalizability • Direct • Rule out false-positive

  16. Life after p-hacking • n>50 • Direct replications • 21 words (Google it) • Compromise writing • Who to hire • What about Bayesian?

  17. How can an organic farmer compete?

  18. How can an organic researchercompete? • If you determined sample size in advance Say it. • If you did not drop variables Say it. • If you did not drop conditions Say it.

  19. 21 Word Solution get .pdf here http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588 Organic Farmer Organic Researcher Footnote 1 We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.

  20. Life after p-hacking • n>50 • Direct replications • 21 words • Compromise writing • Who to hire • What about Bayesian?

  21. Compromise writing • While reviewers still in dark ages. • Have it both ways. • “Clean” version in main text • All studies “worked” & < 2500 words • Supplement/footnote • n=100n=150 • p=.08 w/o exclusion • Data and materials online • Only reformers read small print • Organic 21 words applies. • Everybody likes the paper

  22. Life after p-hacking • n>50 • Direct replications • 21 words • Compromise writing • Who to hire • What about Bayesian?

  23. If you hire based on quantityyou pass on these guys

  24. What’s the alternative to counting papers? • Rookies: Best 1 • Tenure: Best 3 • Full: Best 5 Try it. It is a powerful question. What’s her best paper?

  25. Life after p-hacking • n>50 • Direct replications • 21 words • Compromise writing • Who to hire • What about Bayesian? Only speak for myself here. My prior: Bayesians will be unhappy in 3 2 1

  26. P-hacking also invalidatesBayesian results

  27. P-hacking also invalidatesBayesian results Let me say that again

  28. Bayesian proposals for Psych 1) Bayesian t-test • Replications use it sometimes • Turns out • α = 5% 2) Bayesian estimation • Latest JEP:G . • Turns out • Changes nothing 1%

  29. t-test “vs” Bayesian Estimationchanges nothing How similar? Results change by less than if we dropped 1 observationat random.

  30. But! • Isn’t data-peeking OK for Bayes? • Not when used for hypothesis testing • Also: • Dropped subjects, measures, conditions invalidate all inference.

  31. P-hacking Bayesian stats • Drunk driving leather seats Good reasons to go Bayesian do not include p-hacking.

  32. Next slide is the last.

  33. Life after p-hacking • n>50 • Direct replications • 21 words • Compromise writing • Who to hire • What about Bayesian? Leif Nelson UC Berkeley Joe Simmons Penn Only speak for myself here.

More Related