Know Your Limitations Or, Debunking The Easy Way

1. Know Your LimitationsOr, Debunking The Easy Way UKCSJ London June 18, 2014 Ivan Oransky VP, Global Editorial Director, MedPage Today http://medpagetoday.com Co-founder, Retraction Watch http://retractionwatch.com @ivanoransky

2. Crotch Length and Fertility

3. Crotch Length and Fertility

4. Crotch Length and Fertility “Somehow we feel that the ability to grab eyeballs by putting "crotch length" in a headline dictated editorial judgment on this story.  Because it certainly fell short in delivering the goods that a man would need in order to evaluate the potential of this finding.”

5. Crotch Length and Fertility “Certain limitations warrant mention. As a referral center for male infertility, it was not always possible to blind observers to the men's diagnoses or fatherhood status which theoretically can lead to observer bias. Although, the magnitude of observed differences in AGD between fathers and infertile men (i.e. 40% in mean AGD and 45% in median AGD) suggests that any bias would be unlikely to affect the overall conclusions. Moreover, the current method of AGD measurement in adult men has not been studied, thus its accuracy and reproducibility were difficult to assess other than the performed comparison of measurements between investigators. . .

6. Crotch Length and Fertility “Future studies are necessary to compare techniques for measurement as well as other anatomic locations of the AGD measurement. In addition, only men referred to and evaluated in our clinic were eligible for enrollment; therefore, it is possible that our patient population does not represent all infertile men. . .

7. Crotch Length and Fertility “It is also important to note that the fertile controls were significantly older than the infertile patients. While age was not associated with AGD after accounting for fatherhood status and no evidence of effect modification by age was found, it possible that AGD could change with age. In addition, while all patients were measured in the same position, some men were measured at the time of surgery under general anesthesia while others were awake. It is conceivable that anesthesia may affect measurements, although stratifying by anesthesia status did not affect the conclusions.”

8. Fist Bump

9. Fist Bump “Numerous headlines tout that substituting a fist bump for a handshake reduces transmission of infection. But is this just one of many examples of the media sensationalizing the findings of a paper far beyond what it is due?” Guest Blogger: Skeptical Scalpel

10. Fist Bump

11. Fist Bump “This study was designed as a pilot to explore the potential harmful effects of the handshake within the healthcare system, and as such it has several limiting factors. The study is limited by our small sample size and it could not assess statistical significance. Limited funding curtailed our ability to identify specific strains of bacteria. A larger study is planned to assess the level of significance between the handshake and fist bump as well as to assess virulence of the cultured strains. Furthermore, viral transmission is thought to be commonly transferred by skin contact, but was not assessed in this study.”

12. Walnuts and Diabetes

13. Walnuts and Diabetes “Even large observational studies have inherent limitations, but this story barely cracked the shell on this topic.”

14. Walnuts and Diabetes

15. Walnuts and Diabetes “Our study has several potential limitations. First, our study populations primarily consisted of white female nurses, which may limit the generalizability of the findings to other ethnic groups or males. Second, because diet was assessed by FFQs, measurement error of nut intake is inevitable, which may underestimate the true associations. Third, biochemical markers for type 2 diabetes (fasting glucose, insulin, lipids, and HbA1C, etc.) were not available in the full NHS cohorts, and thus could not be adjusted in the models. . .

16. Walnuts and Diabetes “Furthermore, habitual nut consumption was associated with several healthy lifestyle practices and may be a marker for an overall healthy lifestyle. Although we carefully controlled for a number of diabetes risk factors, unmeasured and residual confounding is still possible to explain the association and we could not fully exclude the potential influence from the overall diet quality and healthy cuisine effects.”

17. Morning Sickness and IQ

18. Morning Sickness and IQ “Limitations of this study include its retrospective component, potential for recall bias, and cohort selectivity limited to the Motherisk database. Also, the use of different versions of the assessment instrument and broad age range of the children may be limiting factors.”

19. Studies: They’re There

20. Studies: They’re There “In a more detailed assessment of the medical literature, in which two independent reviewers assessed the abstract and discussion sections of 300 medical research papers, published in first and second tier general medical and specialty journals, 73% of all papers were found to acknowledge a median of 3 limitations.”

21. One Major Journal Knows It

22. Press Releases: They’re Not There Academic medical centers issue a mean of 49 press releases/year Among 200 randomly selected releases • 87 (44%) promoted animal or laboratory research, of which 64 (74%) explicitly claimed relevance to human health • Among 95 releases about clinical research, 22 (23%) omitted study size and 32 (34%) failed to quantify results • 113 releases promoted human research • 17% promoted randomized trials or meta-analyses • 40% reported on uncontrolled interventions, small samples (<30 participants), surrogate primary outcomes, or unpublished data—yet 58% lacked the relevant cautions Woloshin S et al. Press releases by academic medical centers: not so academic? Ann Intern Med 2009;150:613-618

23. Always Read the Study

24. Always Read the Study Get the full study and read it –“I think it’s journalistic malpractice to not have the full study in front of you when you’re reporting,” Oransky says.

25. How to Get Studies • www.EurekAlert.org for embargoed material • AHCJ membership includes access to Cochrane Library, Health Affairs, JAMA, and many other journals www.healthjournalism.org • ScienceDirect (Elsevier) gives reporters free access to hundreds of journals www.sciencedirect.com • Open access journals (e.g., Public Library of Science www.plos.org) • Ask press officers, or the authors

26. Ask “Dumb” Questions If you lack experience dealing with scientific material, don’t be afraid to ask for definitions of jargon and scientific terms. This is no time to pretend you understand everything. Oransky says the science and medical industries are full of jargon that mask important details. “You’ll get off the phone and have a notebook full of gibberish and jargon,” he says. “You can’t be afraid of asking a dumb question.”

27. Ask Smart Questions • Was it: • Peer-reviewed? • Published? Where? Not all journals are created equal. “Dr. X said they published in Y rather than a clinical journal because the paper was too long for the word limits in the clinical journals. I'm not sure where a detail like that would go…but he was impressed with my question.”

28. Ask Smart Questions • Was it in humans? • It’s remarkable there are any mice left with cancer, depression, or restless leg syndrome

29. Ask Smart Questions • Size matters Look for the power calculation, and ask if you don’t see one

30. Ask Smart Questions • Was it well-designed? From Covering Medical Research, Schwitzer/AHCJ

31. Ask Smart Questions • “Were those your primary endpoints?” • “Looks as though that endpoint reached statistical significance. Is that difference clinically significant?”

32. Who Could Benefit? • How many people have the disease? • Keep potential disease-mongering in mind

33. Look For Other Angles

34. How Effective is the Treatment? • Clinically significant endpoints, or surrogates – does this matter? • Preventing complications? How many? • Always remember to quantify results, not just “patients improved”

35. What Are The Side Effects? • Every treatment has them • Where to look: • Go beyond press releases and abstracts • Look at tables, charts, and results sections

36. Who Dropped Out? • Why did they leave the trial? • Intention to treat analysis

37. How Much Does it Cost? • If it’s ready to be the subject of a story, someone has projected the likely cost and market. • At least ask.

38. Who Has an Interest? • Disclose conflicts • PharmedOut.org • Dollars For Docs series http://projects.propublica.org/docdollars/

39. Are There Alternatives? • Did the study compare the new treatment to existing alternatives, or to placebo? • What are the advantages and disadvantages (and costs) of those existing alternatives? • Consider alternative explanations. Remember coffee and pancreatic cancer?

40. Don’t Rely Only on Study Authors • Find outside sources. Here’s how:

41. Some Math: Alzheimer’s

42. Some Math: Alzheimer’s “The best piece I saw was by John Gever, a dependable debunker at MedPage Today, who led by saying the test's "accuracy fell short of what would normally be acceptable for a screening test." “That's a perspective I hadn't seen anywhere else. And Gever reports that the test has a "positive predictive value" of 35%, which means "nearly two-thirds of positive screening results would be false." In other words, out of every 10 people who tested positive, two-thirds of them--say, six or seven--would be told they will develop Alzheimer's disease when that was not the case. “If Gever's math is correct--I can't vouch for it, but he has a long track record--then the test is nowhere close to being useful for screening patients.”

43. Some Math: Alzheimer’s

44. Some Math: Alzheimer’s “But they left out an important statistic for judging the usefulness of such a test, as it would be applied in the clinic -- the positive predictive value (PPV) or the accuracy of positive results seen in the target population (in this case, cognitively healthy seniors). “Contrary to what I later learned is popular belief, calculating a PPV is easy, requiring nothing more than fourth grade arithmetic.

45. Some Math: Alzheimer’s “So let's look at an assay with sensitivity and specificity rates of 90%, to be used in a population of 1,000 people in which 5% will actually convert to cognitive impairment. We know that 50 of these 1,000 will convert and 950 will not. Of those 50 converters, 10% will falsely test negative, and, of the 950 nonconverters, 10% will falsely test positive. “That means the testing program produces a total of 140 positive results, 45 of which are correct and 95 false. The PPV is just the fraction of total positives that are correct -- in this example, 45/140 or 32%.

46. Some Math: Alzheimer’s “Even if the conversion rate to cognitive impairment in healthy seniors is 10% instead of 5%, the PPV remains low. In 1,000 people, 100 will be true positives and 900 will be true negatives; thus, with the same test sensitivity and specificity, there will be 180 positive results, split evenly between true and false, for a PPV of 50%.

47. NNT

48. A Dirty Little Secret Keep a biostatistician in your back pocket Photo by Peyri Herrera, on Flickr