Recognizing Common Biostatistical Errors: A Case-Based Approach

1. Recognizing Common Biostatistical Errors:A Case-Based Approach Thomas B. Newman, MD, MPH Kristine A. Madsen, MD, MPH

2. Conflicts of Interest/Off-Label Use Tom is promoting his book but he’s donating all his royalties to Physicians for Social Responsibility No other conflict or off-label use

3. Tom’s Book – a must read

4. Why is the graph misleading?

5. Agenda • Review and discuss concepts • Break • Divide into groups, work on “unknowns” • Review “unknowns” in large group • Answers/discussion (and these slides) posted in Tom's Document Repository. Google "Thomas B Newman"

6. Concepts to be Illustrated • Type III errors • Standard deviation/standard error • Confidence Intervals • Between- vs within-group comparisons • Relative Risk/Absolute Risk; NNT • Statistical vs. clinical significance • Relative Risk/Odds Ratio • Violation of the independence assumption • Interpreting regression coefficients

8. Q: What’s wrong here?

9. America on the Move Results (abstract): … the America on the Move group compared with the self-monitor group had a significantly higher percentage of target children who maintained or reduced their BMI for age and, consistently, a significantly lower percentage who increased their BMI for age. % of children that maintained/reduced vs. increased BMI for age over 6 months. aP < .05 using chi-squared test. Pediatrics 2007; 120:e869-75

10. Safety, efficacy, and tolerance of intestinal lavage The safety, efficacy, and patient tolerance of an enterally administered isotonic intestinal lavage solution … was evaluated in 20 pediatric patients (ages 1 1/2 to 19 years) undergoing diagnostic colonoscopy. Emesis occurred in 4 patients, nausea in 11, and abdominal distension in 5. Clear stools were produced in a mean (+/- SE) time of 2.6 +/- 0.3 hours. Of 20 patients, 11 required or requested nasogastric administration of the lavage solution because of its unpleasant taste. We conclude that whole intestinal perfusion with a balanced electrolyte solution containing polyethylene glycol is safe, acceptable, and efficacious in children. J Pediatr 1991; 119:148-52

11. Intestinal Lavage • What is meant by mean time to clear stools = 2.6 ± 0.3 hours (mean ± SE)? • Do you agree that intestinal lavage is “safe, acceptable and efficacious”?

12. Mean difference = 0.08 degrees 95% CI : (0.05 to 0.11 degrees) Does this mean that tympanic temperatures are accurate? If the tympanic temp is 38.0, what can we say about the probable rectal temp? An Infrared Tympanic Membrane Thermometer in Pediatric Patients Table 2: Temperature Measurement by Glass-Mercury and Tympanic Membrane Thermometer Pediatrics 1990; 85:854-8

13. Behavior Abnormalities and Poor School Performance Due to Oral Theophylline Table 4. Teacher Behavioral Assessment Score Pre- and Post-treatment Results are average scores ± SD. Score is sum for 53 items, each of which was scored as: 1, as much as other students; 2, somewhat more than other students; 3, much more than other students. * t-tests Q1: What do the 5 P-values mean and which is most relevant? Q2: What t-test assumption may be violated? Q3: Do you see a Type 3 error? Peds 1986;78:1133

14. Combination Therapy With Desmopressin and an Anticholinergic Medication for Nonresponders to Desmopressin* • Significant decrease in wet nights with anticholinergic med added, not with placebo • Does this show ACh works? Pediatrics, 2008; 122:1027-1032

15. Combination Therapy With Desmopressin and an Anticholinergic Medication for Nonresponders to Desmopressin* • "With a generalized estimating equations approach, there was a significant 66% decrease in the risk of a wet episode…" • Responders (≥ 50% reduction in wet nights): 8/18 (44%) vs 5/16 (31%); P = 0.43 • Conclusion: "This study represents the first prospective, placebo-controlled trial examining the effect of desmopressin in combination with long-acting, anticholinergic, bladder-relaxing therapy for monosymptomatic primary nocturnal enuresis. " Pediatrics, 2008; 122:1027-1032

16. Confidence Intervals Extended Follow-up of Long-Term Survivors of Childhood Acute Lymphoblastic Leukemia “The death rate for the irradiated group slightly exceeded the expected rate in the general U.S. population (standardized mortality ratio, 1.90; 95 percent confidence interval, 1.12 to 3.00), whereas that for the nonirradiated group did not differ from the population norm (standardized mortality ratio, 1.75; 95 percent confidence interval, 0.34 to 5.00).” N Engl J Med 2003; 349:640-649 Q: Do the data suggest the irradiated group had worse survival?

17. Mortality rates after 10.5 years in the Multiple Risk Factor Intervention Trial The MRFIT was a primary prevention trial to test the effect of multifactor intervention on coronary heart disease mortality in high-risk men randomly assigned to special intervention (n = 6428) or to usual health care (n = 6438)… This report describes the mortality findings after 10.5 years… Mortality rates were lower for men who received special intervention than for men who received their usual care by 10.6% for coronary heart disease … These data suggest that multiple risk factor intervention confers a mortality benefit in middle-aged men over a period of about 10 years. JAMA 1990; 263:1795-801 Q: What is missing here?

18. Multiple Risk Factor Intervention Trial Table 2 - Mortality End-Point Findings Through December 31, 1985, for MRFIT SI and UC Men* *MRFIT indicates Multiple Risk Factor Intervention Trial; SI, special Intervention; and UC, usual care. †ICD-9 indicates International Classification of Diseases, Ninth Revision, Clinical Modification. ‡(1-RR)x100%, where the RR (relative risk) is estimated from the proportional hazards regression model.

19. Effect Size: Relative vs Absolute (Dichotomous Outcome Variables) • RR = Relative Risk (Risk Ratio) = Risk in intervention group/Risk in control group • RRR = Relative Risk Reduction = 1-RR • ARR = Absolute Risk Reduction = Risk in control group - Risk in intervention group • NNT = Number Needed to Treat = 1/ARR

20. RRR and ARR Example • If a treatment reduced failure rates from 4% in the placebo group to 1% in the treated group, what would be the relative and absolute risk reductions?

21. Which Risk Reduction is Most Relevant? • Questions of etiology: Relative Risk • Does the treatment work? • Is this a risk factor? • Clinical decisions; weighing benefits vs. costs or risks: Use absolute risk • Is the treatment worth giving

22. Q: What does the 34% reduction mean?

23. Nimotop Ad Graph

24. 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Original figure To scale 11%

25. Diabetes Control and Complications Trial (DCCT): Adolescents 53% reduction in retinopathy (p<0.05), but tripling of hypoglycemia (p<0.05) Study conclusion: “We conclude that the benefits [..of intensive treatment] outweigh the increased risk of hypoglycemia that accompanies such treatment.” J Pediatr, 1994. 125:177-88 Q: Would you agree with this conclusion?

26. DCCT - Rates per 100 person-years of follow-up: 19 episodes of hypoglycemia and 5.5 episodes of seizures or coma per >3 point change in retinopathy score J Pediatr, 1994. 125:177-88

28. Zelnorm® for Constipation • Randomized, double-blind trial of Zelnorm® (tageserod) for chronic constipation in 1264 patients from 128 centers in 18 countries • Results: Zelnorm® superior (P <0.0001) • Does this mean it worked well? Am J Gastroenterol 2005; 100:362-72

29. Zelnorm® for Constipation • Outcome: increase of ≥1 complete spontaneous bowel movement (CSBM) per week over 4 week period • Response rates: 40% with Zelnorm 6 mg bid, 27% with placebo Am J Gastroenterol 2005; 100:362-72

30. Cost of a CSBM • Risk difference? • NNT = ? • If 1 week of Zelnorm 6 mg tablets costs $42, what’s the cost of getting at least 1 more CSBM/wk?

31. Penciclovir for Recurrent Oral HSV • Randomized, double-blind trial sponsored by manufacturer • 1573 patients from 31 US Clinics • 1% Penciclovir or placebo applied q 2 hours while awake • Odds Ratio for healing by day 7 was 1.7 (P<0.001) • Is this clinically significant? Spruance et al. JAMA 1997;277:1374-9

32. Penciclovir Effect Size • Many readers think this means the probability of healing was 1.7 times (70%) greater with penciclovir • But it's the odds of healing that are 1.7 times higher • Actual healing rates at day 7 were 78% vs 68% (0.78/0.68 = 1.15) • Median time to healing : 5.5 vs. 4.8 days

33. Effect Size: Odds Ratio vs. Risk Ratio • The key is the difference between probability and odds: odds = P/(1-P) • Watch out for the odds ratio when the outcome is common

34. Factors Associated with Resident LP Success • “Of 377 performed by trainees, 279 (74%) were successful. • “… LPs were 3 times more likely to be successful among infants >12 weeks of age than among younger infants (odds ratio [OR]: 3.1; 95% confidence interval [CI]: 1.2–8.5).” • Success rate: 70% vs 88% Pediatrics 2006;117:876 Q: Were LPs 3 times more likely to be successful on older infants?

35. Factors Associated with Resident LP Success Pediatrics 2006;117:876

36. RR vs OR • RR is RISK of (disease/side effects/etc) in study population divided by the risk in the comparison population • OR is ODDS of (disease/side effects/etc) in study population divided by odds in control population • When the outcome is rare, the OR approximates the RR; when the outcome is common (>10%), the OR overestimates RR

37. CONVENTION: • Use risk or odds of something BAD, and put comparison group on bottom • RR or OR > 1 means exposure is “bad” • RR or OR < 1 means exposure is “good” • RR or OR = 1 means no relationship

39. Incidence of Bacteriuria in Patients using Clean Intermittent Catheterization (CIC) vs Ileal Loop Diversion (ILD) P < 0.001 Pediatrics 1982; 70:665-9. Q: What’s wrong with this analysis?

40. Bacteremia in Febrile Nonneutropenic Pediatric Oncology Patients From abstract: There were 29 cases of bacteremia in 459 (6.3%) febrile outpatient visits by 167 patients. Pediatr Blood Cancer 2010;54:83–87

41. Incidence of vomiting 2.9% (32 during 1115 feedings) vs 5.4% (67 during 1248 feedings), p=0.001 Q: What’s wrong with this analysis?

42. The perfusion index derived from a pulse oximeter for predicting low superior vena cava flow in very low birth weight infants Perfusion index and SVC flow measured up to 4 times on 27 VLBW infants Reported sensitivity = 7/8= 87.5% (95% CI 47.4 to 99.7%) and specificity = 69/80 = 86.3% based on the table below. What is wrong? J Perinatol 2010;30:265-69

43. Regression Coefficients for Cadmium and Covariates on IQ at Age 5 yrs * P < 0.05 Environ Health Perspect 2009;117

44. What is wrong with this table? Pediatrics, Apr 2009; 123: 1108 - 1115

45. Interpreting Coefficients Q1: What’s the effect of birth weight on survival? Q2: What’s the effect of gestational age? Pediatrics. 2009. Vol. 123:494-502

46. Standardized Betas What do these coeffi-cients mean? Pediatrics, Nov 2008; 122: 1051

47. Unknowns

48. Unknown #1 Number (%) of children with temperature ≥38ºC after receiving DTaP vs DTP vaccine Q: What is wrong with how the P-value was obtained? * P-value for 5/252 vs. 5/65 = 0.032 Peds ID 1993;12:131-5

49. Unknown #2: The Beneficial Effects of Early Dexamethasone Administration in Infants and Children with Bacterial Meningitis Q: How did they calculate the relative risk? “At follow-up examination… 7 of the 51 dexamethasone-treated children (14 percent) and 18 of 48 controls (38 percent) had one or more neurologic or audiologic sequelae (P = 0.007); the relative risk of sequelae for a child receiving placebo as compared with a child receiving dexamethasone was 3.8 (95 percent confidence interval, 1.3 to 11.5).” N Engl J Med 1991; 324:1525-31

50. Unknown #3: Randomized trial of Antibiotic Prophylaxis after Acute Pyelonephritis Absolute Risk Increase in renal scars = 5.6% (95% CI: -3.3% to 14.6%) Q: Based upon the 95% CI for the ARR in renal scars, if treatment actually worked, what is the lowest number of children with reflux you’d need to treat to prevent scarring in one child? Pediatrics 2006;117:629