How to Torture Your Statistician:

1. How to Torture Your Statistician: Ben Herman ACRIN Biostatistics Center Brown University Providence, RI simple ways to maximize pain!

2. Acknowledgements: Although they undoubtedly would like to distance themselves from the title and the rest of this talk, ACRIN receives funding from the National Cancer Institute through the grants U01 CA079778 and U01 CA080098 * Thanks to all the web sites/sources from which I liberally borrowed much of this material (not all of which is cited)

3. Disclaimer: The view expressed in the presentation are wholly my own, they do not necessarily represent ACRIN, ACRIN’s data collection policies, Data Management, real statistics, or the general views of the Biostatistics Center. Shhh… nobody knows I’m giving this talk.

4. Charlatans, Liars & Cheats? And Everybody Else “The group was alarmed to find that if you are a labourer, cleaner or dock worker, you are twice as likely to die than a member of the professional classes”, from The Sunday Times, 31st August 1980.

5. Fortune Example • As quickly as you can do the following: • 2 + 2 + 2 = ? • 7 + 7 + 7 = ? • What is the first VEGETABLE that comes to mind? • Tomatoes are a fruit…… • Did you say Carrot? • 98% of normal people do!

6. Know your subjects Why are Cooperative Group statisticians so hard to break? • Not beholden to the hypothesis • No vested interest in the outcome • Can make objective assessments &recommendations to the DSMC

7. Know your subjects • Normal stressors What does an ACRIN Statistician Do? Opportunities for Mayhem! • Study Design and Analysis Plans • Monitor Trial Progress • Aggregate Information • Report to Monitors (DSMC, NCI, CIP, etc.) • Data Analyses and Reports/Papers Trial

8. Know your subjects • Normal stressors • Tools of the trade The soft Underbelly of Statistics Hit ‘em Where it Hurts! • Information • Information • Information

9. Know your subjects • A $50 Million a year company has entry level positions open for people willing to work their way up. The company pays $30M in salary compensation to its 150 employees. Therefore, the Average salary at the company is $200,000/yr should you take a job? Max = $19M Median= $10K Mean = $200K Median= $200K Mean = $200K

10. Know your subjects Use general terms like average Never disclose your assumptions and bin your data Need Blinding or other interesting study designs Expect P-values but ignore power(until the end) Refuse to accurately identify the Sample/Population Vigorously deny any potential sources of Bias Explore every possible hypothesis conceivable

11. Examples: Conclusions As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomized controlled trials. Advocates of evidence based medicine have criticized the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organized and participated in a double blind, randomized, placebo controlled, crossover trial of the parachute.

12. Examples: • Power v P: measures of Probability not Pain • Null Hypothesis: The effect we are trying to DISPROVE • Alpha (a): Probability of being WRONG! Set a priori • (Falsely rejecting the Null Hypothesis) • Power • Probability of being right given the assumptions. (Correctly rejecting the Null Hypothesis) • P-Values • A measure of evidence under the NULLHypothesis

13. Examples: Examples Were gonna be rich! Hypothesis: Roulette table pays off red 20% or more than black Null Hypothesis: Black and Red occur equally (%R-%B<20%) Alpha: reject at the a=5% level (I.e., P-Value <0.05) 80% Power: used to calculate sample size = 100 BAD Unplanned looks at the data /Multiple looks at the data -- If you look at the data 20 times you expect to discover at least 1 false significant result Result: 61 red of 100 cases (P>0.05) "No one can possibly win at roulette unless he steals money from the table while the croupier isn't looking." — Albert Einstein

14. Examples: Population: The group you want to study (generalize) Sample: The subjects you actually study Population: Americans meeting screening guidelines for CRC Sample: 1600 Asymptomatic Americans >50 yrs old using Protocol specified prep, technique, parameters, etc. To maximize pain at analysis: Refuse to identify any abnormality observed during accrual that might allow the statistician to subset or re-categorize the data.

15. Examples: • Birth Control is 99.9 effective when used according to directions • 1 in 1000 fail but what is the population they are talking about • Is it a single regimen? • Is it a single dose? • Is it per person (1 out of every 1000 users)? • Who are the Failures? • Is there something special about these cases? • How do you collect this special information

16. Examples: • Assumptions may introduce bias into the study • -Forms will always be completed a certain way • -Procedures will always go as planned • -Technical data/Lab values are not significant • -A Yes/No response will be sufficient to answer the question • -All Bias is identifiable • Bias: a systematic error that may alter the outcome • -Approach only those patient you think will complete • -Only help some people complete forms • -Unblind readers to results • Beware: statisticians have ways to correct/report on some forms of bias if they know about it – So never let them know about it until it is too late!

17. Examples: The MitchySchnall Problem The Monty Hall Problem 2 1 3

18. Examples: The MitchySchnall Problem If the Host is biased Switch If the Host is unbiased the probability after switching is 50% 1/3 Chance 2/3 Chance 2 1 3 1/3 1/3 1/3

19. Why are statisticians so secretive? How do we get those secrets out of them? Don’t ask, Don’t Tell

20. Don’t Ask: • Reduce Bias by • Randomization: choose people or treatments at random (in a reproducible manner) • Masking (Blinding): Do not let the patient know their treatment assignment • Double Masking: Don’t let anyone know the treatment assignment • Approach everyone in a predefined systematic manner

21. Don’t Ask: • Feedback loops (tell me how I’m doing) • Past events changes current events • Data Mining/ Exploration/Hypothesis generation • Collected data is explored for correlations/associations • A Large number of “Chance” associations must be explored • Hypothesis testing • A Hypothesis is developed • Data is collected to test the Hypothesis

22. Don’t Ask: • Feedback loops or “Tell us how good are we doing!” • Research is not Training it is hypothesis testing Training and testing data must be kept separate. • Defeats the purpose of masking • Leads to uncorrectable bias and unstable performance • Introduces unknown confounders into the analysis • Data Mining or “Maybe it was this effect!” • At the a=0.05 level, we expect 5% of comparisons to have an effect size that exceeds the threshold. • Associations are not causal relationships

23. Don’t Ask: • DNA testing is 99.99% accurate • It’s wrong in 1 out of 10,000 • If a DNA database has 20,000 individuals • 86% chance of matching a random donor • If a DNA database has 40,000 individuals • 98% chance of matching a random donor • Should we have a national DNA database?

24. KISS (Keep it Simple, Stupid!) The biggest threat to the Primary Aim of a study are the Secondary Aims. After the key elements required for analysis and monitoring, collect whatever is easy and meaningful.

25. Humans Vs Computers Statisticians Vs Humans Data Vs Information Queries What Happens:After you hit enter

26. After you hit Enter: Data Entry (DM/HQ) Central Database Biostats Data Collection Patient Level CRAs Analysis Database DM Web Based DE (CRA/Sites)

27. After you hit Enter: Lady GIGO Garbage in, Garbage out Computers process numbers Humans interpret everything “5” + “A” Computer sees 53+65=118 (“v”?) “5”+ “Bob” = ??? Minimize errors: • Double-data entry • Real time data queries • Range/Logic checks • Cross form validation

28. After you hit Enter: Lady GIGO (cont) Cooperative Garbage Statisticians don’t like data from cooperative/outside groups Data form out side: • Different Aims • Not familiar with our study • Different priorities • Inability to query in a timely manner • Long delays in getting data • No control of data • Hard to audit source • Hard to assign responsibility for data

29. After you hit Enter: • Statisticians • Interact with Computers and Data • Aggregate Data • Don’t generally care about individual cases • Want all meaningful data available • RAs/PIs/DM • Interact with People and Data • Focus on individual cases • Deal with individual data elements

30. After you hit Enter: Current PHS Policies on Research Misconduct (42 CFR Parts 50 and 93) define falsification as, "manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.” • RAs: Want data to be as accurate as possible so put additional information on forms and in comments. • DMs: Want database to accurately reflect what is on the forms • Stats: Want data that is meaningful and analyzable in aggregate Nobody wants to manipulate data, so who corrects the data?

31. Analyze this Let the experts show you how it’s done

32. Analyze this Let the experts show you how it’s done COMMENTS 1tiny multifocal nodule inf sag loc 79.3 22 mm 79/232 47 ML, MIP 107 Sag 92/224 A/P MIP ML MIP, S-I MIP ActuallySeroma not cyst, measured on MIP add area enhancement w multi lob dommas anterolat mass likely = more ca, lge ax. nodes; el 176=1 area of enh.in mass ext. from it is stip area of lateral enhance.,indeterminate assoc. field effect superimposed on mass broad area 6:00, narrow at 12:00 can't evaluate axilla-fatsat failed dominant mass w/assoc.uncontainedenhanc e174 = unable to assess E6=10 E93,94=unknown E93=SEVERE E93=severe extensive proably invades nipple has 2-3 ductal extensions+5mm satellite long ax actuallyAP~60mm nr pectoral musc long dia = oblique cc,meas on AP/coronal many morphologic patterns mass enhance pattern N/A, remove gradual E53=can't assess don't use case for vol/ser MIP measurements MIP-surrounded by fluid ML&APax87.9ser 8/14SIsgse7/14I33/66L90.1 multicentricspiculated masses;e93=severe Multifocality & area enhan leads to ty C None E93=severe not enough room for comments post-surq changes in axilla prior sentinal node biopsy Pt is post-surg biopsy & ax. LN dissect Q17=3 rim enhancing cyst upper inner quadrant T2ax 19/39, 95/224

33. How to Torture Your Statistician: simple ways to maximize pain!