Is Your Study Statistically Sound?

Is Your Study Statistically Sound? • Michelle Secic, M.S. • President • Secic Statistical Consulting, Inc. • www.secicstats.com

Is Your Study Statistically Sound? • Primary goals • CRF’s • Sample size • Randomization • Ethics • Reporting Guidelines

Primary goals • Can your goals be met with the design you picked? • Primary hypothesis • Primary endpoints

Primary goals • Examples • Can goals be met with the design you picked? Paired design - pre and post intervention • All subjects get intervention and study design will allow testing the effect of the intervention. • BUT, without a control group… • can only say that the intervention had a significant effect • no statement can be made on how it compares to current standard

Primary goals • Examples • Primary Hypothesis“This study is designed to prove that device Z controls knee pain.”“This study is designed to assess the effects of device Z on local knee pain 24 hours post anterior cruciate ligament knee surgery compared to the standard device.”

Primary goals • Examples • Primary endpointHow is the primary endpoint measured?

Primary goals • Examples • Clear(easy)death = alive/deadgender = male/female • Convoluted (more decisions involved)height = feet or centimeters?pain = visual analog scale or morphine usage via self pump or via number of pills?

Primary goals • Examples • Surrogate (more complex issues)current lesion size = stage of cancer • state the appropriateness of using surrogate - references, etc. • if study is for assessing the appropriateness of surrogate, need to have longevity - to capture actual endpoint for assessment of agreement

CRF’s • Typically the last thing developed prior to submission - rushing can lead to problems of omission • With a draft of the CRF’s statisticians are able to determine if the appropriate data will be collected for answering the specific questions that the study is designed to answer

CRF’s • Examples • - Researchers claim their study will address prevention of repeated heart attacks in a population that has already suffered from a heart attack. • - Stack of CRF’s arrive and you realize there was not a CRF on family history. UGH! • - Involve a statistician!

Sample size • Big differences need fewer subjects • Small differences need more subjects difference  sample size • Little variability needs fewer subjects • Lots of variability needs more subjects variability  sample size

Sample size • Example: Percent of pressure ulcers study mattress vs. control mattress • Want to show study mattress is as good as (or substantially equivalent to) control mattress • Assume: • Control group has a rate of 20% • Allow 10% difference while still calling groups equivalent • Need approximately 200 patients in each group

Sample size • Other factors that affect sample size: • Power • Significance level • Attrition rate • Number of objectives • Number of interim analyses

Sample size Power Probability of correctly concluding that the data support the desired hypothesis: Basic Examples: - probability of correctly concluding equivalence in an equivalence trial - probability of correctly concluding superiority in a comparative trial Also called “1- beta” - should be at least 80%

Sample size Significance level Probability of incorrectly concluding that the data do not support the desired hypothesis (i.e., probability of missing the desired finding): Basic Examples: - probability of incorrectly concluding non-equivalence in an equivalence trial - probability of incorrectly concluding non-superiority in a comparative trial Also called “alpha” - should 5% or less

Sample size Attrition Rate % of subjects on which you expect incomplete data due to: - withdrawal - death - long study follow-up - complications that prevent them from completing the study

Sample size Number of Objectives An increase in the number of objectives causes an increase the required sample size objectives  sample size If there are 2 primary objectives (safety & efficacy, for example), then the power/alpha have to be adjusted to incorporate both objectives into the calculations

Sample size Number of Interim Analyses An increase in the number of interim analyses causes an increase the required sample size interims  sample size The more time you plan to ‘peek’ at the data, the more subjects you will need for your study

Sample size Number of Interim Analyses • May want to take early peeks at the data • Long studies • Expensive studies • Better efficacy than expected

Randomization What is Randomization? • A way of dividing subjects into groups in such a way that the characteristics of the subjects in the groups are balanced (i.e., similar proportions of males and females, similar ages, etc.) • To achieve this, we allow chance to decide which group each subject is allocated to, so each subject is equally likely to be allocated to any of the available groups

Randomization Why Randomize? • Want to be able to conclude that any differences found between the groups is due to the intervention not because the subjects were inherently different from the start

Randomization How To Randomize? • Toss coin • Problems with this method include: • No audit trail • Researcher can toss again, if they do not like the result • No way to prevent ‘runs’ • Computer generation • Preferred method • Excel has random number generator • Any statistician can generate the randomization scheme

Randomization Decisions in Randomization • Blocking • Purpose of blocking is to prevent ‘runs’ • Example • Block comparison groups by a size of 6 • This guarantees that each block of 6 assignments will have an equal number of treatment assignments (i.e., ABABAB, ABBAAB, BBBAAA, etc.)

Randomization Decisions in Randomization • Stratification • Purpose of stratification is to remove inherent imbalances • Example • Know there are many more women than men who develop breast cancer • Have a new preventative drug that you want to ensure works on both genders • Then you stratify by gender to ensure there are equal numbers of males and females, since you know, inherently, that randomization alone will not work.

Randomization Decisions in Randomization • Minimization • Further ensure balance in subject characteristics between groups when randomization and stratification are still not enough • Example • The first patient is randomized to either A or B. • Researcher picks all factors that need to be reviewed by the computer (i.e., age, history of risk, weight, etc.). • When subsequent subjects are recruited and their prognostic characteristics noted, their allocation is decided such that the overall imbalance in the groups at that point is minimized.

Randomization Decisions in Randomization • Blinding/Masking • Reduces bias by preventing patients, caregivers, and even statisticians from knowing who is in the experimental group and who is in the control group. • In a single-masked study, only the patients are masked. • In a double-masked study, the patients and data collectors (the caregivers, investigators, researchers, coordinators, etc.) are masked. • Although rare, in a triple-masked study, the patients, data collectors, and data evaluators are masked.

Randomization Decisions in Randomization • Procedures • Lists • Sealed envelopes • IVRS

Randomization Decisions in Randomization • Must be prepared by someone who will not be involved in the recruitment • Open lists are not preferred, since bias (intentional or unintentional) can easily enter into the process • Sealed, labeled, numbered, non-transparent envelopes are adequate for most studies • IVRS is preferred for large, long-term, multi-center studies

Ethics • Ethics (3 basic principles) (Hulley & Cummings, 1988) • inform subjects - consent • ensure benefits of research are proportionate to the risks assumed by individual subjects (not just to those who may benefit later) • no single group (disadvantaged or vulnerable) should bear a disproportionate share of the risk

Ethics • Ethics/Issues in: randomization - prevention of selection bias due to determining who is assigned to each group (Meinert, 1986) • Best for studying efficacy, but may lead to some ethical concerns • Researcher believes one device/drug is really superior, or is best for particular subgroups • Not comfortable randomizing to a non-intervention group (placebo) • Cannot have placebo arm when standard of care has already been established (i.e., heparin in cardiology trial is always considered ‘placebo’ arm)

Ethics • Ethics/Issues in:blinding - preventing intervention related bias due to measurement or ascertainment errors (Meinert, 1986) • Sometimes blinding is not possible - need to discuss how potential bias will be addressed • Especially crucial with subjective data

Reporting Guidelines • Reporting Guidelines • Thinking about how you will report your findings or present them to the public aids in designing a “good” study • Ask researcher, “What do you want your main conclusion statement to be?” • helps researcher narrow in on broad ideas such as, “I want to test this new device.” • Book with comprehensive guidelines: • How To Report Statistics in Medicine, (Lang & Secic, 1997)

Is Your Study Statistically Sound? • References • Hulley SB, Cummings SR. Designing Clinical Research. Williams & Wilkins, Baltimore, MD, 1988. • Lang T, Secic M. How To Report Statistics in Medicine: Annotated Guidelines for Authors, Editors, and Reviewers. American College of Physicians, Philadelphia, PA, 1997. • Meinert CL. Clinical Trials: Design, Conduct, and Analysis. Oxford University Press, Inc., New York, NY, 1986.

Is Your Study Statistically Sound?