Experimental designs

Experimental designs The strongest of the research designs Image: www.freeimages.co.uk

Categories of research • Quantitative • Involves numerical data that result from taking measurements on subjects • Is objective • Deductive reasoning • Is used to test theories or ideas to determine whether or not they are true • The researcher is an objective observer Evidence-based Chiropractic Image: www.freeimages.co.uk

Categories of research (cont.) • Qualitative • Involves data derived from words e.g., questionnaires or interviews • Is subjective • Inductive reasoning • Reasoning based on observations which are used to create an idea or theory • The researcher actively involved at times Evidence-based Chiropractic

Quantitative vs. qualitative research • Quantitative research employs the scientific method and is usually regarded at a higher level • But may have limited relevance to clinical practice because of strict methods • Qualitative research often leads to quantitative studies • Both forms of research are important Evidence-based Chiropractic

Pragmatic and explanatory research • Pragmatic research • Used to verify the effectiveness of treatments • i.e., whether they work under real-life conditions • Does not determine how or why the treatments work • Typically used to help make decisions about the effectiveness of new treatments compared with existing treatments Evidence-based Chiropractic

Pragmatic and explanatory research (cont.) • Explanatory research • Used to establish the efficacy of treatments • i.e., how they work under ideal conditions, as in a controlled experiment • Capable of answering questions about how and why treatments work • Strict methods involved are often very different from day-to-day clinical practice • Consequently, results may not be relevant to practitioners Evidence-based Chiropractic

Pragmatic and explanatory research (cont.) • Patient selection is more strict in explanatory studies • Patients are excluded because of things like co-morbid conditions, prior treatment, severity of the condition, age, etc. • This may be a disadvantage because it is not known whether the treatment will work for patients in everyday practice • Patients commonly present with many of the exclusion criteria Evidence-based Chiropractic

Descriptive, relational, and causal research • Descriptive (observational) research • Observes and records various aspects of participants in a study • Descriptive statistics involved • Relational research • Considers relationships that may exist between variables • Correlation and regression Evidence-based Chiropractic

Descriptive, relational, and causal research (cont.) • Causal research • Explores whether an intervention causes or affects one or more outcome variables • The most demanding type of research that involves very detailed methods • Looks for statistically significant differences between groups Evidence-based Chiropractic

Experimental and quasi-experimental research • Experimental research • Random assignment to groups is involved • Capable of determining cause-and-effect relationships • Quasi-experimental research • No random assignment • Provides much less evidence about cause-and-effect relationships Evidence-based Chiropractic

Experimental and quasi-experimental research (cont.) • Non-experimental research • Does not involve random assignment or even a comparison group • Merely involves the observation of one group before and after an intervention Evidence-based Chiropractic

Research design notation • R – random assignment • O – observation or measure • X – treatment or intervention • N – non-equivalent groups • The classic experiment • Randomization and 2 groups Each row represents a group Time Evidence-based Chiropractic

Research designs • A quasi-experiment • 2 groups but no randomization • Non-experiment • Only 1 group Evidence-based Chiropractic

Population • The units from which a sample is drawn • May include people, but can also consist of events or observations • It is rarely possible to include each and every unit of a population • Instead, a smaller number of units (a sample) are selected to represent the entire population • Defined as a subset of observations from a population Evidence-based Chiropractic

Samples • Samples can permit inferences about what is happening in a population based on what is observed in a sample • However, the sample must be representative of the population • Often achieved through random selection of the sample units whereby each unit of the population has an equal chance of being selected Evidence-based Chiropractic

Sample selection A sample is selected Evidence-based Chiropractic

Samples (cont.) • Population parameters that are estimated from random samples are known as unbiased estimates • Random sampling is rarely employed in clinical trials • Patients are obtained using sequentially presenting patients or recruiting through advertisements • Referred to as convenience sampling Evidence-based Chiropractic

Samples (cont.) • Selection criteria in clinical trials • Patients are usually included in a clinical trial only if they meet certain criteria • e.g., severity of the condition, no secondary conditions, history, age, etc. • It is important to consider features of the population in a study when applying its results to a specific patient Evidence-based Chiropractic

Random assignment • Clinical trials often employ random assignment (a.k.a., randomization) • Refers to the way patients are assigned to groups • Used to make groups equivalent regarding prognostic factors (e.g., pain levels) • Sometimes called probabilistic equivalence because there is still a chance the groups will be a different after randomization Evidence-based Chiropractic

Random assignment (cont.) • Blocking • Subjects are separated into homogeneous subgroups based on factors such as age or disease severity • Enhances comparison because the subgroups are more alike than the intact groups Evidence-based Chiropractic

Random assignment (cont.) • Stratified randomization • Intact groups are separated into subgroups based on prognostic factors • e.g., trauma vs. non-trauma patients in a whiplash study Evidence-based Chiropractic

Random assignment (cont.) • Concealment • Assignment is often concealed from researchers to avoid the temptation of allotting patients with certain traits to groups that will receive special treatment • When concealment is inadequate, the apparent effects of the treatment may be distorted as much as or more than the size of the effect being investigated Evidence-based Chiropractic

Sample size determination • Articles about clinical trials should discuss why the number of subjects was chosen • Ethically important • Because no more subjects should be inconvenienced or put at risk than required to find a treatment effect • Economically • Extra resources required to include unnecessary subjects Evidence-based Chiropractic

Sample size determination (cont.) • Too few subjects reduces the power of a study so that a treatment effect may not be noticeable when it actually is present • Extremely large samples may show statistically significant differences between groups that are so small they are not clinically important Evidence-based Chiropractic

The randomized controlled trial (RCT) • Regarded as the ultimate research design in health care • The classic experiment Evidence-based Chiropractic

Placebo • An inert substance or treatment • Compared to the active substance or treatment in RCTs • Used in pharmaceutical trials to establish whether an active drug is more effective than a placebo • The drug and placebo groups are compared to determine if the drug resulted in a statistically significant treatment effect Evidence-based Chiropractic

Sham • A non-therapeutic intervention that imitates the real treatment • Similar to placebo, but refers to something done rather than something taken • Patients should have a very difficult time telling the difference between a sham and the real treatment • A sham chiropractic manipulation is difficult to produce Evidence-based Chiropractic

Treatment effect • The result that a treatment has on outcomes that is attributable specifically to the effect of the intervention • The difference between the mean outcomes observed in a treatment group and a control group Evidence-based Chiropractic

Why patients improve • Natural history • Many acute and some chronic pain conditions resolve on their own • Actual effect of the treatment • Nonspecific effects of the treatment • Linked to the treatment, but actually due to factors other than the active components of the treatment • Sometimes called placebo effects Evidence-based Chiropractic

Components of treatment Evidence-based Chiropractic

Effectiveness of a treatment • Both the placebo and treatment groups typically improve • The difference between groups at the conclusion of the study is what matters • The treatment is considered effective if the mean outcome of the treatment group is significantly better than the placebo group Evidence-based Chiropractic

Bias • Systematic errors in a study that are caused by problems with • The selection or assignment of patients to groups • The measurements involved in the study • Bias can render a study invalid, although all studies have at least some bias Evidence-based Chiropractic

Hawthorne effect • People tend to react differently when participating in experiments • Researchers found that the productivity of workers increased when they new they were involved in a study • True under a variety of conditions • Even conditions that should have reduced productivity Evidence-based Chiropractic

Hawthorne effect (cont.) • Behavior was more influenced by the attention researchers gave to the subjects than the effect of the interventions • The Hawthorne effect is a factor in all clinical studies Evidence-based Chiropractic

Types of bias • Sampling bias (a.k.a, selection bias) • During the selection process, each person does not have an equal chance of being selected from the source population • Random selection is designed to take care of this problem • Results in systematic differences between groups in experimental studies as to factors of prognosis or response to treatment Evidence-based Chiropractic

Types of bias (cont). • Random assignment with concealment is the best safeguard against selection bias in RCTs • The effect of selection bias is reduced by random assignment because it distributes the bias evenly between the treatment and control groups Evidence-based Chiropractic

Types of bias (cont). • Experimenter (researcher) bias • Examining or treating doctors may influence a study’s results because of their expectancies or desires for a certain outcome • Blinding (masking) of researchers and study participants as to group assignment can diminish the effect of this bias • This bias can be divided into detection bias and performance bias Evidence-based Chiropractic

Types of bias (cont). • Exclusion bias • Occurs when patients who drop out of a study are systematically different from subjects who remain • Perhaps dropouts were having a poor response to treatment • Would have changed the results of the study if they had remained Evidence-based Chiropractic

Extraneous and confounding variables • In experiments, researchers are able to manipulate the explanatory variables and then watch what happens to the outcome variable • Internal validity • The ability of an experiment to show that the explanatory variables actually caused the observed changes in the outcome variables Evidence-based Chiropractic

Extraneous and confounding variables (cont.) • Extraneous variables • Uncontrolled factors that can influence the relationship between variables in an experiment • They are not the variables that are being studied, yet they affect the outcome of the experiment • They are unwanted because they create error Evidence-based Chiropractic

Extraneous and confounding variables (cont.) • Error variance due to extraneous variables is distributed evenly between the groups when random assignment is utilized • Confounding variable • A type of extraneous variable that affects the explanatory variables differently • e.g., it affects the treatment group but not the control group • Introduces systematic error into the study Evidence-based Chiropractic

Extraneous and confounding variables (cont.) • The effect of a confounding variable cannot be separated from the outcome variable Explanatory variable e.g., manipulation Outcome variable e.g., low back pain Confounding variable e.g., groups receive manualvs. instrument manipulation Evidence-based Chiropractic

Extraneous and confounding variables (cont.) • Quasi-experimental designs are particularly susceptible to confounding because the individual differences of subjects may act as confounding variables • For example • A quasi-experimental study that assigned headache patients with more severe pain to the treatment group Evidence-based Chiropractic

Threats to internal validity • History • Participants are unintentionally exposed to some historical event during the research project which affects the results • For example • A statewide fitness campaign that coincides with a lower back pain study • Some of the subjects doing the exercises would likely affect the study’s outcome Evidence-based Chiropractic

Threats to internal validity (cont.) • Reliability of measures • Unreliable measures can invalidate a study • Possible causes • Faulty equipment, inconsistent instructions to study participants, unreliable training of examiners, fatigue or boredom of examiners, or examiners becoming more skilled at doing the test Evidence-based Chiropractic

Threats to internal validity (cont.) • Mortality • Subjects dropping out of studies • Drop-outs may be different from those who remain • Occurs for a variety of reasons • e.g., poor response to treatment, exceptional response to treatment, adverse effects • Groups may not be equivalent as a result Evidence-based Chiropractic

Threats to internal validity (cont.) • Maturation • Changes that occur in study participants as time passes that are not caused by the explanatory variables • e.g., in a study investigating strength in children, they would most likely get stronger in time, even without exposure to the explanatory variables Evidence-based Chiropractic

Threats to internal validity (cont.) • Regression to the mean • Extreme scores at the beginning of a study that migrate toward the mean as time passes • Occurs because extreme symptoms tend to return to a more normal state on their own • i.e., high initial patient scores are much more likely to move toward normality than to go even higher • Especially problematic when patients are selected based on high test values, while patients with low values are screened out Evidence-based Chiropractic

Read and bring to class Monday 11/3/08 • Hoiriis et al. A Randomized Clinical Trial Comparing Chiropractic Adjustments To Muscle Relaxants For Subacute Low Back Pain. JMPT 2004;27:388-98 • Bakris, et al.Atlas vertebra realignment and achievement of arterial pressure goal in hypertensive patients: a pilot study. J Hum Hypertens. 2007 May;21(5):347-52. Evidence-based Chiropractic

External validity a.k.a., generalizability • The extent results of a study are applicable to other populations, other settings, and when implemented under different circumstances • Should be comparable regarding the intervention, age, condition severity, etc. • Relating to EBP – Are the results of a study applicable to the management of a particular patient? Evidence-based Chiropractic

Experimental designs