Chapter 16 Treatment Outcomes: Research Methods and Data Analysis

1. Chapter 16Treatment Outcomes: Research Methods and Data Analysis

2. Chapter Overview • Understand why assessment of the effectiveness of therapeutic interventions on patient outcomes is the centerpiece of clinical research. • Be able to describe the tenets of evidence-based practice. • Appreciate the value of evidence-based practice in making clinical decisions. • Understand that the “best” research evidence ideally consists of patient-oriented evidence from well-conducted clinical trials. • Appreciate the issues involving study design, statistical analysis, and interpretation of the results of clinical trials that assess treatment outcomes.

3. Building an Infrastructure to Measure Treatment Outcomes • Clinical trials assessing treatment outcomes must be planned ahead of time. • An infrastructure needs to be created for the regular collection of treatment outcomes (disease-oriented and patient-oriented measures). • The means by which outcomes data will be collected and managed must be determined: • paper-based surveys • entering data electronically • long term follow-up surveys administered by phone or by mail • The performance of clinical trials often involves multiple clinical sites.

4. Study Designs • The design of a clinical trial is the most important factor that influences the “level of evidence” stemming from that study’s results. • The two most common schema for classifying levels of evidence: • Centre for Evidence-Based Medicine (CEBM) • Strength of Recommendation Taxonomy (SORT) The CEBM has five general categories of levels of evidence (5 is lowest and 1 is highest). • Level 5 Evidence: Expert Opinion and Disease-Oriented Evidence (basic and translational research) • Level 4 Evidence: Case Series • Level 3 Evidence: Case Control Studies (retrospective in design) • Level 2 Evidence: Prospective Cohort Studies (lack randomization) • Level 1 Evidence: Randomized Controlled Trials (gold standard for clinical trials methodology)

5. CEBM: Five General Categories of Levels of Evidence Level 4 Evidence: Case Series • A case series represents the reporting of the clinical outcomes of a number of patients with the same pathology who are treated with the same or similar intervention. • Does not utilize an experimental design, which limits the internal validity. Level 3 Evidence: Case Control Studies • The clinical outcomes of two groups of patients are examined after interventions are administered . • Patients are treated with the respective treatments based on clinician judgment and not because of experimental allocation. Level 2 Evidence: Prospective Cohort Studies • Involve baseline measurement before patients receive the prescribed treatment.

6. CEBM: Five General Categories of Levels of Evidence (continued) Level 1 Evidence: Randomized Controlled Trials • Assigning interventions randomly limits the potential of confounding biases between groups. • Guidelines to assure quality in the design and reporting of RCTs: • CONSORT statement • PEDro scale • Dropouts in RCTs: the data should be analyzed using intention to treat analysis. • Intention to treat analysis: subjects are analyzed in their original group regardless of whether they received the full course of intervention. • Missing data points in RCTs: imputationof data should be performed. • Imputation: involves carrying the subjects’ most recent scores forward by using those scores for all future time points as well.

7. CONSORT Flow Chart Recommended in reporting RCTs to illustrate issues of subject retention throughout the entire course of the trial. (CONSORT Group, 2008. Accessed from http://www.consort-statement.org/consort-statement/flow-diagram/February 11, 2010.)

8. Data Analysis: Statistical Significance vs. Clinical Meaningfulness • It is possible to have statistically significant results that are notclinically meaningful. • It is possible for results not to be statistically significant but to have clinical meaningfulness.  • Statistical analysis assesses the statistical significance of the involved comparisons, not the clinical meaningfulness of the results. • Hypothesis testing should be viewed as an integral part of data analysis but not the only means of analysis. • Hypothesis testing does not provide information about the magnitude of mean differences between comparisons. • Measures not easily understood at face value should employ other analysis concepts: confidence intervals, effect sizes, and minimally important clinical differences may be employed.

9. Confidence Intervals • Confidence intervals are estimates of dispersion, or variability, around a point estimate. • In treatment outcomes data, the point estimate is typically a mean, or average, value for a given sample. • The confidence intervals between two comparisons can be evaluated visuallyby assessing whether the confidence intervals overlap. • Confidence intervals are determined by M+w. • M = point estimate • w = width of the confidence interval • W = Tc(SE) • Tc= an established critical value to be used as a multiplier • SE = standard error or SD/√n • The width is influenced by two factors: the variance in the data (indicated by the SD) and the sample size.

10. Effect Sizes • Effect sizes provide an estimate of the strength of a treatment effect and an indication of the meaningfulness of results. • Effect sizes are on a standardized, unitless scale. • There are several ways to compute effect sizes. The most straightforward method is Cohen’s d. • Effect sizes greater than 0.8 are considered “strong.” • Those between 0.5 and 0.8 are considered “moderate.” • Those between 0.2 and 0.5 are considered “small.” • Those less than 0.2 are considered “weak.” • In general, effect sizes are reported as positive values unless there is a specific reason to report them as negative values.

11. Minimally Clinically Important Difference • ROC curves applied to analyses of data related to treatment outcomes and prognosis. • Minimally clinically important difference (MCID) orminimally important difference (MID): avalue established when ROC analysis identifies the change in a health status measure associated with improvements meaningful to patients. • ROC analysis is used in diagnostic studies when the measure in question is continuous rather than dichotomous. • Continuous measure: the number of factors present following patient evaluation that are related to the outcome of treatment. • Multiple approaches to estimating an MCID: ROC analysis, use of standard error measurement values and effect size estimates. Use of a ROC analysis is not always preferable.

12. Clinical Prediction Rules • Flynn et al. (2002) derived a clinical prediction rule to guide the treatment of patients with low back pain with spinal manipulation. • Ultimately, five factors were selected that best discriminate between patients who have a favorable response to spinal manipulation, defined as a greater than 50% improvement on the Modified Oswestry Disability Questionnaire, and those who do not. • The papers by Stucki et al. (1995) and Flynn et al. (2002)  are examples of the utility of ROCs across varying research purposes.  

13. Comprehensive Data Analysis • Traditional hypothesis testing is the most common way to analyze data statistically, but it has limitations in assessing clinical meaningfulness. • Alternative techniques such as: • confidence intervals • effect sizes • minimal clinically important differences should be used with hypothesis testing as part of a comprehensive data analysis plan.

14. Chapter Summary and Key Points • The importance of building a strong research infrastructure for clinical outcomes research cannot be overemphasized. • The design of a clinical trial is the greatest factor influencing the “level of evidence” stemming from that study’s results. • A case series represents the reporting of the clinical outcomes of a number of patients with the same pathology who are treated with the same or similar intervention. • Case control studies are by definition retrospective in nature. • The randomized controlled trial (RCT) is the “gold standard” for clinical investigation. • The individual responsible for coordinating the randomization should not be directly involved in deciding the inclusion criteria of potential subjects or the measurement of outcomes directly from patients. • Hypothesis testing does not provide information about the magnitude of mean differences between comparisons; this is the limitation of this technique.