Literature review designs

1. Literature review designs Narrative ReviewSystematic ReviewMeta-analysis

2. Literature review • Defined as a systematic, explicit, and reproducible way of identifying, evaluating, and interpreting all of the research findings and scholarly work available on a topic • A high-quality review is not haphazard • Ideally, all of the existing work should be included • Considered descriptive or observational Evidence-based Chiropractic

3. The three types of literature review designs Evidence-based Chiropractic

4. Narrative reviews • Summarize in general what is in the literature on a given topic • Often written by experts in a given field • A good source for background information • Do not follow strict systematic methods like the other literature review designs • Therefore, they are prone to bias • Lower in the hierarchy of evidence Evidence-based Chiropractic

5. Narrative reviews (cont.) • Authors like to write them because they are relatively easy to carry out and compose • Practitioners like to read them because they are easier to comprehend than more complex designs • Do not require a lot of background knowledge to understand the message Evidence-based Chiropractic

6. Narrative reviews are prone to bias • They do not employ many of the safeguards needed to control against bias • Authors may be selective as to which articles are included • They may include articles that support their hypothesis and exclude those that do not • Rigorous appraisal methods are not used to evaluate included articles Evidence-based Chiropractic

7. Prone to bias (cont.) • During the literature search • Authors have their own opinions on the topic and may try to find studies that support their viewpoint and overlook conflicting studies • During the synthesis of the literature • The approach to analyzing the collected information is often subjective and disorganized Evidence-based Chiropractic

8. Prone to bias (cont.) • In the discussion and conclusion • The authors’ opinions may be mixed together with evidence • Authors may simply count the number of studies on each side of an issue and then espouse the view presented by the majority without considering the strength of each study • e.g., quality, research design used, the effect size, and sample size Evidence-based Chiropractic

9. Selection bias innarrative reviews • a.k.a., reference bias • Occurs when authors choose articles that support their own conclusions and exclude articles with conflicting views • Results in an erroneous representation of the literature • The review may lead undiscerning readers astray Evidence-based Chiropractic

10. Systematic reviews • Use strict methods to locate, appraise and synthesize all research on a topic • Similar to narrative reviews, but with improved procedural quality • Designed to answer specificclinical questions where several primary studies exist • Thus, a good source of clinical evidence Evidence-based Chiropractic

11. Systematic reviews (cont.) • Articles are evaluated using appraisal instruments • In an attempt to achieve impartiality • More reproducible • Dissimilarities between the findings of studies are investigated • Multiple reviewers are usually involved • Any disagreements are resolved Evidence-based Chiropractic

12. Systematic reviews (cont.) • The search and selection criteria for articles are well defined • Multiple databases should be searched • Explicit inclusion criteria • The results of the included studies are qualitatively or quantitatively synthesized • Qualitative – written information is merged • Quantitative – data are merged Evidence-based Chiropractic

13. Systematic reviews (cont.) • Because of strict methodology and thoroughness, conclusions are typically less biased than narrative reviews • Nevertheless, they may still be influenced by the authors’ opinions • Still a potential for selection bias • Criteria may be applied differently when appraising included studies • Therefore, must be critically appraised Evidence-based Chiropractic

14. Systematic reviews (cont.) • No widely accepted method exists for assessing the validity of studies • i.e., the process is a judgment call • Thus, reviewers sometimes disagree • Information derived from a systematic review may be too narrow to completely answer a specific clinical question • Given that they are designed to answer focused questions about patient care Evidence-based Chiropractic

15. Evidence-based Chiropractic

16. Systematic reviews typically only include RCTs • Studies are included primarily based on their quality • Most systematic reviews only include RCTs because it is the only design that adequately controls for confounding variables and biases • The potential for studies to overrate the treatment effect is higher when bias is present • Studies using other less rigorous designs are usually eliminated Evidence-based Chiropractic

17. Conducting systematic reviews Evidence-based Chiropractic

18. Search strategy • Should be described in enough detail so that another researcher could replicate the results, including: • Database(s) searched • Date the search was performed • Time-frame encompassed by the search • A list of search terms used • Languages Evidence-based Chiropractic

19. Search strategy (cont.) • Conference proceedings, unpublished studies, and hand-searching of journals are sometimes included in the search • If so, the procedures involved should be described • Authors must justify using unpublished work Evidence-based Chiropractic

20. Weighting of studies • Articles may be rejected in a systematic review due to their poor quality • Alternatively, studies are assigned weights in relation to their assessed validity • Studies that are more valid will have more influence on the review’s final results • Based on methodological quality, width of the confidence intervals, and external validity Evidence-based Chiropractic

21. Publication bias • Studies with statistically significant results are more likely to get published than those with non-significant results • Causes of publication bias: • The author or funding source does not consider a “failed study” worthy of submission • Journals are less likely to publish studies that fail to show positive results Evidence-based Chiropractic

22. Publication bias (cont.) • Reviews affected by this bias tend to give an overoptimistic view of the effectiveness of the therapy • The chance of this bias occurring is reduced when authors of systematic reviews search sources other than journals • Publication bias in situ • A type of bias where a portion of a study’s results are suppressed Evidence-based Chiropractic

23. Meta-analysis • a.k.a., quantitative systematic review • A type of systematic review that statistically combines the results from a number of studies • Capable of producing a single estimate of the effect of a treatment • Represents the “average” treatment effect • An estimate of the true treatment effect size Evidence-based Chiropractic

24. Meta-analysis (cont.) • The same explicit methods as systematic reviews are utilized • Systematic reviews and meta-analyses are at the top of the hierarchy of evidence because of their strict methodology Evidence-based Chiropractic

25. Weighted average • Meta-analyses typically produce a weighted average for the treatment effect estimate • Small samples are more susceptible to chance variations than larger studies • Thus, they are given less weight than larger studies so they will have less influence on the final estimate Evidence-based Chiropractic

26. Weighting (cont.) • Weighting is also based on study quality • The quality of the individual studies is rated and resulting numeric scores are calculated • A corresponding weight is assigned for each study prior to analysis Evidence-based Chiropractic

27. Meta-analyses can increase power • Data from individual studies are combined, which in effect increases sample size • Chiropractic studies commonly involve too few subjects to detect true differences between the groups • Pooling data reduces the potential for type II error • More likely to detect a treatment effect, if there actually is one Evidence-based Chiropractic

28. Homogeneity and heterogeneity • Homogeneity • Similarities of included studies that allow them to be compared • Homogeneity is preferred in meta-analyses • Achieved by using suitable inclusion criteria • Heterogeneity • Dissimilarities of studies that hamper or even prevent a realistic comparison of studies Evidence-based Chiropractic

29. Factors that contribute to heterogeneity • Heterogeneity in the study samples • Caused by conflicting inclusion and exclusion criteria, differences in patients’ baseline health status, dissimilar geographical locations of groups, etc. • Heterogeneity in the study design • e.g., the way dropouts were managed in the statistical analysis or the length of time allowed for patient follow-up Evidence-based Chiropractic

30. Factors that contribute to heterogeneity (cont.) • The way patients were handled • Regarding comorbid conditions, handling of complications, the control practitioners had in patient care, or the outcome measures used • Statistical heterogeneity • When the observed treatment effects of studies are more dissimilar than what would be expected by chance Evidence-based Chiropractic

31. Consequences of heterogeneity • When the results of studies in a meta-analysis are inconsistent, it reduces confidence in its conclusions • The meta-analysis may actually be worthless if too dissimilar • For instance, combining studies that used different types of comparison groups • Or outcomes that were dissimilar Evidence-based Chiropractic

32. Forest plot • A type of graph often used in meta-analyses to illustrate the treatment effect sizes of the studies • Each study is represented by a black square that is an estimate of their effect sizes • A horizontal line extends to either side of the squares, the 95% confidence interval Evidence-based Chiropractic

33. Forest plot Evidence-based Chiropractic

34. Interpreting a forest plot • If a study’s 95% CI crosses over the vertical line, it is not statistically significant • A diamond with a CI line is sometimes presented at the bottom of the forest plot to represent an overall estimate • The black squares may vary in size representing the weights of the studies Evidence-based Chiropractic

35. Weighting and overall effect Evidence-based Chiropractic

36. Effect size • The difference between the means of the treatment and control groups • When studies are combined in a meta-analysis, the units of measurement are not always comparable • Effect sizes are standardized to resolve this problem producing the standardized mean difference Evidence-based Chiropractic

37. Standardized mean difference • The effect size divided by the pooled standard deviation • Pooled standard deviation has been adjusted for the differences in the sizes of the groups • Represents the standardized difference between group means • i.e., the relative magnitude of the experimental treatment Evidence-based Chiropractic

38. Cohen’s d Evidence-based Chiropractic

39. Odds ratio (OR) • Cohen’s d is appropriate with continuous data • An OR is appropriate when the study’s outcome measure is dichotomous • e.g., pain versus no pain • OR is a comparison of the odds of the outcome being present in the treatment group against the control group Evidence-based Chiropractic

40. Relative risk (RR) • a.k.a., risk ratio • A comparison of the risk of having the outcome in the treatment group with that of the control group • Sometimes OR is reported and sometimes RR • Experts do not agree on which is most appropriate Evidence-based Chiropractic

41. OR vs. RR • Consider a hypothetical lower back pain study with 25 patients in each group • 5 in treatment group and 10 in control group are still in pain at the study’s end OR = = = .38 RR = = = .5 Evidence-based Chiropractic

42. Meta-analyses are most valid with RCTs • However, about half of meta-analyses include observational studies • Primarily cohort and case-control • Observational studies are much more susceptible to biases and confounding than RCTs • Therefore, it is usually inappropriate to statistically combine the results of such studies Evidence-based Chiropractic

43. Spinal motion palpation: A comparison of studies that assessed intersegmental end-feel versus excursion • The objective of this review was to classify and compare studies based on method of MP utilized • i.e., excursion versus end-feel methods • When only high-quality studies were considered, • 3 out of 24 end-feel studies reported good reliability compared • 1 out of 15 excursion studies. • No statistical support for a difference between the two groupings Evidence-based Chiropractic

44. Evidence-based Chiropractic

45. Evidence-based Chiropractic

46. Subgroup analysis • Meta-analyses typically include patients with a variety of characteristics • e.g., age, gender, condition severity, patient history, etc. • Patients in these subgroups may respond to treatment differently • e.g., low back pain patients with leg pain may respond to treatment differently than low back pain only patients Evidence-based Chiropractic

47. Subgroup analysis (cont.) • Carried out to identify variation between patient groups regarding certain outcomes or findings • The process helps readers to distinguish the effects of a treatment between subgroups • The statistical power of the subgroups will decline as a result Evidence-based Chiropractic

48. Meta-regression • A statistical procedure that adjusts for differences between studies in meta-analyses • May be used in subgroup analyses • Similar to simple regression • Predictor variables: characteristics of the studies • Outcome variable: treatment effect estimate Evidence-based Chiropractic

49. Sensitivity analysis • A type of subgroup analysis that considers non-patient characteristics, e.g., treatment variations or study methodology • Determines the extent heterogeneity affected the results of a meta-analysis • If the results are weak, sensitivity analysis may reveal significant treatment effects when different methods are used Evidence-based Chiropractic

50. Narrative versus systematic reviews and meta-analyses • There are no strict rules regarding the creation of either type • Therefore, it may be difficult to decide if a given review is systematic or narrative • Narrative reviews do not typically use systematic methods • They tend to be subjective and prone to bias • Cover broader topics than systematic reviews Evidence-based Chiropractic