SYSTEMATIC REVIEW META-ANALYSIS

1. SYSTEMATIC REVIEW & META-ANALYSIS

2. outline Systematic review, narrative review and meta-analysis Roles of SR and MA in Public health policy, decision making and guidelines Steps of SR New types of SR and MA (Cumulative, observational and diagnostic studies) Hierarchy/quality of evidence and evidence pyramid

3. THE NEED FOR SYSTEMATIC REVIEWS OF MEDICAL EVIDENCE Over the past half century we have witnessed exponential growth of scientific research and its application in industry as well as daily life. Medicine is not an exception. The medical literature has grown dramatically with over 2 million articles published yearly in more than 20,000 journals Keeping up with this is impossible for medical professionals... In addition we are unable to access all the relevant original articles published on a single topic Although if we had access we could not use them for practice More over, medical questions that arise in everyday clinical practice are often complex.

5. All medical aspects including decisions, tests, therapies, etc suffer from unavoidable errors� For example, a woman who has a false positive mammogram suggesting abnormalities is a candidate for a breast biopsy. While the risk of serious complications resulting from a breast biopsy is low, it still carries small risks of bleeding and infection, and it also creates anxiety in the patient. For other potentially more dangerous tests like liver biopsy, serious bleeding may lead to hospitalization. Hospital admission may lead to other complications, such as acquiring an infection while in the hospital. In all, every action in medicine has to be carefully balanced for its potential benefits and risks, and it is often difficult to take into account all the possible interactions and developments in patients. THE NEED FOR SYSTEMATIC REVIEWS OF MEDICAL EVIDENCE

6. Therefore, �review articles� have gained wide popularity among doctors. These �traditional reviews� are usually written by an expert in the field, either on a personal interest or more commonly on a contract by major journals. These �traditional reviews� provide useful information on general aspects of diseases or new progress. They usually have a general title, are extensively referenced, and reflect points of view of their author(s). The problem rises when you look at different major articles or major reviews on a single topic, which in many instances end up with different conclusions.

7. What is the source of dispute, as these authors are expected to have looked at the same literature? Is there any difference in their resources? Have some of them missed some important publications? Are the authors biased towards their own research and points of view? Have they looked differently to the same research work? Are there research works not published in major English language journals, therefore not available to many who need it, i.e., is there some sort of publication bias? Can the statements in the review be generalized to other (especially our own) conditions? And at last, what can be done to minimize all these?

8. So we know the following: information is produced at a rapid rate, summarizing the new information at a timely manner is essential to modern medical practice, good reviews are necessary to update users (i.e., medical professionals), traditional reviews, despite their usefulness, suffer from several major and sometimes fatal drawbacks. �systematic reviews� have been born to respond to these needs.

9. What is a �systematic review�/ �Meta-analysis�? "Systematic Review = Overview = the application of scientific strategies that limit bias to the systematic assembly, critical appraisal, and synthesis of all relevant studies on a specific topic." Potsdam Conference on Meta-analysis (Cook, Sackett & Spitzer 1994) A systematic review is a comprehensive assessment of the medical literature on a topic of interest using a priori specified rules for the search, identification, and eligibility of the appropriate studies and for the abstraction of relevant data�

10. The systematic nature of the process according to clear-cut rules differentiates a systematic review from a traditional review. Systematic reviews currently have assumed a key place in the generation of recommendations for medical practice and for clinical decision-making. As of the fall of 2003, the Cochrane Library includes 1,754 completed Cochrane reviews and 1,304 protocols for ongoing reviews. Concomitant with this growth, systematic reviews are playing increasing roles in clinical and health care policy decision making.

11. The general approach to conducting a systematic review consists of the following steps: Formulate answerable research question(s) Define the systematic review (meta-analysis) protocol (establish inclusion and exclusion criteria) Perform literature search Screen titles and abstracts of literature search results for potentially relevant studies and retrieve full articles Evaluate full articles according to criteria Critically appraise articles that met criteria Extract data for analysis Perform meta-analyses and sensitivity analyses

12. Depending on the question and the eligibility criteria that are considered appropriate for selecting studies on the question(s)� some systematic reviews may include a large number of eligible studies, while others may systematically and precisely investigate hundreds and thousands of references from the medical literature, only to conclude that no eligible study exists that directly addresses the question of interest. Systematic reviews that identify such lack of evidence are still useful since they clearly show the direction for designing new studies. Even in the presence of data, systematic reviews may often conclude that the available evidence is insufficient, controversial, or inconclusive and that further studies are needed. Even in the presence of systematic methods for locating and appraising evidence, a systematic review can hardly be conclusive unless the collected data can be appropriately synthesized in a quantitative fashion.

13. How much work is a SR? ~ 1139 hours ~ 30 person-weeks of full-time work 588 for protocol, searching and retrieval 144 for statistical analysis 206 for report writing 201 for administration Source: Allen IE. JAMA, 1999;282:634

14. Meta-analysis "A quantitative method of combining the results of independent studies (usually drawn from the published literature) and synthesizing summaries and conclusions that may be used to evaluate therapeutic effectiveness, plan new studies, etc., with application chiefly in the areas of research and medicine." The National Library of Medicine (1989) "Meta-analysis = Quantitative Overview = a systematic review that employs statistical methods to combine and summarize the results of several studies." Potsdam Conference on Meta-analysis (Cook, Sackett & Spitzer 1994) Only 13 meta-analyses were indexed in Medline in the decade from 1970 to 1980 compared with 416 in just 3 years between 1993 and 1996.

15. Meta-analysis in clinical research is based on simple principles: systematically searching out, and� when possible, quantitatively combining the results of all studies that have addressed a similar research question. A meta-analysis is a quantitative synthesis of data from various sources addressing the same question; a systematic review is a prerequisite to a good meta-analysis. Well conducted meta-analyses allow� A more objective appraisal of the evidence than traditional narrative reviews, Provide a more precise estimate of a treatment effect, May explain heterogeneity between the results of individual studies.

16. Rule 1: ��Garbage in, garbage out,��� Systematic reviews cannot improve on the quality of the original research being reviewed In theory, aggregation of data from multiple trials should enhance the precision and accuracy of any pooled result, but� differences in the direction or size of treatment effects may be caused by other factors, including subtle differences in treatments, populations, outcome measures, study design, and study quality. Data access due to� electronic databases are useful, but, used alone, they may miss a substantial proportion of relevant studies. Indexing problems, language differences,� Sometimes the problem for meta-analysts may not be data access but data excess! in a systematic review of the effects of ondansetron on emesis data from nine trials appeared in 23 separate publications, including four pairs of almost identical reports with completely different authors. Defects of �systematic review�/ �Meta-analysis�

17. Another question is that how often MA or SR are wrong? Villar et al examined 30 meta-analyses in perinatal medicine, comparing the results of a meta-analysis of several small trials with a single large trial addressing the same topic. Directionally, 80% of meta-analyses agreed with the results from the larger trial, although concordance for statistically significant findings was much less. Cappelleri et al reviewed 79 meta-analyses and also found about 80% directional agreement. Very recently LeLorier et al arrived at a more pessimistic assessment... Comparing 12 definitive randomized trials to 19 previous meta-analyses, they claimed the meta-analyses would have led to the adoption of an ineffective treatment in 32% of cases and rejection of a useful treatment in 33%.

18. All 1995 issues of 7 widely read epidemiology journals were searched for reviews 29 reviews were found Quality of reviews in Epidemiology, Breslow R. AJPH, 1998;88:475-7

19. In sum, MA and SR have made and continues to make major contributions to medical research, clinical decision making, and standards of research reportage; However, it is no panacea!

20. steps to conduct a systematic review A review earns the adjective systematic if� it is based on a clearly formulated question, identifies relevant studies, appraises their quality and, summarizes the evidence by use of explicit methodology. It is the explicit and systematic approach that distinguishes systematic reviews from traditional reviews and commentaries. We provide a step-by-step explanation (there are just five steps) of the methods behind systematic reviewing�

21. Step 1: Framing questions for a review The problems to be addressed by a systematic review should be specified in the form of clear, unambiguous, answerable, structured and focused questions before beginning the review work. Once the review questions have been set, modifications of populations, interventions, outcomes or study designs should be followed. The research question may initially be stated as a query in free form but reviewers prefer to pose it in a structured and explicit way.

22. For example, for many years we have believed that water fluorination improves dental health� Recently there has been pressure from various interest groups to consider the safety of this public health intervention because they fear that it is causing cancer. Free-form question Is it safe to provide population-wide drinking water fluoridation to prevent caries? Structured question The populations: Populations receiving drinking water sourced through a public water supply The interventions or exposures: Fluoridation of drinking water (natural or artificial) compared with non-fluoridated water The outcomes: Cancer is the main outcome of interest The study designs: case-control

23. Step 2: Identifying relevant work The search for studies should be extensive. Multiple resources (both computerized and printed) should be searched without language restrictions. The study selection criteria should flow directly from the review questions and be specified a priori. Reasons for keywords, inclusion and exclusion should be recorded.

24. Capturing as many relevant citations as possible form a wide range of medical, environmental and scientific databases to identify primary studies of the effects of water fluorination... This effort resulted in 3246 citations from which relevant studies were selected for the review. Their potential relevance was examined, and 2511 citations were excluded as irrelevant. The full papers of the remaining 735 citations were assessed to select those primary studies in man that directly related to fluoride in drinking water supplies, comparing at least two groups. These criteria excluded 481 studies and left 254 in the review. They came from thirty countries, published in fourteen languages between 1939 and 2000. Of these studies 175 were relevant to the question of safety, of which 26 used cancer as an outcome.

25. Step 3: Assessing the quality of studies Study quality assessment is relevant to every step of a review� Question formulation (Step 1) Study selection criteria (Step 2): should describe the minimum acceptable level of design. Selected studies should be subjected to a more refined quality assessment by use of general critical appraisal guides and design-based quality checklists (Step 3). These detailed quality assessments will be used for exploring heterogeneity and informing decisions regarding suitability of meta-analysis (Step 4). In addition they help in assessing the strength of conclusions and making recommendations for future research (Step 5).

26. The objective of the included studies was to compare groups exposed to fluoridated drinking water and those without such exposure for rates of undesirable outcomes, without bias. When examining how the effect of exposure on outcome was established,� reviewers assessed whether the comparison groups were similar in all respects other than their exposure to fluoridated water. For example, if the people exposed to fluoridated water had other risk factors that made them more prone to have cancer,� the apparent association between exposure and outcome might be explained by the more frequent occurrence of these factors among the exposed group. The technical word for such defects is� In a randomized study, confounding factors are expected to be roughly equally distributed between groups. In observational studies their distribution may be unequal.

27. Step 4: Summarizing the evidence Data synthesis consists of tabulation of study characteristics, quality and effects as well as use of statistical methods for exploring differences between studies and combining their effects (meta-analysis). Exploration of heterogeneity and its sources should be planned in advance (Step 3). Step 5: Interpreting the findings The issues highlighted in each of the four steps above should be met. The risk of publication bias and related biases should be explored. Exploration for heterogeneity should help determine whether the overall summary can be trusted, and� if not, the effects observed in high-quality studies should be used for generating conclusions. Any recommendations should be graded by reference to the strengths and weaknesses of the evidence.

28. There exist a large number of statistical methods for combining data across studies... The first step usually is to evaluate whether there is any formal, statistically significant heterogeneity (diversity) between the results. The detection of formal heterogeneity does not mean that something is wrong with the data used in the meta-analysis,� but it provides a hint that the results of the constituent studies differ between themselves. Between-study differences may exist either because of genuine diversity or because of bias, or in some cases chance may have played its role. On the other hand, when the heterogeneity test is not significant, one cannot say completely that genuine diversity and/or bias may NOT exist. Methods of a Meta-analysis

29. The results of each study are given a weight inversely proportional to the uncertainty (variance) of the results,� which is a function of the total number of study subjects and the number of events. Large studies with little variance in their estimates are thus given more weight than smaller studies. In the presence of major differences in the results of the various studies, even this weighting approach may be inappropriate,� since it assumes that all studies have different results due to chance alone. Methods of a Meta-analysis

30. It has been debated whether it is better to have one single large study rather than a meta-analysis of several smaller studies� Large studies are useful, but for most medical questions of interest they are never performed. Therefore, the constellation of several small clinical studies may be all that is available. Even if large studies have been done, it is not certain that their results would be more reliable than the results of smaller studies. Most often, all studies, big and small, offer complementary evidence on a question of interest.

31. Type of Articles (n) Meta-analysis of: Controlled trials 34 Observational studies 25 Methodological article 15 Traditional review 15 Others 11 WHO Systematic review of incidence/prevalence of maternal mortality and morbidity 1997-2002

32. ELEMENTS OF A HIGH QUALITY SYSTEMATIC REVIEW State objectives of the review, and outline eligibility (inclusion/exclusion) criteria for studies Exhaustively search for studies that seem to meet eligibility criteria Tabulate characteristics of each study identified and assess its methodological quality Apply eligibility criteria and justify any exclusion Assemble the most complete dataset feasible, with involvement of investigators Analyze results of eligible studies; Use statistical synthesis of data (meta-analysis) if appropriate & possible Perform sensitivity analyses, if appropriate and possible (including subgroup analyses) Prepare a structured report of the review, stating aims, describing materials and methods, and reporting results

33. OVERVIEW OF THE HIERARCHY OF MEDICAL EVIDENCE