Systematic Reviewing and Qualitative Synthesis

1. Systematic Reviewing and Qualitative Synthesis Workshop 1 – Introduction 16th February 2010

2. Part 1 Introducing the systematic review

3. Workshop Overview Workshop will discuss: • The methods of systematic reviews and the rationale behind their use. • How to systematically search for relevant literature. • The importance of developing a protocol. • Inclusion and exclusion criteria and how to filter findings • Data extraction and quality assessment • Synthesising the findings.

4. What is a systematic review and what is its use? • A systematic review provides information on the effectiveness of interventions by identifying, appraising, and summarising the results of otherwise unmanageable quantities of primary research. • It uses a replicable, scientific and transparent approach which seeks to minimise bias. • Primarily it is used to collate data to produce a summary of the available findings of a given topic. • The approach is needed to inform policy and decision making and is useful in eliminating uncertainty regarding various interventions.

5. Rationale for undertaking SR • Whenever a systematic review is being considered, efforts should be made to ensure that a good quality review in the field of interest does not already exist. • If the available reviews are outdated or of poor quality, it may then become necessary to update existing reviews or conduct a new one. • In order to avoid duplicating research, it should also be established that a relevant review has not already been commissioned or is in progress.

6. Why review this type of research? • Useful for application of knowledge • Policy-makers and practitioners may want to know: - What people need - Whether interventions are acceptable and appropriate - Whether an intervention is effective - Why people do the things they do • Useful to generate knowledge and take stock of what is already known • Achieve an understanding beyond that gained in any individual study.

7. How to begin • Before beginning a systematic review it is important to ensure that the information regarding the subject is required, is there a need for this research? Searching the relevant databases, hand searching key journals and reading across the topic is essential before beginning to collate data. • Some key sources for published and ongoing reviews emerge from the Cochrane Library: • The Cochrane Database of Systematic Reviews (CDSR) • Database of Abstracts of Reviews of effectiveness (DARE) • Health Technology Assessment Database (HTA) • Other useful sources to check are: • HMIC (part of the Kings Fund Library) • MEDLINE • CINAHL • Science and Social Science Citation Index • PsychInfo • Searching the databases and internet resources should also be supplemented (if possible) with key contacts in the field who may be helpful in knowing whether any similar reviews are taking place.

8. Searching for data • The aim of the search is to generate as comprehensive a list as possible of primary studies, both published and unpublished, which may be suitable for answering the questions posed in the review. • A thorough, replicable and unbiased search strategy is crucial. • Constructing an effective combination of search terms (key words) for searching electronic databases requires a structured approach. • It is often helpful to produce a table of terms which can be broken up into population, interventions, outcomes and study design. • All searches should be documented as they develop and saved electronically. • The searches will gradually be filtered so the best available data on the chosen subject is extracted for inclusion in the review, once again this process must be documented. • Not all papers will be relevant for inclusion, in fact most will not be, however all the information retrieved must be saved.

9. Developing a protocol • The first milestone of any review is to develop a protocol. • The protocol specifies the plan which the review will follow to identify, appraise and collate evidence • The protocol should state precisely the main and secondary questions which will be addressed and note the inclusion and exclusion criteria of any potential papers. • Through undertaking some preliminary reading it is possible from the protocol to develop a rough hypothesis of the findings which will help to refine the research question further. Components of a protocol • Background • Review questions • Search strategy including search terms and resources to be searched • Study selection criteria and procedures • Study quality assessment checklists and procedures • Data extraction strategy • Synthesis of the extracted evidence • Project timetable

10. Inclusion and exclusion criteria • Both inclusion and exclusion criteria should follow logically from the review questions and they should be defined in terms of the population, the interventions, the outcomes, and the study designs of interest. Only studies that meet all of the inclusion criteria and none of the exclusion criteria should be included in the review. The review should be piloted to check that they can be reliably interpreted and that they classify the studies appropriately.

11. Data extraction (refer to handout) • To ensure validity the works filtered from the initial search which met the inclusion criteria are subsequently assessed by a minimum of two separate researchers. • A data extraction assessment form is required for this process. • The document will cover: • Study quality - the degree to which a study employs to minimise bias, a set of parameters in the design and conduct of a study that reflects the validity of the outcome. • Bias • Internal validity – the degree to which the results of a study are likely to approximate to the ‘truth’. • External validity – the extent to which the effects observed in a study are applicable outside of the study.

12. What is synthesis? ‘..the product of activity where some set of parts is combined or integrated into a whole.....[synthesis] involved some degree of conceptual innovation, or employment of concepts not found in the characterisation of the parts and a means of creating the whole’ Strike and Posner (1983) • A synthesis is not just a report of the findings of the individual studies in the review, it is a transformation of the data from the primary studies and it is the basis of the systematic review method.

13. Common elements of different methods of synthesis • ALL syntheses involved interpretation of concepts and data • ALL syntheses involve the comparison of settings, contexts and findings of studies • ALL syntheses involve decisions to be made regarding the relevance and trustworthiness of studies to be included

14. Extracting data for synthesis • What counts as the findings of a study? - Concepts • Themes • Theories • Quotes • Numbers • Author(s) interpretations • Where should you look? - Abstract - Results - Discussion - Conclusion

15. Practical 1 • In groups or pairs look through the synthesis exercise on homework evidence. • Using the information try to synthesise the studies transforming the data into a collective summary. • Remember that description is not transformation.

16. What to look for (Practical 1) Key concepts of data synthesis • Descriptive data of the evidence • The variability of difference (heterogeneity) between the studies in terms of key characteristics . • The degree to which the studies are similar (homogeneity) e.g. If the results vary no more than that expected by the play of chance. • The influence of bias and the idea that some studies which go against the norm may not be published.

17. Stages of synthesis

18. Part 2

19. Part 2 overview Will look at: • The intricacies of different methods of synthesis. • Short-hand approaches to urgent systematic information using REA • Go on to look at meta-ethnography as a review method • Look at the idea of first, second and third order concepts. • Interpreting data

20. Rapid Evidence Assessment • Rapid Evidence Assessment (also known as Rapid Appraisal). • Applies the principles of the systematic review in a truncated form. • Uses a very narrowly focused question, concentrates on only specific type of evidence using limited databases and gains less information from each study. • Does not involved such a detailed quality assessment. • A trade off between quality and speed.

21. Examples of qualitative synthesis methods

22. Example of first, second and third order constructs

23. Translating studies into one another

24. Summarising synthesis • Syntheses of people’s perspectives and experiences. • Epistemological reasons: • Crucial part of our knowledge and about the world • Help us consider health or social issues in the specific context of peoples lives • Ethical reason: • Important to listen to those who are the ‘targets’ of professional intervention.

25. Part 3 Theories of change and effectiveness reviews

26. Introducing Theories of Change (ToC) • A theories of change model can often be useful when looking at studies concerning effectiveness. It can inform both the review question and the inclusion criteria alongside aiding the interpretation of the reviews findings. • Built around the pathway of change, a Theory of Change describes the types of interventions (a single program or a comprehensive community initiative) that bring about the outcomes depicted in the pathway of a change map. Each outcome in the pathway of change is tied to an intervention, revealing the often complex web of activity that is required to bring about change. • A Theory of Change is a specific and measurable description of a social change initiative that forms the basis for strategic planning, on-going decision-making and evaluation.

28. Theories of change can change! • Probable intended theory • Theory of change after systematic review

29. Practical 2 • In pairs: • Read through the summaries of effectiveness studies • Develop a ‘theory of change’ that encompasses the ways in which the interventions are thought to impact on children’s healthy eating Combine with another pair and prepare to present your findings to the rest of the group.

30. Part 4 Summary of workshop 1

31. What kind of research is included? • Interviews (different degrees of structure) • Questionnaire-based surveys • Focus groups • Could be nested within an RCT • Numerical or statistical analysis • Thematic analysis

32. Some examples of review topics • Older people’s views of hospital discharge • Young people’s views on what impacts on their motivation to learn in the classroom • Lay experiences of diabetes and diabetes care • Experiences of being a teenage mother in the UK • Experiences of patients with coronary heart disease • Examining the effectiveness of new interventions

33. Workshop 2 (2nd March) Will look at: • the different ways of presenting an interventions effect using statistical measurements. • Binary outcomes • Odds ratio • Calculating relative risk • Standardised mean difference • Combining the quantitative and the qualitative.