A worked example of META-AGGREGATION

1. A worked example of META-AGGREGATION Its history Its philosophy A worked example

2. Meta-aggregation • A structured, and process driven approach to systematic review drawing on the classical understandings and methods associated with systematic review of quantitative literature as practiced by the Cochrane and Campbell Collaboration • Based on an a-priori protocol • Established, answerable question • Explicit criteria for inclusion • Documented review methods for searching, appraisal, extraction and synthesis of data

4. Meta-aggregation: History • Participative consensus project involving Australian academics • The outcome = an aggregative approach to the synthesis of qualitative evidence that: • emphasised the complexity of interpretive and critical understandings of phenomena; recognised the need to ensure that the process is practical and usable; • balanced utility of the outcomes with the complexity of the material; • was grounded in pragmatism.

5. Meta-aggregation: A pragmatic approach • The philosophy of pragmatism - Charles Peirce, William James, John Dewey • the value of any thought lies in its practical use and consequences. It is the focus on practical consequences that characterizes meta-aggregation as a synthesis approach • Central to pragmatism were communities of inquiry or groups of people with a shared interest, problem or issue trying to resolve doubt through critical reasoning. • Actions are assessed in the light of their practical consequences. Aggregation concentrates on the original researchers’ findings (processed data) and summarizes common and competing findings to produce cross-study generalizations that lead to recommendations for action  The utility of an outcome is important. Research articles

6. Meta-aggregation: a worked example • Review Topic: Obstacles to the implementation of Evidence-Based Practice in Belgium: a context-specific study • Question formulation: • Setting: Belgium (Belgian health care system) • Population: Health Care Practitioners • (Phenomena of) Interest: Obstacles toward the implementation of Evidence-Based Practice • Comparison: (additional literature search on strategies developed by countries with other systems) • Evaluation: Experiences and perceptions

7. Meta-aggregation: a worked example • Search strategy: • Major databases: Medline, CINAHL, Psychinfo, Embase, Social Sciences abstracts and ERIC (1990-May 2008). • Where possible we used a methodological filter for qualitative research. • Other information sources: • ‘Federal research actions’-database from the Belgian governmental department of science • Consultation of Belgian experts in qualitative research methods and/or Evidence-Based Practice to check on any other published material that could be of use for the synthesis. • Screening of references • Keywords: • ‘Evidence-Based Practice (Medical Subject Heading (MeSH) term)’, including Evidence-Based Dentistry, Nursing and Medicine. • Additional keywords: evidence-based combined with the geographical notion ‘Belgian’, ‘Flemish/Flanders’ or ‘Walloon’.

8. Meta-aggregation: a worked example • Inclusion criteria: • Study type: Qualitative, empirical research papers. Opinion pieces and descriptive articles were excluded. • Study participants: All types of health care practitioners e.g. physicians, dentists, nurses, physiotherapists, psychologists • Topic of interest:Obstacles to the implementation of EBP. • Context: the Belgian health care system. • Outcome of interest: Experiences and/or perceptions from participants. 8 studies met the inclusion criteria

9. Meta-aggregation: a worked example • Critical appraisal of selected studies: • No criteria based quality appraisal, although the software QARI provides a standard critical appraisal checklist. • limited amount of studies found • the majority of the selected studies were written by the lead reviewer, who would also be one of the appraisers. • An overall judgment approach was used instead. This approach has been proven to deliver the same outcome (Dixon-Woods, 2007). However, it tends to be less explicit about potential reasons for exclusion. • An evaluation based on the JBI-critical appraisal instrument is recommended for other review teams opting for a meta-aggregative approach to synthesis.

10. Meta-aggregation: a worked example • 85 different findings were found • The findings were classified in 9 major categories: • (1) evidence is hard to implement • (2) decision making processes are influenced by practitioner variables • (3) decision-making processes are influenced by practitioner variables • (4) commercial/financial interests affect EBP • (5) governmental regulations influence the process of implementation, • (6) EBP is more feasible for practitioners working in a scientific setting compared to those working in other settings • (7) physicians’ display of power hampers other professionals to put evidence-based practice into effect • (8) a lack of knowledge and skills hinders the implementation of evidence-based practice • (9) attitudes of health care practitioners hinder the implementation of EBP.

11. Meta-aggregation: a worked example • These categories were further analysed to produce 4 synthesized statements • Synthesis 1: Evidence might have a limited role in decision-making processes in daily practice, if the importance of the scientific component is not stressed (categories 1, 2 and 3). • Synthesis 2: Aspects other than quality of care will steer the EBP agenda, if governmental regulations and economic interests are contra-productive for delivering the best possible care (categories 4 and 5). • Synthesis 3: Although EBP is intended to serve all practitioners, some health care providers will benefit less from EBP than others, if inequity issues between practitioners are not solved and support for field workers is not established (categories 6 and 7). • Synthesis 4: A lack of competences will hinder the implementation of EBP, if gaps in knowledge and skills are not being filled and efforts to change contra-productive attitudes are not undertaken (categories 8 and 9).

12. Meta-aggregation: a workedexample

13. Meta-aggregation: a workedexample • Implications for practice and policy • Consider educating patients and exploring potential information channels that influence patients’ opinions to create space for well-informed decisions (synthesis 1). • Provide easy and free access to well-structured, compact and relevant information targeted to a particular discipline and consider helpdesks. Screen information, control its quality and translate it to the field (synthesis 1). • Consider updating the Belgian nomenclature and reimbursement system to bring it in line with the latest evidence (synthesis 2). • Consider incentives for those practitioners who are engaging themselves for the implementation of EBP in daily practice, to keep them motivated and to prevent from creating a negative spiral in which practitioners tend to believe that efforts to improve practice automatically lead to a loss of income (synthesis 2).

14. Meta-aggregation: a worked example • Implications for practice and policy • Enhance communication and cooperation between physicians and other partners in health care, via their professional groups and journals (synthesis 3) • Consider programs tailored to the needs of specific disciplines and stimulate multidisciplinary education should be stimulated to create mutual understanding across disciplines (synthesis 3, 4). • Consider direct access to allied health services to increase autonomy (synthesis 3) • Integrate EBP in the basic curricula (synthesis 4).

15. The Qualitative Assessment and Review Instrument (QARI) Designed to provide a systematic process mirroring that taken for systematic reviews of quantitative research, whilst being sensitive to the nature of qualitative data.

20. Data Extraction Reduce The findings of many studies into a single document Summarise Methods Phenomena Findings

22. The units of extraction in this process are specific findings and illustrations from the text that demonstrate the origins of the findings. In QARI a finding is defined as: A conclusion reached by the researcher(s) and often presented as themes or metaphors.

23. Levels of credibility Unequivocal - relates to evidence beyond reasonable doubt which may include findings that are matter of fact, directly reported/observed and not open to challenge Credible - those that are, albeit interpretations, plausible in light of data and theoretical framework. They can be logically inferred from the data. Because the findings are interpretive they can be challenged. Not Supported - when 1 nor 2 apply and when most notably findings are not supported by the data

24. Meta-Synthesis The aim of meta-synthesis is to assemble findings; categorise these findings into groups on the basis of similarity in meaning; and to aggregate these to generate a set of statements that adequately represent that aggregation. These statements are referred to as synthesised findings - and they can be used as a basis for evidence based practice.

30. In QARI, a synthesised finding is defined as an overarching description of a group of categorised findings that allow for the generation of recommendations for practice.

31. Can be stated propositionally as “if-then” statements - for example: “If students are advised reconsider their choice of study , their relatives will sometimes feel as if they are not involved”. (a somewhat awkward and eccentric form) the declamatory form that emphasizes the probability of the claim: “Relatives of students that are advised to reconsider their choice of study may feel as if they are not involved if strategies to include them are not pursued”.

33. Should look familiar!