Issues and challenges to scoping and focusing the question

1. Issues and challenges to scoping and focusing the question ESQUIRE Qualitative Systematic Review Workshop University of Sheffield 8 September 2011 Janet Harris Cochrane Qualitative Research Methods Group

2. Session aims • To identify the issues and challenges in formulating different types of qualitative questions for qualitative evidence synthesis

3. The challenges Utility Relevance Timeliness Explanatory value The issue: What question?

4. How effectiveness questions are developed Clinical expertise + research evidence are used to: Refine definition of problem Identify problem Test solutions Identify possible causes and effects Margerunm-Leys http://www-personal.umich.edu/~jmargeru/conceptmap/types.htm

5. Effectiveness question: In adults with Low Back Pain does provision of individual education reduce pain, improve functioning and promote return to work? P: Adults with back pain (BP) Assumed solution: Patients need written and/or verbal education delivered by a health professional Problem and solution (as defined by the researchers) I: Information on how to self-manage BP O: Reduce pain, improve functioning, return to work

6. Possible results from a search using an informing question Single studies / evidence syntheses identifying factors that might make us rethink assumptions Problem Definition(s) Informing question (Background question) Causes Effects

7. Explore assumptions with informing questions P: Important characteristics relating to I and O Refine P Consider subgroup analysis Informing question (Background question) I: Factors that affect implementation Consistency of implementation O: Factors that may dilute potential effectiveness Such as participants’ views

8. Back pain patients’ experiences of receiving education for self management

9. Informing questions may identify theories • Theories are explanations for a phenomenon or an event e.g. Why something happens in relation to another event • They can be grand theories, they can be middle range theories, or they can be single-case theories • Middle range theory goes beyond individual studies, providing a cross cutting explanation for patterns that are seen across settings and different types of people • Whenever possible, interventions testing effectiveness should be based on theory

10. Issues raised from informing questions • There may be ‘essential ingredients’ that need to be included in interventions • For example, the search on back pain education identified an issue with appropriateness and relevance of information • Informing questions need to produce convincing evidence about these ingredients to justify changing an effectiveness question. • How much evidence is needed to change the original effectiveness question?

11. In adults with Back Pain does provision of individual education by a health professional reduce pain, improve functioning and promote return to work? Revised effectiveness question: P = adults with Low Back Pain I = information explained by health professional C= information explained by a health professional with additional peer support O = reduce pain, improve functioning and promote return to work Additional ingredients that theoretically increase relevance Revising the effectiveness question: An example

12. Issues with the refined question • Is the quality of reporting for effectiveness studies adequate? • For example, do the interventions describe how information is explained by the health professionals? • Is the amount of time for the explanation documented? • Does the comparison describe the type of peer support offered? • Qualitative evidence can indicate how a question should be refined, but if effectiveness studies do not report the relevant information then the end product could be an empty review

13. Enhancing questions to explain the results of effectiveness reviews • Enhancing question • Within-study data • Parallel-study data • Similar-study data • Monitors pathway from intervention to outcomes • May highlight problems with Quan studies

14. Adequate data within the study about PICO characteristics? Adequate data in embedded qualitative studies? Adequate data in parallel sibling studies? Can ‘similar’ qualitative studies be found? What does ‘similar’ mean – across embedded, sibling, and independently conducted studies? Comparability of data - how to synthesize data across studies? Enhancing questions: Issues and challenges

15. Extending question: Can explore an issue related to population, intervention, comparison or outcome Problem noted: Lack of theory-based education on how to self-manage BP Extending question: How do people make sense of and use Back Pain information? What educational theories promote better use of BP information? Utility of findings: Used to critique design and PICO in included studies; subgroup analysis of theory-based interventions.. Cause: Low awareness of educational approaches for behaviour change Negative effect: Increased utilization of services and absenteeism

16. Extending questions require a separate qualitative evidence synthesis, either conducted within or alongside an effectiveness review. Funding needs to be justified by considering: Timeliness: Is the information perceived to be critical to using the results of the effectiveness review? Relevance: Is the issue important to funders, policymakers and effectiveness reviewers ? Utility: did the scoping process specifically note issues that need to be informed by qualitative research? Explanatory value: Willthe qualitative synthesis contribute to explaining the outcomes of effectiveness studies and/or the broader context that influences effectiveness? Extending questions: Issues and challenges

17. Extending questions require coordination of a team including qualitative evidence synthesis (QES) members working alongside effectiveness review (ER) members The parallel processes need to be considered because Findings from the QES may identify quality issues in the effectiveness studies. At what point are these quality issues brought into the effectiveness review – at abstract review? Development of inclusion criteria? Critical appraisal? Metaanalysis? Reporting QES process and findings: Approaches are still being developed. At the same time, the software for Cochrane ERs – RevMan – does not yet have the facility for including Extending reviews. Extending questions: Issues and challenges

18. Supplementing questions Effectiveness review Supplementing review: Conducted independently Before or after an ER On a wider issue than effectiveness Effectiveness review

19. Supplementing questions: Used in stand alone reviews to produce information on a wider issue Does peer support help people with chronic conditions? Does peer support promote health literacy (help people to understand and use health information) Issues Timeliness: Will the theory be used over time? Relevance: Does the theory address issues noted by ER authors and those trying to implement review results (External validity) Utility: will the theory enable development of better effectiveness studies? Explanatory value: Can the theory be used to explain some of the differences in effect within and across existing ERs? Can the theory be used to inform development of future ERs? Supplementing question issues

20. Summary: What guides qualitative question formulation • The need to ensure that effectiveness questions are informed by existing theory on what works (Informing questions) • The need to provide real-time explanations for effectiveness studies (Enhancing questions) • The need to further explore issues identified in the review by reviewing a broader range of research on effectiveness (Extending question) • The need to explore the context surrounding a condition or an intervention, including situations and experiences that may not be directly related to effectiveness (Supplementing question)

21. Is it worth it? • When scoping and focusing the question, we need to ask: Does this question pass the TRUE test? • Timely: Will the question still be relevant when the review is finally completed? • Relevant: How will this question contribute information that is relevant to health services policymakers? • Useful: Which of (the many) findings from the qualitative scoping will be useful in focusing the review and/or explaining the results? • Explanatory value: Does the synthesis contribute to explaining the outcomes of effectiveness studies and/or the broader context that influences effectiveness?

22. Further challenges • Team skills • Team coordination • Informed and transparent decisions about when and how to useQES/ER data