1 / 29

What is a Rapid Evidence Assessment? What’s involved?

What is a Rapid Evidence Assessment? What’s involved?. Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information Unit. Evidence reviews: The what, why and how… Evidence reviews in practice: Examples of recent Rapid Evidence Assessments.

lloyd
Download Presentation

What is a Rapid Evidence Assessment? What’s involved?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information Unit

  2. Evidence reviews: The what, why and how… Evidence reviews in practice: Examples of recent Rapid Evidence Assessments This session

  3. Why review evidence? “The sheer amount of potential research evidence in most substantive areas of social science and public policy… make[s] it almost impossible to keep abreast of the research literature in any one area”. Davies, 2003 Evidence reviews:the what, why and how…

  4. Methods for reviewing evidence…

  5. Literature reviews Look! The breadcrumbs lead here, this MUST be the answer!

  6. Literature reviews AHAHAHAHA! I have tricked you into only reviewing only *some* of the available evidence!

  7. Are we happy for professionals to only have some of the evidence when making decisions?

  8. Antman et al, 1992. Study comparing recommendations for treating heart attacks based on literature reviews with recommendations based on a systematic meta-analysis. We need to look at all the evidence… • Literature reviews often failed to mention important advances or exhibited delays in recommending effective preventive measures. • In some cases, treatments that have no effect on mortality or are potentially harmful continued to be recommended by several clinical experts.

  9. Systematic Reviews

  10. Systematic reviews Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police. Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.

  11. Of those that had heard of them, some had concerns: Timeliness Relevance Usefulness Some had difficulty distinguishing them from literature reviews, even when explained Campbell S et al (2007) Analysis for policy: evidence-based policy in practice Policy opinions of systematic reviews

  12. Use systematic principles and the same process as a systematic review… …but make compromises given available time and resources Pragmatic and transparent approach Rapid Evidence Assessments

  13. Draft search terms Draft sift criteria Sift received abstracts Request relevant papers Read and ‘grade’ papers Write it up (‘synthesis’) The process (in a nutshell)

  14. Demonstrate consistency in searching/ sifting Document search and sift process Process should be transparent and repeatable Specify required quality of evidence Systematic reviews only? Pre-post studies only? All ‘empirical’ papers? Be explicit/ transparent about the limitations of the approach Some key principles

  15. Some examplesWhat it is like in practice?

  16. Review of Police Leadership and Training commissioned by the Home Secretary CC Peter Neyroud needed the best evidence he could get on “What works in training and behaviour change?” Three weeks to deliver an evidence review Not a full REA, but… More than literature or scoping reviews Used systematic principles Due to time limit, search restricted to evaluations and systematic reviews only One practical example

  17. Search terms training OR learning OR development AND evaluat* OR assess* OR what works OR impact AND systematic review Limitations Searched 11 databases and 2 websites English language only Systematic review Training Evaluation Searching (Training) Initial search identified 1,015 abstracts to sift

  18. Sifting (Training) • Q1: Is the study about adult training, learning, or development? • Q2: Is the study: An evaluation (at least pre & post level) OR a systematic review? • Secondary sift to remove papers related to inappropriate populations and specific medical conditions 38 32 22 10

  19. Read, Appraise and Synthesise papers – What works in training?

  20. Time Availability of papers (10/22) Unable to pilot search terms Available databases Only those available to the NPL English language only Literature focussed in different areas Almost nothing on policing Papers mostly from healthcare Limitations

  21. Forces are adopting business improvement techniques to examine current practices and explore scope to change processes to release savings Techniques include QUEST, CI, Lean, Six Sigma, Kaizen, etc. Organisational change and business improvement • So – are these techniques the answer? • Is that magic potion? Or is it snake oil?

  22. Rapid Evidence Assessment Step 1 Systematic search (11,960 abstracts) Step 2 Quality assess & critically appraise (181 empirical studies) Step 3 Synthesise findings (41 studies with useful findings) Draw conclusions Example of the two searches in the organisational change and business improvement REA

  23. Organisational change and business improvement Total of approx 12,000 abstracts; 181 full papers were requested; 41 were included… • Why so many papers? • Better search terms needed? • Dodgy descriptions of papers in abstracts? • Problems with searching?

  24. Organisational change – what works? Potential success factors for organisational change

  25. Organisational change: potential success factors Leadership stability of supervision throughout implementation direct support from supervisors - ‘on-the job’ training staff involvement in decisions transformational leadership behaviour = reduced employee cynicism Engagement staff active participation in decision making & ‘room to experiment’ degree to which staff understood rationale for change communication found to influence self reports of job performance

  26. Piloting your searches is critical (we’ll do this tomorrow) to give you an idea of the size of the job, and if there is much available material It’s important to focus on the end result and how will the findings be used – make it relevant… Recognise (and accept) there are limitations of the approach Make sure you are researching the right question Follow the key principles: Demonstrate consistency in searching/ sifting Document search and sift process Specify required quality of evidence Be explicit about the limitations of the approach REAs should be replicable Lessons learned...

  27. The process (for Evidence Base Camp)

  28. Search terms are hugely important. • Tomorrow is all about developing search terms and seeing the impact of using different terms. • We will be running live pilot searches in your groups with College librarians.

  29. Civil service REA toolkit: http://www.civilservice.gov.uk/networks/gsr/resources-and-guidance/rapid-evidence-assessment Campbell Collaboration (social interventions, e.g. crime and justice) http://www.campbellcollaboration.org/ Cochrane Collaboration (medical interventions) http://www.cochrane.org/ EPPI centre (education) http://eppi.ioe.ac.uk/cms/ Useful links

More Related