1 / 32

Optimizing the F ormat for Presenting U pdates of Evidence R eviews: Seeking User Preferences

Optimizing the F ormat for Presenting U pdates of Evidence R eviews: Seeking User Preferences. Sydne Newberry, PhD Southern California Evidence-Based Practice Center September 11, 2012. Agenda . Introduction Examples of executive summaries we created Results of focus group testing 

quanda
Download Presentation

Optimizing the F ormat for Presenting U pdates of Evidence R eviews: Seeking User Preferences

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Optimizing the Format for Presenting Updates of Evidence Reviews: Seeking User Preferences Sydne Newberry, PhD Southern California Evidence-Based Practice Center September 11, 2012

  2. Agenda • Introduction • Examples of executive summaries we created • Results of focus group testing  • Conclusions • Next steps

  3. No one knows the optimal format for updates • Do users prefer to see the original and the updated findings side by side? • Numerical or narrative? • Do users care about seeing the original findingsat all? • Does the answer depend on the users? • And who are they??? • We tested reactions to a variety of formats among various categories of users

  4. The Goals for Optimizing the Formats for Update Reports • To achieve some degree of consistency across EPC CER updates • But be flexible enough to allow for different situations • e.g., the number and kind of new studies • No single format may be appropriate for every report • or every user…SO • Accessibility to the various stakeholders is key

  5. What we did… Created 5 test executive summaries Identified formats in use (and users) Chose a sample report to modify Summarized results Assessed reactions of a focus group (clinical decision makers) to the summaries online and via conference call Assessed reactions of 2nd focus group (research decision makers) to the summaries Created 2 new versions of the summaries and discarded 1 based on feedback Assessed reactions of 3rd group (community physicians) to the summaries

  6. We transformed an update report using the formats of 3 other update reports • A typical update report of a large consortium • An update report for a government agency • The EPC update review on acute otitis media itself • Executive summary only • Meta-analyses and quantitative results • An EPC update review on osteoporosis treatment • Executive summary • Qualitative findings only

  7. Health plan managers Other payers (private insurers, CMS) Clinical guideline developers (professional practice societies, USPSTF, …) Research planners Congress Clinicians at the point of care? EPC’s Eisenberg Center focuses on this group Who Uses Evidence Reports?

  8. What we did… Created 5 test executive summaries Identified formats in use (and users) Chose a sample report to modify Summarized results Assessed reactions of a focus group (clinical decision makers) to the summaries online and via conference call Assessed reactions of 2nd focus group (research decision makers) to the summaries Created 2 new versions of the summaries and discarded 1 based on feedback Assessed reactions of 3rd group (community physicians) to the summaries

  9. Version 1… • Shows new publication date • Shows new search end date • Lists new included studies, if any, in evidence table • Provides a table of changes to staff, key questions, methods at the end of the report • In other words, previous and updated pooled results are not explicitly compared

  10. Version 1 replaces old with new Overview The objective of this report was to analyze the evidence on the initial management of uncomplicated acute otitis media (AOM)… The Technical Expert Panel and project staff developed a literature search strategy. The initial strategy was developed for MEDLINE and was customized for other databases. Project staff searched MEDLINE (January 1998-July 2010), the Cochrane Library ( January 1998-July 2010),and the Web of Science . Additional articles were identified by review of reference lists in proceedings, published articles, reports, and guidelines.

  11. Focus Group 1 Included 7 Participants • Health plan pediatrician who serves on guidelines committees and has been on 2 TEPs • Academic family practitioner who has been on 2 TEPS • Insurance company research director, former EPC project leader • Academic physician/health services researcher/ guidelines developer • Health plan physician/manager and guideline developer • MS-level coordinator for guidelines committees • External affairs officer (former congressional staffer)

  12. Focus group 1’s reactions to Version 1 wereuniformly negative • BUT thought this format might be preferred by the average clinician • i.e., one who is less interested in what is old vs. new, and simply needs to know what is up-to-date • The health plan guidelines writer shared that his organization’s guidelines for practitioners are formatted exactly this way

  13. Version 2 “Highlights” Changes The objective of this report was to analyze the evidence on the initial management of uncomplicated acute otitis media (AOM) in children. AOM represents the most common childhood infection for which antibiotics are prescribed in the United States. Data from the National Ambulatory Medical Care Surveys (NAMCS), which did not differentiate between AOM and otitis media with effusion, indicated that the number of office visits for AOM increased more than two-fold from 1975 to 1990. Gates (1996a) believed that the majority of these cases represented AOM. A diagnosis of AOM requires 1) a history of acute onset of signs and symptoms, 2) the presence of middle ear effusion (MEE), and 3) signs and symptoms of middle-ear inflammation. (Marcy, Takata, Shekelle, et al., 2001). represents the most common childhood infection for which antibiotics are prescribed in the United States.

  14. Enthusiasm increased if the update report had identical key questions and only quantitative findings changed Several participants slightly preferred Version 2

  15. Version 3 Used Revision Marks to Show Changes to Text • In general we could not make definitive conclusions regarding differences in adverse event rates among antibiotics when taking into account a MCID of 5%. Adverse events were generally more frequent for amoxicillin-clavulanate than for cefdinir, ceftriaxone, or azithromycin. • Meta-analysis demonstrated that children treated with 7 to 10 day amoxicillin-clavulanate had an12.918 percent (95 percent confidence intervals,4.58 percent and 21.228 percent) greater rate of gastrointestinal adverse effects than children treated with 5-day azithromycin (although not reported in the studies, the clavulanate concentration was most likely 31.25 mg per 125 mg of amoxicillin, i.e., original formulation.)

  16. No one in Focus Group 1liked this version!!!

  17. Version 4 represented entirely new text but featured comparison tables

  18. Version 5 Features a Narrative Summary Table with Bolding, and Shading Some SRs have only narrative findings

  19. Focus group 1 participants infinitely preferred Versions 4 and 5!! • They shared that • Version 4 would be preferable for a report with quantitative findings (i.e., MA and pooled effect sizes), whereas • Version 5 would be preferable for a report with qualitative findings • Tables like these might even make version 1 more useful to a broad range of users…

  20. More Suggestions to Emphasize Important Changes in an Update Report • Show updated findings in context of old, • most readers won’t have the earlier report on hand • compare results in text • Include both qualitative AND quantitative tables • or at least include strength of evidence in table • Show changes to key findings in a bulleted list • For decisionmakers, provide the figures and tables with the actual data in addition to the summary tables • One of our focus group participants shared the format used by his health plan… Based on the feedback, we tossed out Version 3, combined Versions 4 and 5, and created a new version

  21. What we did… Created 5 test executive summaries Identified formats in use Chose a sample report to modify Summarized results Assessed reactions of a focus group (clinical decision makers) to the summaries online and via conference call Assessed reactions of 2nd focus group (research decision makers) to the summaries Created 2 new versions of the summaries and discarded 1 based on feedback Assessed reactions of 3rd group (community physicians) to the summaries

  22. The new Version 3 simply included both quantitative and narrative tables…

  23. The new version 4 contained no narrative text Table S-1. Scope of the Report and Definitions • Results: • Quantitative tables • Narrative tables Table S-2. Search Criteria • Conclusions • Limitations • Future Research Needs

  24. What we did… Created 5 test executive summaries Identified formats in use (and users) Chose a sample report to modify Summarized results Assessed reactions of a focus group (clinical decision makers) to the summaries online and via conference call Assessed reactions of 2nd focus group (research decision makers) to the summaries Created 2 new versions of the summaries and discarded 1 based on feedback Assessed reactions of 3rd group (community physicians) to the summaries

  25. Focus Group 2 Included 5 Participants • A director of an office at FDA • A director of a federally funded academic research center • An FDA microbiologist • An administrator at CMS • A former congressional staffer with health policy experience

  26. Focus group 2 participants preferred versions 3 and 4 • Versions 1 and 2 were not helpful • Want to see what changed and why • New version 3 (both sets of tables) was better, but • show how the new findings contributed to changing the conclusions; • use color coding or different fonts to increase the salience of what changed; and • include some wording that would put the changes in context. • Show whyconclusions changed • New version 4 was even better but not a substitute for full report

  27. What we did… Created 5 test executive summaries Identified formats in use (and users) Chose a sample report to modify Summarized results Assessed reactions of a focus group (clinical decision makers) to the summaries online and via conference call Assessed reactions of 2nd focus group (research decision makers) to the summaries Created 2 new versions of the summaries and discarded 1 based on feedback Assessed reactions of 3rd group (community physicians) to the summaries

  28. Community physicians… • May be the busiest of all groups we included! • Disliked all but version 4! • Preferred graphic display and lack of narrative text

  29. Tested only 1 report Included only the executive summary, not the full report, but Equivalent to format used by large HMO Original report included (almost) only quantitative results Small number of focus groups and participants Possible sample bias Our study had a number of limitations…

  30. Conclusions • Update reports need to show at least what changed • A stand-alone executive summary without narrative text may suffice for point-of-care clinicians • Policy makers need full reports • But stand-alone summary serves as useful map to full report • Optimum method of presenting conclusions depends on whether meta-analytic or narrative…

  31. Next Steps • We have submitted draft report to AHRQ • Followup testing with more reports and focus groups would be helpful • We propose submitting an update report with a “Version 4” style executive summary for peer review and public comment

More Related