AcademyHealth Annual Research Meeting 27 June 2006

1. Translating Emerging Evidence:Lessons from the MATCH Study(Methods for Developing Actionable Evidence for Consumers of Health Services Research) AcademyHealth Annual Research Meeting 27 June 2006

2. Research Team John Hsu, MD, MBA, MSCE Thomas Rundall, PhD Mark Gibson Pam Curtis, MS Laura Arroyo Ilana Graetz Estee Neuwirth, PhD Julie Schmittdiel, PhD Peter Martelli, PhD Candidate Rodney McCurdy, MHA, PhD Candidate Kaiser Permanente Division of Research, Institute for Health Policy, and Care Management Institute University of California, Berkeley, Center for Health Research Oregon Health Services University, Center for Evidence Based Policy Funding Source: Agency for Healthcare Research and Quality

3. Background • Despite the potential value of using research evidence for organizational decisions, decision-makers rarely use evidence • Information needed on the reasons for the gap between research generation and use

4. Study Overview • Objectives: To identify methods for closing the gap between the creation and use of evidence • Determine the evidence needs of consumers of health services research and effective methods for communicating evidence • Develop tools to help organizational decision-makers

5. Study Methods • Qualitative study • Dynamic, semi-structured focus groups • Four focus groups with 31 senior health care managers from a range of organizations (private and public) and regions (West, Midwest, East) • Full day sessions with moderator • Discussion focus: • High-priority strategic decisions • Current levels of research evidence use • Defined characteristics of useful evidence • Organizational barriers & facilitators of using evidence • Suggestions for improving the use of evidence • Content analysis used to synthesize responses • Codes • Themes

6. Limited Use of Evidence • Use of research evidence is limited • Evidence does not meet decision-makers’ needs, e.g. no actionable findings or discussion of applicable situations “For the vast majority of questions I have, I have found the answers are not out there. Research is not being done in a way that answers my questions.” “What people really want is not the evidence, they want the answer.” • Range of organizational barriers to using evidence cited: • Culture (values, beliefs, priorities, incentives) • Resources (money, time, technology, data) • Capacity (lack of experience and expertise for assessing evidence)

7. Definitions of Evidence • Broad definition of evidence: • Colloquial, experience-based, and tacit knowledge • Pilot and case studies, and qualitative assessments of operational or strategic decisions by experts • Internal measurements • Traditional scientific evidence • Criteria for useful evidence: • Accurate • Applicable • Actionable • Accessible • Perceptions that research evidence often not useful • Not the right question for a decision • Different set of standards, e.g. bias vs. certainty, timeliness • Not applicable to the organizational context and environment • Not actionable findings • Limited accessibility • Disconnect between definitions and evaluation of evidence

8. Recommendations • Presentation of evidence addresses four criteria: • Accuracy (e.g., establishes causal relationship, credible sources) • Applicability (e.g., relevant to management decision, setting and situation) • Actionable items (e.g., information on what needs to be done and implications, fits into time frame) • Accessibility (e.g., easy to obtain - “at our fingertips”) • Characteristics of effective communication: • Emphasizes main points (4 A’s) • Uses multiple dissemination approaches, e.g. websites and conferences • Linked to a trusted information source, e.g. a clearinghouse of vetted evidence

9. Decision-maker Comments “One of the things we struggle with is having concise information. [Something that is] not 300 pages. Formatting too. Ease of use is really important.” “There’s an issue of transparency [for research and knowledge brokers, e.g. consultants] - both in the evidence and in the way it is gathered. Most people don’t have the resources to do this on their own.” “We want a deeper understanding of the sources of evidence and vetting that evidence. Management research is more empirical. Can we get evidence in a timeframe that is actionable?” “If you don’t make evidence available, easily available, in the face of the person, at the moment that they need it, they don’t get it. They’re too rushed.”