Evidence and Organizational Decision Making: Never the Twain Shall Meet?

1. Evidence and Organizational Decision Making:Never the Twain Shall Meet? Thomas Rundall, PhD Academy Health, Annual Research Meeting June 27, 2006

2. Research Team • John Hsu, MD, MBA, MSCE • Estee Neuwirth, PhD • Julie Schmittdiel, PhD • Laura Arroyo • Ilana Graetz • Kaiser Permanente Division of Research and Institute for Health Policy • Kaiser Permanente Care Management Institute • Thomas Rundall, PhD • Peter Martelli, PhD Candidate • Rodney McCurdy, MHA, PhD Candidate • U. C. Berkeley, Center for Health Research • Mark Gibson • Pam Curtis, MS • Oregon Health Sciences University, Center for Evidence Based Policy • Financial Support: Agency for Healthcare Research and Quality

3. Introduction • There is a substantial gap between health services decision makers and the research community • Studies of health organization decision makers in Canada, the United Kingdom, and the United States report that decision makers rarely use research evidence • The research community needs to understand the reasons for this “gap” and work with managers and policy makers to close it • Source: Kovner and Rundall, Evidence-based Management Reconsidered. Frontiers of Health Services Management, Spring 2006.

4. Rationale for the Study • Using research evidence when making decisions about the organization, financing and delivery of health care has great appeal • Decision makers, however, do not appear to use this information • We queried health organization decision makers about their definition and use of evidence when making strategic decisions • The findings of this study will help inform the construction of an evidence-based management toolbox

5. Study Design • Qualitative • Four peer-to-peer discussion groups (full day) • Questions designed to elicit perceptions of • High-priority decisions • Current levels of research evidence use • Characteristics and definitions of useful evidence • Barriers and facilitators to using evidence • Recommended methods for communicating evidence

6. Methods • Focus group discussions were held in San Francisco, Oakland, Chicago, and New York • Purposive sampling of health care senior managers • 26 organizations of various types • Private provider groups • Health insurance plans and insurers • Public health systems • Employers and employer groups • Regulatory agencies • Organizations with limited resources and wealthier ones • 5-10 managers in each focus group (31 total)

7. Methods • Moderator posed questions and recorded major themes • Two researchers recorded all comments and compared notes with meeting audiotapes • Using content analysis techniques, two researchers independently coded the text • Initial 72.1% agreement on codes; discrepant codes resolved • Investigators categorized codes into themes • Coded themes compared with moderators’ themes

8. Results • Broad list of high-priority decisions • Purchasing to improve quality and reduce costs • Organization of patient care • Organizational sustainability • Benefit design and coverage • Workforce recruitment development, retention • Limited use of research evidence in decision making • Difficulty distinguishing between evidence-based medicine and evidence-based management

9. Results • Broad definition of acceptable evidence • Internal data/measurements • Colloquial evidence • Externally dictated standards • Scientific evidence • Useful evidence must be • Accurate • Applicable • Actionable • Accessible • Research (scientific) evidence often not useful • Disconnect between definitions and evaluation of evidence • Lack of experience, resources, and incentives for assessing evidence • Desire for tools and mechanisms for validating evidence

10. Results • Tools need to be concise and focused • Need for processes to help decision makers appraise evidence • Desire for evidence that is “vetted”

11. Conclusions • There is a substantial gap between health services decision makers and the research community • Decision makers rarely report using research evidence and instead rely on other types of information such as peers with experience. • Decision makers perceive that existing research does not meet their needs, both in content and accessibility.

12. Next Steps? • Build an evidence-based decision making tool box for managers • Continue to foster collaborative research between health services researchers and managers (e.g. The work of the Center for Health Management Research) • Encourage national organizations to work with appropriate information technology companies and experts to improve access to relevant research • Training and re-training of health services researchers to improve their ability to conduct and communicate research that will be used by health services managers and policymakers

13. Next Steps? • Work with health organizations to build cultures, processes, and structures that support the use of research evidence in decision making • Recognize the prevalence of sub-optimal management decisions • Conceptualize, measure, and assess managerial decision making • How about a new private/public partnership program: TRIM (Translating Research into Management)