A Researcher’s Perspective on Quality Improvement Research

1. A Researcher’s Perspective on Quality Improvement Research Neil R. Powe, MD, MPH, MBA Professor of Medicine, Epidemiology and Health Policy and Management Director, Welch Center for Prevention, Epidemiology and Clinical Research Johns Hopkins School of Medicine and Bloomberg School of Public Health

2. Outline • Research and quality improvement • How did we get ourselves into this conundrum? • It is not new and it is not just a problem for healthcare • Are all QI efforts created equal? • What is the evidence that quality improvement research does harm? • Options and Solutions

3. Definitions • Research: A systematic investigation including research development, testing and evaluation designed to develop or contribute to generalizable knowledge (45 CFR 46.102) • Quality improvement: a systematic pattern of actions that is constantly optimizing productivity, communication and value within an organization in order to achieve the aim of measuring the attributes, properties, and characteristics of a product/service in the context of the expectations and needs of customers and user of that product. (Institute of Medicine)

4. Research versus Quality Improvement Modified from Cambridge Health Alliance

5. The Overlap is the Issue Quality Improvement Research

6. How did we got ourselves into this conundrum?

7. Call for Science of QI Statement on Health Care Research and Quality ImprovementJohn Eisenberg, M.D., Administrator, AHCPR. President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry, Quality Improvement Environment Subcommittee, Nov 18, 1997, Washington, D.C. “The improvement in health care quality will depend not only on traditional methods like excellence in education, continuing education, professional standards and principled practice behavior, as important as they are. It will also depend on our developing the science of clinical practice, on our taking a fresh look at ways of measuring and improving care, and on our holding both the practice of medicine and the practice of quality improvement to the same levels of scrutiny and evidence as we hold technical clinical innovations.”

8. Rigor in QI Methods Eccles M, Grimshaw J, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of change and improvement strategies Qual Saf Health Care 2003;12:47-52 “The methods of evaluating change and improvement strategies are not well described. The general principle under-lying the choice of evaluative design is, however, simple—those conducting such evaluations should use the most robust design possible to minimize bias and maximize generalizability.” Institute of Medicine. The Learning Healthcare System: Workshop Summary. Washington, DC: 2007 A roundtable compendium encouraging movement toward increasingly sophisticated research designs other than RCTs to assess quality improvement or patient-safety initiatives. Speroff T, O’Connor GT. Study designs for PDSA quality improvement research. Qual Manag Health Care.2004;13:17-32. “Improving the rigor of the quality improvement literature will build a stronger foundation and more convincing justification for the study and practice of quality improvement in health care.”

9. Guidelines for QI Appraisal Speroff T, James BC, Nelson EC, Headrick LA, Brommels. Guidelines for Appraisal and Publication of PDSA Quality Improvement. Quality Management in Health Care. 2004. 13(1):33-39. “Users of quality improvement could benefit with markers to gauge the "best" science. …. Analogous to existing research guidelines, the PDSA quality improvement guidelines will provide researchers and reviewers with succinct standards of methodological rigor to assist in critical appraisal of quality improvement protocols and publications.” Books on the Science on QI: Grol R, Baker R, Moss R. editors. Quality Improvement Research: Understanding the science of change in health care. Blackwell BMJ Books

10. It is not new and it is not just a problem for healthcare

11. Action Research • Kurt Lewin then a professor at MIT, first coined the term “action research” in 1944 • “Action Research and Minority Problems” 1946. • Paper described “a comparative research on the conditions and effects of various forms of social action and research leading to social action” that uses “a spiral of steps, each of which is composed of a circle of planning, action, and fact-finding about the result of the action”.

12. Action Research & QI Similarities Action research • Reflective process of progressive problem solving led by individuals working with others in teams or as part of a "community of practice" to improve the way they address issues and solve problems. • Can be undertaken by larger organizations or institutions, assisted or guided by professional researchers, with the aim of improving their strategies, practices, and knowledge of the environments within which they practice. Quality improvement • Team effort of identifying opportunities for improvements, measuring performance, and involving the frontline providers and staff members to find ways to improve performance. • Goes through repetitive cycles of measuring performance, testing change concepts and then re-evaluation of outcomes measures

13. Action Research Operates Occurs at Many Levels • 1st person research: my research on my own action aimed primarily at personal change • 2nd person research: our research on our group (family/team) aimed primarily at improving the group • 3rd person research: scholarly’ research aimed primarily at theoretical generalization and/or large scale change

14. Action Research QI

15. Action Research Goes on in Many Places • Medicine • Quality improvement • Public Health • Public health practice • Community-based participatory research • Education • Primary, secondary and higher education

16. All QI efforts are not created equal • Tension exists between those that are more driven by the researcher’s agenda to those more driven by participants • For QI researcher, the Study in PDSA is paramount • For the QI doer, the Do and Act in PDSA are paramount

17. What is the Evidence That Quality Improvement Research Does Harm? • We do not really know! • Possible threats • Pressure of cost containment • Misapplication of an intervention to different patient populations • Unintended consequences • How much risk is involved?

18. The Risk Escalation Principle of Protection As the risk of the research rises about the threshold of “minimal risk” there are restrictions and additional protections, particularly for vulnerable subjects

19. Level of Risk Determines Level of Review by Institutional Review Boards Full IRB Committee Review Expedited Review Minimal Risk Exempt from Review

20. Definition of Minimal Risk in Subpart A of Code of Federal Regulations • Minimal risk means that the probability and magnitude of harm or discomfort anticipated in the research are not greater in and of themselves than those ordinarily encountered in daily life* or during the performance of routine physical or psychological examinations or tests *Daily lives of the subjects of the research, not healthy individuals 45 CFR 46.1029(i)

21. Which is greater? • The probability and magnitude of harm or discomfort anticipated in the research • Harm or discomfort ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests OR

22. Why is this a problem? One example • Practice Improvement Modules (PIMs) are required for board certification by American Board of Internal Medicine and others for Maintenance of Certification • PIMs provide physicians with a way to apply quality measurement to their practices, and then use the resulting data to take actions to improve care. • Physicians who are competent in practice-based learning and improvement and systems-based practice will be able to demonstrate the quality of care they deliver with data from their practice, and use that data for improvement, reporting, and upholding our professional value of accountability.

23. Why is this a problem? One example • PIMs are Web-based, self-evaluation tools that guide physicians through collecting data from their own practice, using chart reviews, patient surveys and a practice system survey to create a multi-dimensional practice performance assessment. • ABIM uses the data submitted electronically by the physician to provide an interactive report. • Detailed performance data serve as the basis for selecting areas for improvement and developing an improvement plan. • After implementing and testing the improvement plan, physicians report the impact of the implemented changes to ABIM. • What would be the impact on IRB review on doctors practicing in different settings?

24. Policy Options • Establish separate committees to review QI projects • IRB guidelines and submission of all projects • Hang a sign on front door of every hospital and physician’s office in the country

25. Superior Health Care Front Door Sign Thank you for deciding to receive your care here. Your safety and health is our primary concern. We strive to improve the care that you will receive here by engaging in a rigorous process of quality improvement in which we continuously plan new initiatives, implement them, study them rigorously, and act on what we find that works. By receiving your care here, you become a fully engaged participant in this process.