Choices and Outcomes: The Effects of Improvement Project Portfolio Choices on Clinical Outcomes

1. Choices and Outcomes: The Effects of Improvement Project Portfolio Choices on Clinical Outcomes Anita L. Tucker Assistant Professor, U. of Pennsylvania Senior Fellow, Leonard Davis Institute of Health Economics Ingrid M. NembhardDoctoral Candidate, Harvard Business SchoolHarvard Graduate School of Arts and Sciences KLIC 4: INFORMS ConferenceNovember 6, 2006 Financial Support from HBS DOR, Wharton’s Fishman Davidson Center In Collaboration with Jeffrey Horbar, Richard Bohmer and Amy Edmondson

2. Background: Health Care in America The Data: Publicity: 1999: Institute of Medicine (IOM) reports 100,000 Americans die annually from preventable medical errors 1996: Dartmouth Atlas Project shows inappropriately geographic variations in care + 2000: IOM finds quality problems are a systemic property, requiring process improvement 1998: National Roundtable reports “serious and widespread problems”exist in American medicine” . . Problems of underuse, overuse, misuse . . “Quality of care is the problem.” 2003: RAND Study reports only 55% of patients receive the recommended care for their condition = Need for improvement projects in health care organizations

3. Improvement Projects Improvement Projects “solve complex organizational problems through the work of formal teams that use a structured improvement method” (Christianson et al., 2005: p. 610) • Healthcare Project Examples Target Area • Increase handwashing (Reduce Infections) • Reduce heelsticks in Neonates (Pain management) • Increase collaboration among MDs and RNs (Staff Retention) Portfolio of Improvement Projects: An organization’s set of improvement projects that are in progress at the same time and draw on the same limited set of human, managerial and financial resources (Cooper, Edgett & Kleinschmidt, 1999; Wheelwright & Clark, 1992)

4. Research Question How should health care organizations structure their portfolios of improvement projects to achieve better outcomes? • Hypotheses about Portfolio Choices • Number of projects (+) • Concentration within a target area (+) • Level of evidence for portfolio (+) • Novices should start with clinically-oriented portfolio (+) • Extent of physician involvement (+)

5. Research Setting: Vermont Oxford Network • Collaborative of Neonatal Intensive Care Units (NICU) • 44 NICUs working together for 2 ½ years (Apr 2002 - Oct 2004) • Identified 7 target areas for improvement and 93 improvement projects across those areas • Met twice a year to learn QI methods (PDSA cycles), work on developing best practice guidelines and share experiences • In between meetings, implemented practices, conducted site visits to other NICUs and phone conferences Horbar, J. D. et al, 2001. Collaborative quality improvement for neonatal intensive care. Pediatrics 107 (1) 14-22.

6. Portfolio Project Options Clinically-oriented Operationally-oriented **Each NICU indicated which projects they included in their project portfolio to the collaborative sponsor.

7. Implemented during collaborative Working on Unique Portfolios of Practices Excerpt from the practices from Pain Management Hospital 100 Implemented/Working on: Reducing frequency of tracheal suctioning, standarized sucrose analgesia, peripheral vascular procedures, circumcision, post op pain, weaning from opiods VERSUS Hospital 102: Reducing frequency of heelsticks

8. Evidence-base for the portfolio • Level of evidence for all projects within each target area assessed by target area team using Muir-Gray (MG) score to rate articles 1 = strong evidence from at least one systematic review of multiple, well-designed, randomized, controlled trials 2 = properly designed randomized control trial of appropriate size 3 = well-designed trials without randomization 4 = well-designed non-experimental trials 5 = opinions of respected authorities, based on clinical evidence, descriptive studies or reports of expert committees • Evidence base for NICU portfolio = the average MG score of the portfolio

9. Outcome: Standardized Mortality Ratio SMR level of analysis: Babies nested in NICU • LOGIT model (clustered by NICU) Outcome Death (0,1) • Independent Variables: Established risk factors (Zupanic et al. 2006) • Predict probability of death for each baby • By NiCU, sum up probability of death, actual deaths • Compute ratio: • SMR < 1 Outcomes BETTER than expected • SMR = 1 Outcomes equal to expected • SMR > 1 Outcomes WORSE than expected

10. Means, SD, and correlations (N=27)

11. H1: Supported u-shape H3: Less evidence->Imp H4: Supported OLS Regression results (H1, H3, H4) N = 26, (std error) ^ significant at 10%; * significant at 5%; ** significant at 1%

12. H2: Supported RH2: Concentrating the within a target area helps N = 26, (std error) ^ significant at 10%; * significant at 5%; ** significant at 1%

13. H5: Supported R H5: Initial portfolio orientation matters N= 11 (Robust std error) ^ significant at 10%; * significant at 5%; ** significant at 1%

14. Summary and Implications • An effective improvement project portfolio: • Includes neither too few or too many projects to manage the tradeoff between synergy and distraction • Concentrates its efforts within a target area to maximize inter-project learning • Focuses on operationally-oriented projects which build performance-improvement capability • For novices is clinically-oriented where clearer benchmarks are available • Is led by a team with significant physician membership.