1 / 9

The search for internal validity in improvement

The search for internal validity in improvement. Frank Davidoff Learning Lab – 2013 IHI Forum. Improvement has a two-part mantra. Part 1: All improvement involves change Part 2: Not all change is improvement. Local project, part 1: make change.

shubha
Download Presentation

The search for internal validity in improvement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The search for internal validity in improvement Frank Davidoff Learning Lab – 2013 IHI Forum

  2. Improvement has a two-part mantra • Part 1: • All improvement involves change • Part 2: • Not all change is improvement

  3. Local project, part 1: make change • “Here’s how we made (system-level) change happen…” • Identified a dysfunction in the system • Came up with an innovation (better process; change strategy for getting there) • Implemented our strategy in local context • Used small tests of change to refine innovation • Spread and maintained changes • (Way different from giving a pill…) • So what’s next?

  4. Local project, part 2: find out whether change is improvement • “Here’s how we learned whether our change was improvement…” • Chose outcomes: processes, patients’ clinical condition • Developed outcome measures • Created informal systems for collecting, displaying, using outcomes data (quantitative, qualitative) • Used these data locally to study the impact of changes, modify change strategy

  5. Yes, Virginia, there is “study” in local improvement projects • Informal study is an inherent part of all meaningful improvement • Used to check on impact of change (“Did it work?”) • Especially visible in “Plan-Do-Study-Act” cycles (originally called “Plan-Do-Check-Act” cycles) • Not related to whether project is meant “for publication”

  6. Where have we gotten in our local project? • In part 1: we made change happen • In part 2: we produced informal outcomes data to demonstrate improvement • Good enough: allows project staff to modify, spread, maintain change • BUT data quality (completeness, accuracy) is uncertain; no control for confounders, biases – “lite” study data • Result: weak internal validity! • i.e., unlikely to convince skeptics elsewhere about improvement

  7. How can we strengthen the evidence? • Shift “up” to formal planning and study • Identify plausible theory of performance change • Adopt specific study design • Select/define relevant outcomes • Develop reliable data collection process, robust data quality control • Analyze results (e.g., grounded theory; statistics; time series, esp. statistical process control) • Creates “research level” data • Result: stronger internal validity! • i.e., more likely to convince editors, peer reviewers, rest of the world – that our change was really an improvement

  8. Summing up • Making changelocallyincludes informal study of outcomes • Useful: makes project possible – but data somewhat “fuzzy” • Result: Internal validity is weak • Formal study of change process and outcome requires “research level” methods • Scholarly: contributes to general knowledge • Not every formal study feature is required, but the more features the better • Result: internal validity is stronger

  9. Internal validity in improvement studies:key reference • Solberg L, et al. The three faces of performance measurement: improvement, accountability, and research. Joint Comm J Qual Improvement 1997;23:135-47.

More Related