Measurement theory and provider profiling

1. VA Ann Arbor HSR&D Center of Excellence Dept. Medicine, University of Michigan Measurement theory and provider profiling Timothy P. Hofer MD

2. construct indicator Quality ei(jk) ? The measurement problem

3. Levels of care Site (clinic, hospital) Provider (physician) Patient Indicators i i i

4. Implications of the measurement model • The indicator is a fallible measure of the construct • Some indicators are less precise than others • Quality indicators are very imprecise for a variety of reasons • You need to account for the measurement error • The location of the construct variability can suggest different causes, interventions and measurement procedures

5. Intra-class correlation(=reliability) • Ability to distinguish between physicians (or sites) • single observation under a specified set of conditions of measurement.

6. Vol. 281 No. 22, pp. 2065-2160, June 9, 1999

7. MD laboratory utilization profiles

9. VA Network 11 Diabetes Care Project

10. Resources available • VA Diabetes Registry Project (1998-2001) • Automated Clinical Databases • Data warehouse (VA Healthcare and analysis group) • Database Components • Encounter records (OPC/PTF ) • Outpatient Pharmacy • Lab • primary care provider database (PCMM () • Vitals • Cohort identification procedure • Data quality and measure validation • Kerr EA , et al. Journal on Quality Improvement 2002; 28(10):555-65.

11. Quality Selected Measures:Resource Use • Cost of hypoglycemic medications • Cost of home glucose monitoring for patients not on insulin • Cost of calcium channel blockers Outcomes Intermediate Outcomes Processes

12. Quality Selected Measures :Intermediate Outcomes • Last A1c value • A1c  9.5% • Last LDL value • LDL  3.6 mmol/L (140mg/dl) Outcomes Intermediate Outcomes Processes

13. Quality Selected Measures:Process Measures • Hemoglobin A1c obtained • LDL-C obtained • Lipid profile obtained Outcomes Intermediate Outcomes Processes

14. Selected Measures:Mixed or Linked Measure • LDL  3.6 mmol/L (140mg/dl) oron a statin

15. Are there differences between physicians? • What are the sources of variation? • Noise • Unmeasured differences • Physician effects • Clinic or group effects • Health System/payor effects

16. Outcomes

17. Intermediate outcomes

18. Process measures

19. Physician effect size

20. Physician effect size Negligible Small Moderate PCP Effect 200 Cost of homeglucose monitoring for patients not on Insulin Last LDL-C Value (1%) 150 Last LDL-C value <3.6 mmol/L or on a statin (5%) 100 Panel size Hemoglobin A1c obtained (8%) 50 Median PCP Panel size in study sample 0 .08 .10 .02 .04 Variance attributable to level of care

21. Implicit chart review – site level • Trained physician reviewers • 621 records • 26 clinical sites

22. Conclusions • Measurement models are fundamentally important to measuring and profiling quality. • There is often little reason or capability to profile at the physician level. • Profiles that ignore measurement error • Misrepresent the variability in quality • Are difficult (or impossible) to validate

24. Example – the imprecise thermometer • Budget cuts inspire innovation in the clinic

25. Observed temperature

26. Observed vs. true temperature

27. Strength in numbers 105 100 Body temperature(F) 95 90 85 true observed average

28. Scale transformation

29. Reliability “A person with one watch knows what time it is” “A person with two watches is never quite sure”

30. Effect of gaming