AHRQ Quality Indicators

AHRQ Quality Indicators EPC Team (PSI Development) PI: Kathryn McDonald, M.M., Stanford Patrick Romano, M.D., M.P.H, UC Davis Jeffrey Geppert, J.D., Ed.M., Stanford Sheryl Davies, M.A., Stanford Bradford Duncan, M.D., M.A., Stanford Kaveh G. Shojania, M.D., UCSF Developed by Stanford-UCSF Evidence Based Practice Center Funded by the Agency for Healthcare Research and Quality Support of Quality Indicators PI: Kathryn McDonald, M.M., Stanford Sheryl Davies, M.A., Stanford Patrick Romano, M.D., M.P.H, UC Davis Jeffrey Geppert, J.D. Ed.M., Stanford Mark Gritz, PhD, Battelle Greg Hubert, Battelle Denise Remus, RN PhD, AHRQ Project Officer

Acknowledgements Funded by AHRQ Contract No. 290-97-0013 Support of Quality Indicators Contract No. 290-02-0007 Presentation funded by AHRQ Data used for analyses: Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality State Inpatient Databases (SID), 1997 (19 states). Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality

Acknowledgements • We gratefully acknowledge the data organizations in participating states that contributed data to HCUP and that we used in this study: the Arizona Department of Health Services; California Office of Statewide Health and Development; Colorado Health and Hospital Association; CHIME, Inc. (Connecticut); Florida Agency for Health Care Administration; Georgia Hospital Association; Hawaii Health Information Corporation; Illinois Health Care Cost Containment Council; Iowa Hospital Association; Kansas Hospital Association; Maryland Health Services Cost Review Commission; Massachusetts Division of Health Care Finance and Policy; Missouri Hospital Industry Data Institute; New Jersey Department of Health and Senior Services; New York State Department of Health; Oregon Association of Hospitals and Health Systems; Pennsylvania Health Care Cost Containment Council; South Carolina State Budget and Control Board; Tennessee Hospital Association; Utah Department of Health; Washington State Department of Health; and Wisconsin Department of Health and Family Service.

Outline • Administrative data and quality indicators • AHRQ Quality Indicators (QI) • Development of AHRQ QIs • Risk adjustment & MSX smoothing methods • Application of QIs to research and quality

History of AHRQ QIs/PSIs • Healthcare Cost and Utilization Project (HCUP) • HCUP discharge data collection (1988) • HCUP Quality Indicators • Mortality for Inpatient Procedures • Complication Rates • Potentially Inappropriate Utilization • Potentially Avoidable Hospital Admissions

Refinement of HCUP QIs • Refinement commissioned by AHRQ in 1999 • Completed by UCSF-Stanford EPC • Two related projects • Two technical reviews • Refinement of the HCUP Quality Indicators • Measures of Patient Safety Based on Administrative Data • Three indicator sets, AHRQ QIs • Inpatient Quality Indicators (IQIs) • Prevention Quality Indicators (PQIs) • Patient Safety Indicators (PSIs)

Opportunities Coding practices improving Data availability improving (e.g., less truncation) More specific codes Large data sets improve precision Comprehensive: all hospitals Quality screening feasible Obstacles Coding errors introduce noise Lack of information on timing, comorbidity vs. adverse events Varying number of secondary diagnoses fields can cause bias Heterogeneous severity within single code Administrative Data & Quality Improvement

Administrative Data State Inpatient Databases • Includes ICD-9-CM dx and procedure codes, DRG, dates, age, sex, payer, race, discharge disposition, hospital and/or patient zip codes • 1995-2002 • 33 States • 80% + of all U.S. hospital discharges • 18 states available for purchase • In 27 state sample, approximately 3200 hospitals

Administrative Data Nationwide Inpatient Sample (NIS) • Sampling of State Inpatient Databases • 1988-2001 • 7.5 million discharges/1,000 hospitals/33 States • Approximates 20% sample of nonfederal acute care hospitals • Discharge level weights applied for national estimates • Available for purchase

HCUPnet • http:///hcupnet.ahrq.gov/ • Web-based tool to query NIS and KIDS databases, 1993-2001 • Pre-run tables for 1997-2001 • Query based on ICD-9-CM, DRG or CCS • Information on hospitalizations, charges, length of stay, mortality, discharge status • Stratification by age, sex, race, income, insurance, hospital characteristics • Rank order hospitalizations

Sample AHRQ QI definition

Prevention Quality Indicators (PQIs) • Defined using area population as denominator • Potentially avoidable hospitalizations or ambulatory care sensitive conditions • Conditions for which good outpatient care can potentially prevent the need for hospitalization or for which early intervention can prevent complications or more severe disease • Public health, comprehensive health care systems • Based on existing, validated indicators set, but modified and updated

Prevention Quality Indicators (PQIs) • Bacterial pneumonia • Dehydration • Pediatric gastroenteritis • Urinary tract infection • Perforated appendix • Low birth weight • Angina without procedure • Congestive heart failure • Hypertension • Adult asthma • Pediatric asthma • Chronic obstructive pulmonary disease • Diabetes short-term complication • Diabetes long-term complication • Uncontrolled diabetes • Lower-extremity amputation among patients with diabetes

Inpatient Quality Indicators (IQIs) • Defined using both hospital admissions and area population as denominator • Inpatient mortality for certain procedures and medical conditions • Utilization of procedures for which there are questions of overuse, underuse, and misuse • Volume of certain procedures • Risk-adjusted using APR-DRGs • Potential for internal quality improvement purposes • Based on existing, validated indicators

Mortality Rates for Conditions Acute myocardial infarction (2 versions) Congestive heart failure Gastrointestinal hemorrhage Hip fracture Pneumonia Stroke Mortality Rates for Procedures Abdominal aortic aneurysm repair Coronary artery bypass graft Craniotomy Esophageal resection Hip replacement Pancreatic resection Pediatric heart surgery Hospital-level Procedure Utilization Rates Cesarean section delivery (primary and total) Incidental appendectomy in the elderly Bi-lateral cardiac catheterization Vaginal birth after Cesarean section (2 versions) Laparoscopic cholecystectomy Area-level Utilization Rates Coronary artery bypass graft Hysterectomy Laminectomy or spinal fusion PTCA Volume of Procedures Abdominal aortic aneurysm repair Carotid endarterectomy Coronary artery bypass graft Esophageal resection Pancreatic resection Pediatric heart surgery PTCA Inpatient Quality Indicators (IQIs)

Patient Safety Indicators (PSIs) • Defined using hospital admissions as denominator • Inpatient complications of care and potential patient safety events • Potential for internal quality improvement purposes, monitoring of patient safety events • Novel indicators, based on concepts reported in the literature

Provider-level Patient Safety Indicators Accidental puncture or laceration during procedure Complications of anesthesia Death in low mortality DRGs Decubitus ulcer Failure to rescue Foreign body left in during procedure Iatrogenic pneumothorax Selected infection due to medical care Postoperative hemorrhage or hematoma Postoperative hip fracture Postoperative physiologic and metabolic derangements Obstetric trauma – vaginal delivery with instrument Obstetric trauma – vaginal delivery without instrument Obstetric trauma – cesarean section delivery Postoperative pulmonary embolism or deep vein thrombosis Postoperative respiratory failure Postoperative sepsis Transfusion reaction Postoperative wound dehiscence in abdominopelvic surgical patients Birth trauma – injury to neonate Area-level Patient Safety Indicators Foreign body left in during procedure Iatrogenic pneumothorax Infection due to medical care Technical difficulty with medical care Transfusion reaction Postoperative wound dehiscence in abdominopelvic surgical patients Patient Safety Indicators (PSIs)

PQI Rates Source: SID, 2000. AHRQ Prevention Quality Indicators SAS Software Version 2.1 Revision 3.

IQI Rates Source: SID, 2000. AHRQ Inpatient Quality Indicators SAS Software Version 2.1, Revision 3. Release pending.

PSI Rates Source: NIS, 2000. AHRQ Patient Safety Indicators SAS Software Version 2.1 Revision 2. Release pending.

Methods • Evaluation framework • Literature review • Identification of indicators • Gray literature/interviews • Identification of indicators • Literature review • Evidence for indicators • Empirical analyses • ICD-9-CM coding review (PSI only) • Clinical panel reviews (PSI only)

Evaluation Framework • Face validity: does the indicator capture an aspect of quality that is widely regarded as important and subject to provider or public health system control? • Precision: is there a substantial amount of provider or community level variation that is not attributable to random variation? • Minimum Bias: is there either little effect on the indicator of variations in patient disease severity and comorbidities, or is it possible to apply risk adjustment and statistical methods to remove most or all bias? • Construct validity: does the indicator perform well in identifying true (or actual) quality of care problems? • Fosters real quality improvement: Is the indicator insulated from perverse incentives for providers to improve their reported performance by avoiding difficult or complex cases, or by other responses that do not improve quality of care? • Application: Has the measure been used effectively in practice? Does it have potential for working well with other indicators?

Literature ReviewIdentification of Indicators • Systematic review to identify indicators • Thousands of articles screened • Over 200 abstracted • Only 20 + articles actually described indicators, most of which had overlapping indicators • Grey literature searched to identify over 200 indicators

Empirical Analyses • Used novel statistical methods to measure • Precision/Reliability • Bias • Inter-relatedness of indicators • Precision criteria of 1.0% or more systematic variation among providers • Then, literature review conducted

Literature ReviewEvidence for Each Indicator Identified and reported evidence for: • Face validity • Precision and reliability • Potential bias • Construct validity • Fosters true quality improvement (gaming) • Current use

PSIs Methods Development of Candidate Indicator List • Background literature review • Little evidence in peer reviewed journals • Complications Screening Program • Miller et al. Patient Safety Indicators • Review of ICD-9-CM code book • Codes from above sources grouped into indicators and assigned denominators • Review of CSP evidence to retain indicators • Final refinements of indicators

PSIs Methods Review of Candidate Indicators • Literature review of potential indicators • Coding validity/consistency • Construct validity • ICD-9-CM coding review • Clinical panel review (face validity) • Results used to define final set of indicators

PSIs Methods Clinical Panel Review • Intended to establish consensual validity • Modified RAND/UCLA Appropriateness Method • Doctors of various specialties/subspecialties, nurses, specialized (e.g., midwife, pharmacist) • Initial rating, followed by conference call, followed by final rating • Rated indicator on: • Overall usefulness • Present on admission • Preventability of complication • Likelihood due to medical error • Extent indicator subject to bias • Eight multispecialty panels, three surgical panels (5-9 members on each panel)

Postop Pneumonia Decubitus Ulcer (5) (8) (7) (8) (4) (8) (8) (2) (7) (6) (3) (7) Example reviewsMultispecialty Panels • Overall rating • Not present on admission • Preventability (4) • Due to medical error (2) • Charting by physicians (6) • Not biased (3)

PSIs Methods Final Selection of Indicators • Indicators for which “overall usefulness” rating was high • Some changes in indicator set based on coding review and operationalization concerns (e.g., reopening of surgical site) • Empirical analyses of nationwide rates, variation, impact of risk adjustment, and relationship between indicators

Risk-Adjustment Criteria • User-specified criteria for evaluating risk-adjustment systems 1) “Open” systems preferred 2) Data collection costs minimized and well-justified 3) Multiple-use coding system 4) Official recognition

Evidence on DRG-based Systems • Open systems • Widely adopted by state agencies • Based on existing data collection systems • Use for reimbursement ensures improved data quality • Evidence suggests at least equivalent performance across broad spectrum of conditions • Studies underway to examine alternatives

3M APR-DRG • All-patient refined (956 categories in version 15.0, including pediatrics) • Severity of illness subclass that reflect presence of co morbidity/complication and level • Risk of mortality subclass • Differential impact of secondary diagnosis by condition

Evidence on 3M APR-DRG • All-patient refined (956 categories in version 15.0, including pediatrics) • Severity of illness subclasses that reflect presence of co morbidity/complication and level • Risk of mortality subclasses • Differential impact of secondary diagnosis by condition

Evidence on 3M APR-DRG • Better empirical performance than DRG-based alternatives on predicting mortality (especially for surgical patients; patients at large, urban, teaching hospitals) • Better empirical performance than DRG-based alternatives on predicting resource use (especially for medical patients; patients over 65, at children, teaching hospitals) • Better at reflecting the distribution of patient severity at the extremes

Risk-Adjustment Conclusions • No single system based on administrative or clinical data is clearly superior • DRG-based systems perform as well, and often better, than alternatives • Data enhancements may improve performance (e.g., condition present on admission, key clinical variables)

Risk-Adjustment ModelInpatient Quality Indicators • Direct standardization • Indirect standardization RA = (OR / ER) * PR (RA – risk adjusted; OR – observed; ER – expected; PR – population)

Risk-Adjustment Model Expected rate – Assuming the hospital’s case-mix and the population rates Risk-adjusted rate – Assuming the population’s case-mix and the hospital’s rates

Risk-Adjustment Model • Linear regression model: observed rate = hospital effect + demographic effect + condition effect + error • Model estimated on the SID, 2000 (25 million discharges)

Risk-Adjustment Model • IQI – Age, sex, APR-DRG (with risk of mortality or severity of illness subclass) (linear with hospital fixed effects) • PQI – Age and sex (linear with area fixed effects) • PSI – Age, sex, modified CMS DRG and AHRQ comorbidity (logistic)

How it Works: CABG Mortality

MSX Smoothing Model • Observed quality measure = true quality (signal) + error (noise) • Smaller hospitals and/or less frequent conditions have more noise • Difficult to compare hospitals, trend over time, and identify best practices • Confidence intervals reflect but do not address the problem

Key Features of MSX Approach • Removes noise – uses redundancy over time and among measures • Improves forecasts – predicting current quality based on past performance • Reduces dimensionality appropriately - allows meaningful summary measures • Reveals and helps reduce biases, identify best practices

Caveats of Use • Validity of data • Validity of coding • Present on admission • Outpatient care • Linking of admissions and impact of LOS • Incomplete risk adjustment

Using the AHRQ QI • State monitoring of rates • Hospital quality improvement • National Healthcare Quality Report • PQIs and PSIs • CMS Pay for Performance Demonstration Project • Postoperative Hemorrhage or Hematoma • Postoperative Metabolic and Physiologic Derangement • Romano et al • PSI National trends, (HA, Mar/Apr ’03)

AHRQ Quality Indicators