Robert D Rosenberg and Patricia Carney for Breast Cancer Surveillance Consortium

1. Screening Mammography Benchmarks – Modified Angoff: Screening Performance and Guidelines for Practice • Robert D Rosenberg and Patricia Carney for Breast Cancer Surveillance Consortium

2. Overview - Performance Benchmarks • Mammography Audit and Practice Recommendations before the BCSC • BCSC Early Efforts - Methods • BCSC Screening Audit results • BCSC and the Modified Angoff Process

3. Mammography Audit - History in the US • Single Practice • Frankl 1983/Wolfe 1987 (3,000/Yr) • Spring-Kimbrel/Sickles 1987-1990 (6,000/Yr) • Community Practice • Sienko 1993 and Rosenberg 1996 (42,000/yr) • Multi-community Practice • Yankaskas 2005 (240,000/yr)

4. BiRads Committee & Audit - ~ 1991 to Present • Collaborative work – ACR, ACS, NIH • Specific Discrete Results!!! • Importance of the Audit!! • Specified Audit Measurements • Preliminary Definitions • Recommendations and Assessments • Screening and Diagnostic

5. BCSC and the Audit • Methods: Standardized • Operational Definitions • Computerized methods • Applicable across practices • Assessment vs. Recommendation • Extended Types of Practices with Audits • From Selected Individual Practices to Regionally selected Practices

6. BCSC and the Audit II • Creation of Methods/Definitions for Research that are also Clinically Applicable • Consultation with ACR and Members of BiRads Committee • Consultation with Community Radiologists • Extensive Validation of Methods • How to ask Questions, and specific answers

7. Screening Benchmarks Distribution of Performance by Radiologists • 2,500,000 Screening Studies • > 300 Radiologists • Audit Measures: • Recall Rate, PPV 1,2,3, Sensitivity, Specificity, Cancer detection rate, Cancer Size and Stage Radiology: Volume 241: Number 1—October 2006

8. PPV of Biopsy Rec Sensitivity Specificity Cancer Detection Rate

9. Great Job Dad–Now What? • We’ve Created performance measures for the Community, what do we do with them? • What should they be/could they be?

10. Existing Guidelines: Expert Opinion • 1994 - AHCPR Clinical Practice Guideline #13: “Desirable Goals Achieved by Highly Skilled Experts” • i.e. - Expert Radiologists at Dedicated Facilities • Three Goals for Mammography Defined • High Sensitivity • Reasonable Rates of Recall and PPV2 • Cancers detected are small and Localized

11. Modified - Angoff Meeting • Meeting funded and assisted by American Cancer Society • Moderated and Organized by Patricia Carney • Assemble group of 10 “Expert” mammographers • Mix of academic and private practice radiologists • Mix of regions of the country

12. Angoff Method • Process Approach for Setting Cut-Point Criteria for Low Performers • Developed in the 1970s and applied in International and National Medical Student Assessment for USMLE-CPX • Purpose is to Increase “Accountability” for Meeting a Standard of Proficiency. • Most Commonly Used Method to Set Educational Performance Standards Today

13. Modified - Angoff Process • Question to Expert Panel: Beyond what range of performance would you recommend consideration for additional training? • Anonymous answers given and displayed • Experts view responses, discuss, repeat goals • Experts then given actual performance ranges in clinical practice - and repeat the process • Repeat for each performance measurement

14. AHCPR Expert performance 1994 vs. Angoff ~ Minimally acceptable expectations Radiology: Volume 255: Number 2—May 2010

15. Angoff - Implications for Care • The selected cut - points could recommend 18% to 49% of physicians for remediation for one or more measures. • If all physicians fell within the boundaries, there would be 14 more cancers/100,000 women screened and 880 fewer false positive studies.

16. Summary • It is a complex process to create valid benchmarks for community Radiology • These Benchmarks inform experts on realistic recommendations for community performance