Metrics And Measurements Justification, Definition, Application

1. Metrics And Measurements Justification, Definition, Application Daniel E Ford, MD, MPH Vice Dean for Clinical Investigation Johns Hopkins University

2. Why do we need research study initiation performance measures? • Perception of sponsors and research faculty we are inefficient • No other accreditation organization has taken on this role • Association of Human Resource Protection Programs (AHRPP) • Office of Human Research Protections (OHRP/HHS) • Federal Drug Administration (FDA) • Opportunity for “the doers” to be proactive and lead the way

3. Why do we need research study initiation performance measures? • Measurement is the first step to understanding the process • Same metrics across multiple academic sites allows comparisons and identification of best processes (ask for more resources if not performing as expected) • Measurement of performance metrics tends to reduce variability

4. Why do we need research study initiation performance measures? • Posting metrics for sponsors and public demonstrates responsibility and accountability • Once we demonstrate accountability, can ask for same from sponsors and research teams

5. Performance Metrics Should Be • Specific • Measurable • Attainable • Realistic • Timely

6. Challenges to measuring performance • Selecting the right performance measures • Are study protocols in the performance measure representative or informative? • Are the performance measures easy to measure? • Can the performance measures be measured with reliability? • Does the performance measure vary too much by type of study to be informative? (pediatric vs. adult, first site versus 50th site) • Do the performance measures provide hints about the process or what the path to improvement might be?

7. Challenges to measuring performance • Is speed the only measure of performance? What about quality of IRB review? What about quality of a contract that is clear and covers multiple potential problems? • Are there ways to “game” the performance measures?

8. Examples? • HEDIS1 quality of care measures • Pay for performance measures • CMS2 quality of care measures • Healthcare Effectiveness Data and Information Set (HEDIS) is a tool used by more than 90 percent of America's health plans to measure performance. • Centers for Medicare & Medicaid Services (CMS)

9. First Performance Metric • Duration in days from receipt of first “ready” study protocol/site agreement received to time both the clinical investigational site and sponsor have executed the agreement

10. Performance Metric Definitions • What studies are included? • Multicenter clinical trials • Easier to assign date when site was able to start study initiation/approval process • Include all sponsors (NIH and commercial) • Dates • 2007 chosen as most study protocols will have had opportunity to be activated

11. Performance Metric Definitions • When is start date? • Date agreement sent from sponsor • Date agreement is received • Research teams may not have recorded this date

12. Performance Metric Definitions • Time to executed site agreement • Is it date when one party signed agreement or when all parties signed agreement? • Possible to “game” this time by signing agreements quickly and then need multiple amendments to really have study ready to enroll participants

13. Performance Metric Definitions • Days to execution of agreements • How to count protocols that never were completed? • What about outliers? • Use median or 90th percentile? • 25, 50 and 75 percentile?

14. First Performance Metric • Duration in days from receipt of first “ready” study protocol/site agreement received to time both the clinical investigational site and sponsor have executed the agreement

16. Days to Contract Execution by Percentiles

17. IRB Performance Metric • Duration in days from PI receipt of protocol to site IRB approval • Measure protocol receipt to initial (department or IRB) submission • Measure protocol receipt to IRB submission complete • Measure protocol receipt to first IRB review

18. IRB Performance Metric

19. IRB Outcome Measures

20. Lessons Learned • Separate offices for grants, contracts and the IRB, and separate databases, make it difficult to track performance measures • Commercially and federally funded studies should be tracked separately because the data points are different (e.g., the initial data point for federally funded studies is not the date the site agreement was received) • Research administration databases usually track by funding source; difficult to track studies with multiple funding sources • The PI on the grant may not be the same as the PI on the IRB submission, making it hard to locate the study • The IRB and research administration do not have a common tracking number for each study

21. Lessons Learned • Date that the PI received the protocol is not being tracked in existing databases – PIs had to be contacted to get this information • 58 PIs contacted - 42 provided information 7 sponsors contacted - 5 provided information • Although the PIs and sponsors were helpful, this was a time consuming process and the information will need to be captured systematically in a database to be used as a performance measure