PrelimCheck: Software to Facilitate Follow-Up of Resident Preliminary Reports

1. PrelimCheck: Software to Facilitate Follow-Up of Resident Preliminary Reports Phillip Cheng, MD MS Assistant Clinical Professor of Radiology

2. Objective The purpose of this exhibit is to illustrate an open source desktop tool for follow-up of resident preliminary notes, allowing residents to rapidly evaluate their notes for discrepancies compared to final dictated reports. Patient and study identifiers shown in this presentation have been obscured, altered or synthesized.

3. Introduction • At our teaching institution, radiology residents begin taking independent call at night in their second year of training. • During the night, their preliminary interpretations are recorded as notes in the PACS (Fuji Synapse). Typically 50-100 notes are placed by a resident per call night. • The attending dictates the final reports in the morning and reviews preliminary notes by the resident.

4. Resident Notes in PACS

5. Resident Notes in PACS Attending note in the morning, resident now post-call at home Resident preliminary note at night

6. A Feedback Problem • Independent call experience is considered by our residents and faculty as one of the most important educational experiences of their residency. • However, feedback is an essential part of this education, and is not immediately available since the resident leaves in the morning before the attendings dictate final reads for the overnight studies. • In particular, there is no mechanism in the PACS for discrepancies between preliminary and final reads to be flagged for later review.

7. Reviewing Discrepancies We have reviewed several unsatisfactory options for reviewing overnight discrepancies with residents: • Morning readout with attending: post-call resident is typically tired and not receptive to feedback • Attending e-mail to residents: requires extra effort by attending especially for minor discrepancies, so such feedback is sporadic • Residents reviewing own reads from a previous call night • Attractive since the residents are usually motivated to review their call performance; however, identification and sorting of the preliminary and final reads from a call night is tedious and prone to incompleteness.

8. PrelimCheck: Facilitating Feedback PrelimCheck PrelimCheck is a tool that completes the communication loop between attendings and residents. ? Resident Notes in PACS, Dictated reports Notes in PACS Attending

9. PrelimCheck • Open source, portable Windows desktop application • Written in C#, requires .NET Framework 2.0 • Communicates directly with Fuji Synapse PACS

10. PrelimCheck - Usage • Easy-to-use tool for residents to compare their preliminary notes with attending follow-up notes and final reports • A resident specifies the location and time period of the call shift and clicks the Download button: Time Period Location

11. PrelimCheck - Usage Download • Performs NTLM authentication to Synapse PACS server • SQL query over network for notes performed in requested time period • Retrieve text of each note, and any dictated report for the corresponding study • Typical download time for 12 hr shift: approx 1 minute

12. PrelimCheck - Usage Sortable list of studies with notes placed in the specified time frame Search list by keywords or save to comma separated value (CSV) file All preliminary notes for highlighted patient study Dictated report, automatically scrolled to impression section

13. PrelimCheck - Usage • Requires approximately 5-10 minutes for review of overnight call cases • No built-in intelligence to detect and flag discrepancies between preliminary and final report • Ease of rapid review partially obviates this problem

14. Survey • Use of the program is encouraged but optional in our residency • One year after release, residents of all training levels were anonymously surveyed on usage, survey response rate 83%

15. Survey - Usage Most respondents used the program. The main stated reason for non-usage was lack of familiarity with program, or lack of call experience.

16. Survey – Usability Most respondents rated the program high in usability / ease of use.

17. Survey - Effectiveness Despite lack of automated flagging of discrepancies, most respondents indicated that the program improved discrepancy detection.

18. Sample of Anonymous Comments

19. Limitations and Future Work • Lack of automated discrepancy detection • Attendings can use standardized code words in notes to accurately flag and rate discrepancies • Slow network client-based implementation • Requires approximately 1 minute to download notes for a call shift • Faster response possible with a server-side implementation • Inability to filter results during network retrieval • Since notes are not indexed on PACS server, all notes within time period are retrieved and then filtered on the client side • Solvable with server-side implementation

20. Source Code Availability • PrelimCheck source code is available under the Apache 2.0 open source license. • All source available at Github: http://github.com/pmcheng/prelimcheck

21. Summary PrelimCheck, an open source desktop application, facilitates rapid resident self-evaluation of on-call preliminary radiologic interpretations. http://hsc.usc.edu/~phillimc/prelimcheck phillip.cheng@usc.edu