VI Safe & Sound - #708

1. VI Safe & Sound - #708 Bonnie Matlack, R.T. VistA Imaging Coordinator Kansas City VA Medical Center

2. Introduction • Background • Clinical – Radiology for 25 year • VistA Imaging Coordinator since 1999 • I truly am a “Jack-of-all-trades” however, can’t possibly know everything!

3. Plan to FailDo NotFail to Plan • This pertains to EVERYONE, there really are no exceptions • Think the unimaginable • Anything can happen to anyone, any facility, anywhere • Physical location is not the ONLY thing to consider! • Does your facility have true “back-up” personnel who are knowledgeable with the VistA Imaging??????

4. Start to Plan, Now • Implement a Daily Procedure or Check List • Have on-hand a Disaster Recovery Plan • Prepare a Backup Strategy • Label key pieces of equipment with critical information • Educate staff (IT, also Security Staff!) • Keep Callback List at Home • Did I miss anything from my list that maybe you are doing at your facility?

5. Daily Check List • Status indicator check – scan all hardware briefly noting any indicators that appear “out of the ordinary”, including, but not limited to, hard drives, systems, switches, gateways, Jukebox status panel, etc. • On Cluster servers, review Cluster Admin and Event Logs • Check all Servers and Gateways for error conditions, scanning for anomalous messages, including DICOM process windows, modality listeners, routing and telnet windows. • Check status of MediaStor and DiskExtender, ensure Extended Drive is online, media Move Group has sufficient platter space (add more platters if needed), pending move file count is decreasing, and jukebox and drives are online.

6. Daily Check List • On Background Processor, verify queue entries are processed and check Queue Manager for Jukebox entries that need re-queued • Remove and label completed copy media to secure location (copies should not be stored in the same location as the Jukebox). • Gateways should be scanned periodically through the day as permitted. • Check Vista error trap for any Imaging related issues.

7. Back Up Files • Types of Backup - Type of backup depends on scope of purpose and anticipated recovery scenario: Are we recovering from a mechanical/software failure on an individual device, or are we recovering from a large scale disaster? • Rule of thumb has been that patient images should exist on three simultaneous locations at once (RAID, Tape, Jukebox / Tape, Jukebox, Copy / etc). In other words as long as files still exist on RAID (have not been purged) it may be considered sufficient to store additionally to tape/jukebox, or jukebox/copy, however for files that HAVE been purged from RAID, those purged files should also exist on tape/Jukebox/copy simultaneously. • Ideally a 4 prong combination of RAID, Tape, Jukebox, and Media copies may be applied to provide the greatest redundancy.

8. Backup Types

9. Imaging File Details

10. Tape Backup - Full • Monthly or Quarterly • Depends on what type of tape system you actually have in place • How fast can your tape system back up • Can you get the tapes needed • All Image Shares

11. Tape Backup - Incremental • Daily • All Image shares • Dictionary files • BP log files • Platter reports • Compaction logs • Backup Exec catalog files

12. Tape Backup - System • Monthly • Dictionary files • BP log files • Platter reports • Compaction logs • Backup Exec catalog files

13. Platter Copies • Optical Media in Jukebox • Continuous • Store offsite

14. Back Up Tapes & Optical Media • Distance Matters and is something that needs to be evaluated for your specific site requirements and/or needs. • Example: • In the mountains, 20 miles may be considered to be a reasonable distance for offsite storage. • On the Coast however, the reasonable distance may be 100 miles inland for offsite storage of backups.

15. Offsite storage-Jukebox Copies • Platter copies and tapes in storage boxes • Store offsite

16. Hardware – RAID Drives - Label • Label, in ink, the disks in the RAID. In event they are taken offsite they can be inserted in the correct order. • Below is labeled using Port Target LUN

17. Resource Guide • Accurate Table of Contents • Emergency Contacts • System overview with good diagrams • Key information for each piece of equipment. • Function • Startup/Shutdown procedure • Contingency Plan • Install dates • Software version • Serial Number • Have a Review Schedule for your Plan-When/how often? • When a document is updated or changed, the document should reflect the “date” of the update (i.e. in the header or footer).

18. Resource Guide – Example – Flow Chart

19. Resource Guide – Example - JB Hardware Errors Jukebox Hardware error Date: 01/02/2010 If the JB is “offline” and gives you a “hardware error” you will need to Shut-down the JB and IMMJB1 to correct. To confirm that this is the problem, go to the Media Store application window and check VHAXXXJB1. Is it “brown” in color? If yes, right mouse click on it and try and put the JB back on-line. In this option, you will not want to inventory all shelves but only force the hardware to reset. If the message back to you tells you that there is a hardware error start the following procedure: Close the Disk Administrator application window on IMMJB1 Close the Media Store application window on IMMJB1 Shut down IMM2 Turn off the JB (white power switch in the lower right corner of device) Open the side panel (next to the server cabinet) and switch the path light to off Restart Procedure- Turn the Path Light back on, wait 2-3 minutes Turn the power back on to the JB, watch the LED on the left side. After the unit has run thru it’s routine diagnostics the display will read: DISC 525 and the “on-line” light will be green. Turn on IMMJB1 and let the server restart. Once server has restarted, open DEX admin application and the Media Store application from the icons on the desktop. May take a minute but you should then see the media come back on-line as well as the vhakanjb1.

20. Resource Guide – Example - Routing Instruct. ROUTING –BGP3 & DIG2 AUTO ROUTING- AUTO Routing Location: Dr. A Transmit #610 AUTO Routing Location: Dr. B Transmit #644 On-Demand Routing Locations: Dr. C Transmit #6 or network location KC-C **Do not auto route to Dr. C, only use the on-demand routing via Vista Rad Cache must be running Start a Telnet Session: 127.0.0.1 Log in w/vista credentials Select menu option #3-Routing On Routing menu, select #3 – Start Evaluator, hit return Start another Telnet Session: 127.0.0.1 Log in w/vista credentials Select menu option #3-Routing, hit return Select menu option #1 – Start Transmission Processor, hit return At next prompt or bottom of routing transmission list, type in 610 (this selects Dr. A site/location), hit return At next prompt or bottom of what looks/appears the same as #10, type in a single period . , hit return Routing will now start. You will see the evaluator run each image file and determine if the file is to be routed and then in the transmission window you will see the actual file Transmission.

21. Resource Guide – Example - Off-Site Storage Log Log contains pertinent information as to location and date of media transfer/movement. Current/most recent set of back-up tapes are sent to an off-site storage location. Includes information such as: Location of media Date Media sent to off-site location Back-up set ID/name Tape/Optical #’s Date Range of saved data Date Media Returned to the Medical Center

22. Resource Guide – Example - Back Up Policy PURPOSE: The Vista Imaging System software saves medical images as part of the patient's electronic record. Because these medical images are part of the patient's medical record they are protected by the Federal Privacy Act. PROCEDURE: 1. INCREMENTAL BACKUPS As images are captured, they are stored on magnetic servers and immediately copied to optical disk jukebox servers by the Background Processor (BGP1). This ensures that there are initially two copies of each image. The image files eventually are aged off of the magnetic file serves and only one copy of each image will exist on the optical platters. Because of the possibility of an optical platter failure, a secondary back-up of image files will exist via tape backup. Incremental backups will be performed on the following image share drives: vhaxxxclu3a - Image Shares 1$ - 8$ vhaxxxclu2a - Image Shares 1$ - 13$ This procedure will backup new images captured since the last backup was performed. Backup tapes are to be stored off-site. Media is not to be overwritten. Continuous incremental backup of the magnetic shares will be performed nightly (M-F). An initial 2. FULL SYSTEM BACKUPS Full backup of the magnetic image shares have been completed in _________________,2010. The most recent set of full system backup tapes are stored off-site. Media is not to be overwritten. Off-site Storage Location: ______________________________

23. Disaster Recovery Plan • The cornerstone of a good System Management Program is the implementation of a sound Disaster Recovery Plan. • Should be Thorough • Should be Accurate • Should be Up to Date • Should be Available

24. Do Not Fail to Plan REMEMBER, This pertains to EVERYONE, there really are no exceptions & FINALLY, You are responsible for your system so, prepare for the unthinkable