User Interfaces and Human-Computer Interaction

1. Randy Ellis, Ph.D User Interfaces and Human-Computer Interaction

2. Control Loop: Classical View Command (control input) Decision (difference between command and sensor) Sensor (state of WORLD) Action (effect on WORLD) WORLD: an unknown dynamical system

3. Control Loop: Surgeon’s View Command (control input) Computer: Integrates plans, sensors, etc. Sensor (screen update) Action (updates internals) INTERNALS: an unknown system (to the user)

4. Driven by Requirements A requirement is a property that, if not present, means that the software is useless Functional requirements include: • Features, speed, order of actions, etc. Nonfunctional requirements include: • Appearance, documentation, etc. Technical constraints include: • Hardware, operating system, etc.

5. Application-Oriented Architecture Base (common) software Trans-Gastric app. Neurosurgery app. MRI Biopsy app.

6. Storyboards for Applications Engineer and clinicians work together: • Define the basic approach to the interventional procedure • Prototype screen appearances (paper cartoons or PowerPoint are useful) • Specify screen-to-screen actions • Implement “happy-day” scenario • Elaborate alternative workflows (e.g., error handling, new image acquisitions) Result: fast development of well tested software

7. Technical Overview • Requirements drive the process • Application-specific: • Rapid specification of ideal workflow • Catch design problems early • Elaborate necessary extensions only, not all possible variants • Test-oriented: • The specification produces tests before code is written • Repeat all tests whenever changes are made Result: robust, trustable systems

8. Example Application: CT-Guided Brain Tumor Biopsy

9. Clinical Problem: Biopsy in CT Scanner Idea: • Perform brain biopsy using 2D slices Technique: • Attach fiducials to head holder • Scan holder and patient • Select slices showing structures at risk • Superimpose needle trajectory on slices • Verify biopsy with new CT acquisition(s)

10. CT Biopsy Phase 1: Procedure Workflow • Prepare patient (including fiducials) • Transfer patient to CT scanner • Acquire the CT scan (?inject?, image) • Load scan into Navigator • Software calculates registration • Software displays needle trajectory in CT • Perform the biopsy • Close the patient

11. CT Biopsy Phase 2: Software Actions • Load instrument descriptions • Load CT scan • Detect fiducials • Calculate registration • Select key CT slices • Track the needle guide • Overlay needle trajectory on slices

12. CT Biopsy Phase 3: Procedure Steps • Screen to select scan from disk • Screen to give registration progress • Screen to select CT slices • Screen to display navigation Additional Requirements: • Forward/backward buttons • Exit button • Tool status display (e.g., visibility)

13. CT scan found & loaded Fiducials detected & computed Slices selected by clinician CT Biopsy Phase 4: State Transitions File Screen Registration Screen Selection Screen Navigation Screen

14. Tool Status Display EXIT LOGO GOES HERE CT Biopsy Storyboard: File Selection All we need here is a simple dialog box

15. Tool Status Display EXIT LOGO GOES HERE CT Biopsy Storyboard : Registration Need a progress bar during registration, then report RMS error after registration

16. Tool Status Display EXIT LOGO GOES HERE CT Biopsy Storyboard: Selection Idea #1 SELECT SLICE 1/2/3/4 Show original slices in here  Scroll bar 

17. Tool Status Display EXIT LOGO GOES HERE CT Biopsy Storyboard: Selection Idea #2 Show original slices in here Drag/Drop into here  Scroll bar 

18. Tool Status Display EXIT LOGO GOES HERE CT Biopsy Storyboard: 2D Navigation Show the needle trajectory on top of each slice Slice #1 Slice #2 Slice #3 Slice #4

19. CT Biopsy Phase 5: Implement and Test • Create scans with/without fiducials (to test registration algorithms) • Build phantoms to measure navigation accuracy • Power cycle computer to test system robustness • Unplug/plug tracking system, etc.

20. CT Biopsy Phase 6: Next Iteration Next round, we may need to: • Calibrate the biopsy needle intraoperatively • Navigate the biopsy needle • Use needle-oriented reslicing • Migrate system to use MRI scans (new application: software re-use) • Control MRI scanner for in-scanner biopsy (new application: software re-use)

21. Example Application: Ventriculoscopy for Brain Tumor Biopsy

22. Clinical Problem: Deep Brain Biopsy Idea: • Use flexible scope to biopsy from within a brain ventricle Technique: • Same setup as for CT biopsy navigation • Show 3D location of scope inside volumetric image • Needs calibration of optical image to tracking device

23. Deep-Biopsy Phase 1: Procedure Workflow • Prepare patient (including fiducials) • Transfer patient to CT or MRI scanner • Acquire the 3D scan (inject & image) • Load scan into Navigator • Software calculates registration • Calibrate ventriculoscope • Software displays scope in volumetric scan • Perform the biopsy • Close the patient

24. Deep-Biopsy Phase 2: Software Actions • Load instrument descriptions • Load volumetric (MRI/PET/CET) scan • Detect fiducials • Calculate registration • Calibrate ventriculoscope to tracker device • Confirm calibration using a sharp probe • Track the ventriculoscope • Show scope in volumetric (MRI/PET/CET) scan

25. Deep-Biopsy Phase 3: Procedure Steps • Screen to select scan from disk • Screen to give registration progess • Screen to calibrate ventriculoscope • Screen to verify calibration • Screen to display navigation Additional Requirements: • Forward/backward/exit buttons • “Tabs” to other tasks  label “forward”

26. MRI/PET/CT scan found & loaded Fiducials detected & computed Calibration accepted by surgeon Calibration verified by surgeon Deep-Biopsy Phase 4: State Transitions File Screen Registration Screen Calibration Screen Verification Screen Navigation Screen

27. LOAD CALIBRATE VERIFY NAVIGATE REGISTER EXIT LOGO GOES HERE Deep-Biopsy Storyboard: File Selection All we need here is a simple dialog box ACCEPT

28. LOAD CALIBRATE VERIFY NAVIGATE REGISTER EXIT LOGO GOES HERE Deep-Biopsy Storyboard: Registration Need a progress bar during registration, then report RMS error after registration ACCEPT

29. LOAD CALIBRATE VERIFY NAVIGATE REGISTER EXIT LOGO GOES HERE Deep-Biopsy Storyboard: Calibrate Vscope Show live video feed in here CALIBRATE

30. LOAD CALIBRATE VERIFY NAVIGATE REGISTER EXIT LOGO GOES HERE Deep-Biopsy Storyboard: Verification 3D probe model on top of live video feed here ACCEPT

31. LOAD CALIBRATE VERIFY NAVIGATE REGISTER EXIT LOGO GOES HERE Deep-Biopsy Storyboard: 3D Navigation Scope field-of-view cone inside volume Live video feed from scope Forward rendering from scope

32. Deep-Biopsy Phase 5: Implement and Test • Create scans with/without fiducials (to test registration algorithms) • Calibrate optics to tracker device • Verify calibration with sharp probe • Build phantoms to measure navigation accuracy • Power cycle computer to test system robustness • Unplug/plug tracking system, etc.

33. Deep-Biopsy Phase 6: Next Iteration Next round, we may need to: • Calibrate the biopsy needle • Navigate the biopsy needle • Use needle-oriented reslicing • Acquire/integrate intraoperative MRI (new application: software re-use) • Migrate to laparoscopic ultrasound (new application: software re-use)