Advanced UI for Medical Imagery: Interactive Learning, Contextual Zooming, Gesture Recognition

1. An Advanced User Interface for Pattern Recognition in Medical Imagery:Interactive Learning, Contextual Zooming, and Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University

2. Outline • Introduction • Techniques: Segmentation, Magnification, Exploration • Solutions: • Interactive Learning • Contextual Zooming • Gesture Recognition • Conclusions

3. Introduction Medical imagery… • Consists of millions of images produced annually which doctors must gather and analyze • Entails several modalities for each patient, such as MRI, CT, and PET Refine techniques for facilitating comprehension of this data

4. Outline • Introduction • Techniques: Segmentation, Magnification, Exploration • Solutions: • Interactive Learning • Contextual Zooming • Gesture Recognition • Conclusions

5. Techniques • Common techniques for facilitating data comprehension: • Segmentation – Labeling of images • Magnification – Precision viewing • Exploration – Interacting intuitively with complex, 3D data

6. Why Segmentation? • Doctors and radiologists: • Spend several hours daily analyzing patient images (ie. MRI scans of the brain) • Search for patterns in images that are standard and well-known to doctors • Why not have the doctor teach the computer to find these patterns in the images?

7. Why Magnification? • Doctors and radiologists: • Must be able to precisely view and select regions/pixels of the image to train the computer • Can easily lose where they are looking in the image when using magnification • Why not use visualization techniques to preserve context while allowing precise selections?

8. Why Exploration? • Doctors and radiologists: • Need to intuitively interact with the system to maximize task performance • Need to perform this interaction while being unencumbered • Why not use vision-based recognition to allow interaction with the data?

9. Outline • Introduction • Techniques: Segmentation, Magnification, Exploration • Solutions: • Interactive Learning • Contextual Zooming • Gesture Recognition • Conclusions

10. Problems & Solutions • Problem #1: Segmentation • Solution #1: Interactive Learning • Problem #2: Magnification • Solution #2: Contextual Zoom • Problem #3: Exploration • Solution #3: Gesture Recognition

11. Platform • Med-LIFE: • “L”earning of MRI image patterns • “I”mage “F”usion of multiple MRI images • “E”xploration of the fusion and learning results in an intuitive 3D environment • Images used from “The Whole Brain Atlas” • http://www.med.harvard.edu/AANLIB/home.html

12. Outline • Introduction • Techniques: Segmentation, Magnification, Exploration • Solutions: • Interactive Learning • Contextual Zooming • Gesture Recognition • Conclusions

13. Simplified Fuzzy ARTMAP • Simplified Fuzzy ARTMAP (SFAM) • An AI neural network (NN) system • Capable of online, incremental learning • Takes seconds for tasks that take backpropagation NNs days or weeks to perform

14. Vector-based Learning • Two “vectors” are sent to this system for learning: • Input feature vector provides the data from which SFAM can learn • ‘Teacher’ signal indicates whether that vector is an example or counterexample

15. Feature Vector • Pixel values from images (16 for each slice)

16. 0.30 0.45 Category 1 - 2 members Category 2 - 1 member y Category 4 - 3 members x Learning Visualization • Vector-based graphic visualization of learning Array of Pixel Values

17. Full Results Detailed Results T2 Learning Associations

18. Varying Vigilance • Only one tunable parameter – vigilance • Vigilance can be set from 0 to 1 and corresponds to the generality by which things are classified (ie. vig=0.3=>human, vig=0.6=>male, 0.9=>Joshua New) 0.675 0.75 0.825

19. Category 1 - 2 members Category 2 - 1 member y Category 4 - 3 members x Vector 1 Vector 2 Vector 3 Input Order Dependence • SFAM is sensitive to the order of the inputs

20. Heterogeneous Network • Voting scheme of 5 Heterogeneous SFAM networks to overcome vigilance and input order dependence • 3 networks: random input order, set vigilance • 2 networks: 3rd network order, vigilance ± 10%

21. Network Segmentation Results

22. Segmentation Results Threshold results Trans-slice results Overlay results

23. Segmentation Screenshot

24. System Demonstration Interactive Learning

25. Segmentation Solution • Doctors and radiologists: • Spend several hours daily analyzing patient images (ie. MRI scans of the brain) • Search for patterns in images that are standard and well-known to doctors • Solution: • Doctors and radiologists can teach the computer to recognize abnormal brain tissue • They can refine the learning systems results interactively

26. Outline • Introduction • Techniques: Segmentation, Magnification, Exploration • Solutions: • Interactive Learning • Contextual Zooming • Gesture Recognition • Conclusions

27. Zooming Approaches Inset Overlay Chip Window

28. Research & Business • Carpendale PhD Thesis • Elastic Presentation Space – rubber sheet images via mathematical constructs • IDELIX (www.idelix.com) • Pliable Display Technology – software development kit (SDK) product • Boeing: 20% increase in productivity

29. Zoom Visualization Contextual Zoom Wireframe View

30. System Demonstration Contextual Zoom

31. System Comparison Contextual Zoom Previous System Zoom Overlay

32. Magnification Solution • Doctors and radiologists: • Must be able to precisely view and select regions/pixels of the image to train the computer • Can easily lose where they are looking in the image when using magnification • Solution • They can precisely select targets/non-targets • They can zoom for precision while maintaining context of the entire image • The interface facilitates task performance through interactive display of segmentation results

33. Outline • Introduction • Techniques: Segmentation, Magnification, Exploration • Solutions: • Interactive Learning • Contextual Zooming • Gesture Recognition • Conclusions

34. Motivation • Gesturing is a natural form of communication: • Gesture naturally while talking • Babies gesture before they can talk • Interaction problems with the mouse: • Have to locate cursor • Hard for some to control (Parkinsons or people on a train) • Limited forms of input from the mouse

35. Motivation • Problems with the Virtual Reality Glove as a gesture recognition device: • Reliability • Always connected • Encumbrance

36. Gesture Recognition System System Diagram User Rendering Update Object User Interface Display Hand Movement Image Capture Image Input Standard Web Camera

37. System Performance • System: • OpenCV and IPL libraries (from Intel) • Input: • 640x480 video image • Hand calibration measure • Output: • Rough estimate of centroid • Refined estimate of centroid • Number of fingers being held up • Manipulation of 3D skull in QT interface in response to gesturing

38. Calibration Measure • Max hand size in x and y orientation(number of pixels in 640x480 image)

39. Saturation Extraction Saturation Channel Extraction (HSL space): Original Image Hue Lightness Saturation

40. Extract Saturation Channel Threshold Saturation Channel Find Largest Connected Contour Gesture Recognition Pipeline

41. Segment Hand From Arm Calculate Refined Centroid Calculate Centroid Gesture Recognition Pipeline

42. Gesture Recognition Pipeline a) 0th moment of an image: b) 1st moment for x and y of an image, respectively: c) 2nd moment for x and y of an image, respectively: d) Orientation ofimage major axis: Calculate Orientation

43. Count Number of Fingers Gesture Recognition Pipeline • The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses • A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance) • Total fingers is number of fingers minus 1 for the hand itself

44. System Setup System Configuration System GUI Layout

45. Interaction Mapping Gesture to Interaction Mapping Number of Fingers: 2 – Roll Left 3 – Roll Right 4 – Zoom In 5 – Zoom Out

46. Gesture Recognition Demo

47. Exploration Solution • Doctors and radiologists: • Need to intuitively interact with the system to maximize task performance • Need to perform this interaction while being unencumbered • Solution • Can use intuitive gesturing to interact with complex, 3D data • Can interact by simply moving their hand in front of a camera, requiring no physical device manipulation

48. Outline • Introduction • Techniques: Segmentation, Magnification, Exploration • Solutions: • Interactive Learning • Contextual Zooming • Gesture Recognition • Conclusions and Future Work

49. Interactive Learning • Users can teach the computer to recognize abnormal brain tissue • They can refine the learning systems results interactively • They can save/load agents for background diagnosis on a database of medical images or to allow expert analysis in the absence of a well-paid expert

50. Contextual Zoom • They can zoom for precisely viewing and selecting targets/non-targets while maintaining context of the entire image • The interface facilitates task performance through interactive and customizable display of segmentation results • This system can be used with any 2D images and even with 3D datasets with some minor alterations