Dynamics of human eye-movements & visual search in medical imaging

1. Dynamics of human eye-movements & visual search in medical imaging Cyber-enabled Discovery and Innovation Workshop May 9, 2007 Deborah J. Aks RU-Center for Cognitive Sciences (RuCCs) Background research:Perceptual dynamics http://aks.rutgers.edu/.. daks@rci.rutgers.edu

2. CDI goals & themesComputationally based discovery concepts & (dynamical) toolsto understand complex, data-rich, and interacting systems. • --------------------------------------------------------- • Current project uses information & computing for discovery • *Knowledge extraction: Feature detection & pattern recognition • Learning from studying human (eye-movement) behavior: Expert vs. novice • Computational experimentation: Behavioral & modeling approach to learning about effective search strategies • Educating researchers and students in computational discoveryDynamical tools -- novel approach for cognitive researchers • Interacting elements Theoretical basis – brain as complex system & interacting network of neurons. Predicts implicit guidance of eye-movements, power law distributions & 1/f self-organizing behavior.

3. Visual search & tumor detection • Mammograms x-rays, CT-scans • Ultrasound • MRI…

4. Abnormal Features Image source:Edward J. Delp; Purdue University School of Electrical and Computer Engineering; Video and Image Processing Laboratory (VIPER) West Lafayette, Indiana, ace@ecn.purdue.edu http://bmrc.berkeley.edu/courseware/cs298/fall99/delp/berkeley99.htm http://www.ece.purdue.edu/~ace

5. Normal Mammograms Images source: E. J. Delp -- Purdue University http://www.ece.purdue.edu/~ace

6. Diagnostic Features • Abnormal Markings: • Shape & contours: --spicules or “arms” • --irregular borders • Size: • --variable: mm to cm+ • --Larger the tumor center, the longer its spicules • ------------------------------------------------------------------------ • Normal Markings: • Linear & smooth masses • ~ Normal ducts & connective tissue elements Abnormal & Normal Masses, Calcification..

7. Human -vs- Computer-aided detection • Which is better?Both use search, feature detection & classification • detecting abnormal structures • classifying lesions as benign or malignant • Human advantage: • Pattern recognition • Implicit memory/learning • Fewer false positives • (Unsystematic) search patterns can be effective • Computer advantage: • No fatigue • Explicit memory • Only biases are those built into algorithm • Thorough & systematic search

8. Challenging search task:Is there a tumor in this mammogram? (x, y) = (0,0) (1024, 0) mov (1024, 768) (x, y) = (0,768) Image source: Society for Breast Imaging (SBI)

9. Challenging search task:Is there a tumor in this mammogram? (x, y) = (0,0) (1024, 0) mov (1024, 768) (x, y) = (0,768) Image source: Society for Breast Imaging (SBI)

10. (0,0) (1024, 0) X,Y - Eye Samples (1 per ms) (1024, 0) (x, y) = (0,768) (1024, 768)

11. (0,0) (1024, 0) Saccades (0,768) (1024, 768)

12. (0,0) (1024, 0) Fixations (0,768) (1024, 768)

13. (0,0) (1024, 0) What search patterns produce effective search? (0,768) (1024, 768)

14. Dynamical toolsTime series (x,y eye-samples) x y

15. Time series Saccades Fixations

16. Time series Pupil Velocity Eye-samples: Saccade x y Fixation y Acceleration down x rt

17. Velocity y Acceleration Saccades Fixations x

18. Time series Saccades Fixations

19. Time series Saccades Fixations y x

20. Time series (x,y eye-samples) x rt lft up y down

21. Map trajectory of eye scan-paths: • x,y coordinates(position over time) • ------------------------------------------------------- • Saccades & Fixations: • Differences (xn – xn+1 ) & (yn – yn+1) • Distance =(x2 + y2)1/2 • Direction = Arctan (y/x) • Duration(msec, sec…)

22. Time series (x,y eye-samples)

23. Time Series10K+ fixations across search experiment 102 - 103K+ eye-samples!

24. Dynamical tools • Descriptive & Correlational Statistics • Scatter & Delay plots • Probability Distributions (PDFs) • Power spectra (FFT)… • Autocorrelation • Recurrent maps • Relative Dispersion (SD/M) • Rescaled range (R/S) • Iterated Functions Systems (IFS)

25. EyeFixations

26. Delay Plot of Fixations yn-vs- y n+1

27. Scaling across trials: Changes in fixations & scan-paths over time: • Frequency • Duration • Position …

28. Heavy-tail distributions • Power-laws • Small eye-mvmts are (very) common; large ones are rare. xn - x n+1

29. PDF’s & Networks A. Kurakin

30. Spectral analysis (e.g., FFT) Characterize frequencies making up time series f a f -2 = 1/ f 2

31. Noisy time series White Pink Brown

32. “Color’ or pattern White 1/f 0 1/f Pink 1/f 2 Brown

33. PowerSpectra of raw fixations 

34. PowerSpectra of first differences across fixations  = -.6

35. Distance across eye fixations (x2 + y2) 1/2  = -.47

36. Distance across eye fixations (x2 + y2) 1/2  = -.47  = -0.3  = -1.8

37. What patterns produce effective search? 1/f a ?

38. Ongoing research • Evaluate dynamic under different conditions • (learned vs. unlearned; expert vs. novice) • Does the dynamic change? • (white  brown  pink pattern) • Which pattern tends to produce most effective search? • (1/f pink?) • Evaluate biologically plausible models • (SOC & recurrent?) • Neural interactions --> search dynamic • (Complex system  effective guide to target info?)

39. Mainzer, K. (1997). Thinking in complexity: The complex dynamics of matter, mind & mankind. Berlin: Springer. Pg. 128