1 / 42

Depth Matters: Influence of Depth Cues on Visual Saliency

Depth Matters: Influence of Depth Cues on Visual Saliency. Congyan Lang , Tam V. Nguyen, Harish Katti , Karthik Yadati , Mohan Kankanhalli , and Shuicheng Yan Todays ’ Presenter : Daniel Segal Computer Vision – ECCV 2012

oki
Download Presentation

Depth Matters: Influence of Depth Cues on Visual Saliency

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Depth Matters: Influence of Depth Cues on Visual Saliency Congyan Lang , Tam V. Nguyen, Harish Katti , KarthikYadati , Mohan Kankanhalli , and Shuicheng Yan Todays’ Presenter : Daniel Segal Computer Vision – ECCV 2012 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part II

  2. Authors Congyan Lang Tam V. Nguye Harish Katti

  3. Seminar outline • Problem motivation • Challenges • Related work • Proposed solution • Pros & Cons • Limitations • Future ideas

  4. Problem motivation • Problem presentation • Does depth affects saliency? • If so How to incorporate depth data?

  5. Problem motivation • Why saliency? • Surveillance • Search and Rescue • Medical application • Image automated cropping • Advertising • Target recognition • Video summarizations • Preprocess for other algorithms

  6. Problem motivation • Why incorporate depth? • Are observers fixation different when viewing 3D images? • Human natural visual attention evolved in 3D environment • Absent of 3D fixation data

  7. Challenges • Finding efficient saliency models • Difficult to model top-down processes effect • Integrating depth additional information • Absent datasets for 3D stimulus • Absent of images dataset with corresponding depth maps • Subjective Which one is more conspicuous?

  8. Related work Computing Visual Attention from Scene Depth (2000) in ICPR Authors :NabilOuerhani and Heinz Hiigli • Intro • Based on Itti & Koch saliency algorithm • Easy to compute in parallel • Easy to incorporate depth features • Only visual evaluation method

  9. Top-Down overview Integration Feature extraction Feature Map 1 Center Surround Conspicuity Map 1 Range Finder Conspicuity Map i Saliency Map Feature Map i Center Surround Video Camera Feature Map n+m Center Surround Conspicuity Map n+m

  10. Integration step • Assign weights that promotes conspicuity M Wi Σ m Wi Conspicuity map i

  11. Limitation • No statistical analysis to prove improvements • No comparison with other methods • No quantities evaluation method • Proposed solution advantages • A new 3D dataset was created for statitical analysis • Evaluation and comparison with different methods • Experiments to investigate 3D saliency

  12. Related work Pre-Attentive detection of depth saliency using stereo vision(2010) in AIPR Authors :M. Zaheer Aziz and BarbelMertsching • Intro • Depth approximation using stereo images • Relates only to depth saliency

  13. Top-Down overview IR Algorithm IL Depth saliency magnitude

  14. Preprocessing Clipping Smoothing Segmentation IR IR 1 1 1 1 1 1 1 1 1/9 IL Depth saliency magnitude

  15. Key insight IR IL IR IL

  16. Main Algorithm explained/1 Segmentation IR Δ(x, y) Remove extra stripes - IL

  17. Main Algorithm explained/2 Remove occluded stripes Assign region depth

  18. Experiments and results • Human subjects marked depth saliency labels • Efficiency factor defined • Capability factor defined Nf Nf Nf<Ns Ns Ns Nf>Ns Penalty Nf =# of labels found Ns =# of labels

  19. Experiments and results

  20. Related work • Limitation • Requires stereo images • Approximated depth calculation • Emphasis on runtime • Experiments somehow dubious • Self invented evaluation index • Hard to compare with other methods • Ignoring fusion with contrast saliency algo.

  21. Related work • Proposed solution advantages • Messured depth data • Statistical analysis and comparison • Applicable on all 2D saliency algorithms • Using conventional evaluation index

  22. Proposed solution • Top-Down overview • Dataset collection (3D/2D) and analysis • Extracting Stereoscopic image pair generation for 3D display • Perform experiments to gain fixation maps • Observations and Statistics • Incorporating depth priors • Experiments and results

  23. Experiment setup

  24. Dataset Examples • Rejected images from dataset • images overlapping content with other images • Images with significant artifacts after the smoothing process

  25. Apply smoothing to depth maps D==0 ? D==0 ? 1 1 1 1 0 1 1 1 1 Laplacian Smooth Depth map 1/8∙ In the same super pixel Depth Map

  26. Stereoscopic image pair generation for 3D display Experiments conducted using active shutter glasses on a 3D LCD display Pixels translation: xl = xp+ρ, xr = xp−ρ ρ = parallax/2

  27. Quantitative evaluation methods/1 • Three quantitative methods for performance evaluation • Correlation coefficient(for all images in dataset) • Correlation between to given maps for different depth range bins • Saliency Ratio • Saliency mean energy in depth bin • AUC of the ROC • Use each one saliency map to predict another

  28. Quantitative evaluation methods/2 • Correlation coefficient • Saliency Ratio • AUC of the ROC

  29. Observations and statistics • Observation 1 • Depth cues modulate visual saliency at farther depth ranges • humans fixate preferentially at closer depth ranges

  30. Observations and statistics • Observation 2 • few objects account for majority of the fixations The average AUC for the entire 3D fixation dataset is 0.7399 and 0.7046 for 2D fixation dataset Using fixation maps as predictor for labeled object

  31. Observations and statistics • Observation 3 • The relationship between depth and saliency is non-linear and characteristic for low and high depth-of-field scenes

  32. Observations and statistics • Observation 4: • Fixation distribution discrepancy when multiple salient stimuli present different depth planes CC – Correlation coefficient

  33. Incorporating depth priors/1 • Proposed pdf • P(s,d/DOF) is GMM (Gaussian Mixture Model) Training set (80% of the dataset) Down Sample EM Algo. GMM model Parameters For each DOF bin P(s(x)|d(x), DOF) 200x200 pix 640x480 pix S(x) = ψ(x)(⊕/⊗)P(s/d,DOF)

  34. Incorporating depth priors/2 GMM Pdf From training Selected GMM Pdf Calc DOF Depth map Improved Saliency map +/* Saliency map Saliency Algo. Input image

  35. Experiments and results

  36. Experiments and results

  37. Experiments and results

  38. Proposed solution • Depth saliency

  39. Pros & Cons • Pros • Novel and provides a basis for further research • Simple approach • Gives an overall improvement for existing alg. • Cons • Few missing implementation details • No source code/other descriptive material • Requires a depth map • Still no basis for comparison with other methods Future ideas Integrate depth priors in various algo. instead of late fusion

  40. Questions ?

  41. Thanks

More Related