1 / 89

Advanced Stereo Vision Techniques: Binocular Stereo Matching

Explore stereo basics, binocular stereo matching, and advanced techniques for modeling from multiple views using cameras and dome setups. Learn about human vision, autostereograms, and Panum's fusional area in this insightful resource. Dive into disparity functions, occlusions, and rectified stereo geometry for in-depth understanding.

youngr
Download Presentation

Advanced Stereo Vision Techniques: Binocular Stereo Matching

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Stereo Vision ECE 847:Digital Image Processing Stan Birchfield Clemson University

  2. Outline • Stereo basics • Binocular stereo matching • Advanced stereo techniques

  3. Modeling from multiple views # cameras camera dome multi-baseline stereo ... trinocular stereo human vision binocular stereo ... photograph camcorder two frames time stereoV – Greek for solid S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  4. Stereoscope Invented by Wheatstone in 1838 S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  5. Modern version S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  6. Can you fuse these? left right No special instrument needed Just relax your eyes L R S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  7. Random dot stereogram invented by Bela Julesz in 1959 http://www.magiceye.com/faq.htm S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  8. Autostereogram Do you see the shark? http://en.wikipedia.org/wiki/Autostereogram S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  9. Can you cross-fuse these? right left Note: Cross-fusion is necessary if distance between images is greater than inter-ocular distance L R instead, trick the brain: impossible: L R Tsukuba stereo images courtesy of Y. Ohta and Y. Nakamura at the University of Tsukuba S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  10. Human stereo geometry fixation point disparity aR aL corresponding points http://webvision.med.utah.edu/space_perception.html S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  11. Horopter • Horopter: surfacewhere disparity is zero • For round retina,the theoretical horopteris a circle(Vieth-Muller circle) http://webvision.med.utah.edu/space_perception.html S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  12. Cyclopean image http://bearah718.tripod.com/sitebuildercontent/sitebuilderpictures/cyclops.jpg http://webvision.med.utah.edu/space_perception.html S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  13. Panum’s fusional area (volume) • Human visual system is only capable of fusing the two images with a narrow range of disparities around fixation point • This area (volume) is Panum’s fusional area • Outside this area we get double-vision (diplopia) http://www.allaboutvision.com/conditions/double-vision.htm S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  14. Human visual pathway S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  15. Prey and predator Cheetah: More accurate depth estimation Antelope: larger field of view photos courtesy California Academy of Science S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  16. Example: Motion parallel to image scanlines Epipoles are at infinity Scanlines are the epipolar lines In this case, the images are said to be “rectified” Tsukuba stereo images courtesy of Y. Ohta and Y. Nakamura at the University of Tsukuba

  17. Perspective projection X X x x M. Pollefeys, http://www.cs.unc.edu/Research/vision/comp256fall03/

  18. Perspective projection f f S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  19. Perspective projection X f x Z X x f Z S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  20. Standard stereo geometry • disparity is inversely proportional to depth • stereo vision is less useful for distant objects M. Pollefeys, http://www.cs.unc.edu/Research/vision/comp256fall03/

  21. Rectified geometry left optical axis world point IL right optical axis IR S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  22. Rectified geometry two cameras overlapped (for display) d = x1 – x2 = f (X1-X2) / Z = f b / Z disparity baseline depth S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  23. Matching space y xR xL S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  24. Matching space d=7 d=2 d=1 d=0 possible match between pixel 7 in left scanline and pixel 4 in right scanline impossible matches S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  25. Outline • Stereo basics • Binocular stereo matching • Advanced stereo techniques

  26. epipolar constraint Binocular rectified stereo 1D search:look for similar pixel in other image left right disparity map depth discontinuities S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  27. Disparity function left right occluded pixels higher slope = larger disparity = closer to camera smaller slope = smaller disparity = farther from camera lamp disparity wall pixel S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  28. Occlusions left: right: occluded pixels object disparity background pixel

  29. Matching a pixel • Pixel’s value is not unique • Only 256 values but ~100,000 pixels! • Also, noise affects value • Solution: use more than one pixel • Assume neighbors have similar disparity • Correlation window around pixel • Can use any similarity measure S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  30. Block matching • compute best disparity for each pixel • store result in disparity map left disparity map S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  31. Block matching (cont.) x x y y left right compare value for this disparity best so far? Yes Note: Window only moves left. Why? store it S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  32. Block matching disparity dissimilarity 5 nested for loops!!!!! S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  33. Block matching disparity dissimilarity 5 nested for loops!!!!! S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  34. Eliminating redundant computations for same disparity, overlapping windows recompute the same dissimilarities for many pixels S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  35. Block matching: another view • Alternatively, • precomputeD(x,y,d) = dissim( IL(x,y), IR(x-d,y) )for all x, y, d • then for each (x,y) select the best d x y d S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  36. More efficient block matching } separable Key idea: Summation over window is convolution with box filter, which is separable Running sum improves efficiency even more (only 3 nested for loops!!!) S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  37. More efficient block matching separable Key idea: Summation over window is convolution with box filter, which is separable Running sum improves efficiency even more (only 3 nested for loops!!!) S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  38. Comparing image regions Compare intensities pixel-by-pixel I(x,y) I´(x,y) Dissimilarity measures • Sum of Square Differences Note: SAD is fast approximation (replace square with absolute value) M. Pollefeys, http://www.cs.unc.edu/Research/vision/comp256fall03/

  39. Comparing image regions Compare intensities pixel-by-pixel I(x,y) I´(x,y) Dissimilarity measures If energy does not change much, then minimizing SSD equals maximizing cross-correlation M. Pollefeys, http://www.cs.unc.edu/Research/vision/comp256fall03/

  40. Comparing image regions Compare intensities pixel-by-pixel I(x,y) I´(x,y) Similarity measures Zero-mean Normalized Cross Correlation M. Pollefeys, http://www.cs.unc.edu/Research/vision/comp256fall03/

  41. Dissimilarity measures Most common: Connection between SSD and cross correlation: Also normalized correlation, rank, census, sampling-insensitive ... S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  42. Comparing image regions Compare intensities pixel-by-pixel I(x,y) I´(x,y) Similarity measures Census only compare bit signature using XOR, SAD, or Hamming distance (all equivalent) (Real-time chip from TZYX based on Census) M. Pollefeys, http://www.cs.unc.edu/Research/vision/comp256fall03/

  43. Sampling-InsensitivePixel Dissimilarity d(xL,xR) IL IR xL xR Our dissimilarity measure: d(xL,xR) = min{d(xL,xR) ,d(xR,xL)} [Birchfield & Tomasi 1998] S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  44. Dissimilarity Measure Theorems Given: An interval A such that [xL – ½ , xL + ½] _ A, and [xR – ½ , xR + ½] _ A If | xL – xR | ≤ ½, then d(xL,xR) = 0| xL – xR | ≤ ½ iff d(xL,xR) = 0 ∩ ∩ Theorem 1: (when A is convex or concave) Theorem 2: (when A is linear) [Birchfield & Tomasi 1998] S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  45. Small windows disparities similar more ambiguities accurate when correct Large windows larger disp. variation more discriminant often more robust use shiftable windows to deal with discontinuities Aggregation window sizes (Illustration from Pascal Fua)

  46. Occlusions left: right: If pixel matches do not agree in both directions, then unreliable

  47. Left-right consistency check d • Search left-to-right, then right-to-left • Retain disparity only if they agree Do minima coincide? xL Conceptually, S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  48. Left-right consistency check d for pixel (x,y) in left image, choices are D(x,y,0), D(x,y,1), D(x,y,2), …, D(x,y,max_disp) for pixel (x,y) in right image, choices are D(x,y,0), D(x+1,y,1), D(x+2,y,2), …, D(x+max_disp,y,max_disp) xL xL: xR: because xL = xR + disparity S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  49. Left-right consistency check d xL Actually, S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

  50. With left-right check inefficient: more efficient: S. Birchfield, Clemson Univ., ECE 847, http://www.ces.clemson.edu/~stb/ece847

More Related