Stereovision Image Noise

1. StereovisionImage Noise John Morris Department of Computer Science, Tamaki Campus The University of Auckland Iolanthe at 14 knots on Cockburn Sound, Perth

2. Stereo Image Noise Sources • Signal noise • Electromagnetic interference eg cross-talk • Quantum behaviour of electronic devices eg resistor shot-noise • Quantization: digitization of real-valued signals • Geometric sources • Discrete pixel sensors with finite area • Occlusions • Perspective distortion • Opto-Electronic sources • Sensitivity variations between cameras • Different ‘dark noise’ levels • Real lenses • Depth-of-focus • Optical Surface properties • Single camera sources • Stereo (2-camera) sources Note that we use the term ‘noise’ for all problems!

3. Stereo Image Noise Sources • Optical Surface Properties • Lambertian scatterers • A “perfect” scatterer scatters light uniformly in all directions • Most correspondence algorithms assume perfect (Lambertian) scatterers • This means that surface patches will appear with the same intensity – independent of viewing angle .. • Simple matching • Intensities should be the same (perturbed by random noise!) • Reflectors

4. Stereo Image Noise Sources • Ordering Constraint • Imaged points appear in the same order in both images • Often used to simplify matching algorithms • Particularly dynamic programming • Violated by ‘poles’ – narrow objects in front of planes Scene points and image points in same order

5. Stereo Image Noise Sources Scene: A B C D • Ordering Constraint • Imaged points appear in the same order in both images • Violated by ‘poles’ – narrow objects in front of planes Left: a b d Right: d b c

6. Electronic Noise • Antennae (Receivers) • Wires act as antennae for EM waves • ‘Wire’ includes discrete wiresbut also • Tracks on circuit boards • Interconnects on chips • Transmitters • Any wire with a changing current emits EM waves

7. Electronic Noise • Digital circuits  very rapid transitions (switching events) • High frequency signals • Crosstalk • One wire is influenced by neighbouring wires Ideal digital signal Real digital signal ‘Instaneous’ rise or fall ≡ infinite frequency perfect radiator ‘Fast’ rise or fall high frequency very good radiator Signal driven into purple wire EM coupling Signal picked up on green wire

8. Electronic noise • Quantum effects • Resistor ‘shot’ noise • Resistive element is composed of discrete atoms • Always in motion for all T > 0oK (absolute zero) • Moving atoms ‘collide’ with electrons moving to form the current • Random fluctuations in current or • Noise as effective resistance changes • Similar effects in all current carrying or producing devices • Transistors • Capacitors • Inductors, etc e- e-

9. Electronic noise • Digitization noise • Analogue signal • Taking all possible values • At least at a macroscopic level! • Digital signal • Represented by a range of integers • 0 .. 255 (8 bit signal) • 0 .. 4095 (12 bit signal) • -2048 .. 2047 (12 bit signed signal) • A to D converter • Decides to which integer value to map a real value • Discretization • Values which differed (in real domain) become the same (in integer domain)

10. Opto-electronic noise Stereo Problem • Cameras have different gain settings • Amplifiers are not ‘matched’ perfectly • Sensors have different ‘dark current’ characteristics • All sensors produce some electrons (current) with no light • Quantum ‘tunneling’ out of the sensor device Different slopes Gains differ Current Different offsets Dark currents differ Light Intensity

11. Electronic Noise Summary

12. Geometric noise • ‘Pixelisation’ of images • Sensor is divided into discrete regions – pixels • ‘Edges’ in images don’t conveniently fall onto pixel boundaries Blue object Real image has blurred edges Red object

13. Geometric noise Stereo Problem MR ML • Occlusions • Points visible from one camera only • Points which it is impossible to match • Perspective distortion • Field of view in one camera differs from other • Left and right images contain different numbers of pixels • Impossible to match all pixels correctly B B B Binocularly visible ML Monocular Left MR Monocular Right

14. Perspective problem • Case 1 – ‘Fronto-planar’ object • Object with face •  to image planes •  to optic axes • extent,a • In image plane, • AL = AR = az / f • Case 2 – Angled object • AL > AR • Effect exacerbated by • Increased baseline • Increased angle of object • Verged axes,   

15. Optical effects Real lens Image plane • Real lenses • Finite depth of field • Image is ‘in focus’ over finite region only • Due to deviations of real (thick) lens from ideal shape • Depth of field increased by decreasing aperture • Always work with a small aperture! • f-stop • Ratio of focal length to aperture • ‘f-stop’ = f / a • ‘f’ = 1.4 – aperture wide open • ‘f’ = 22 – aperture closed down a f Mechanicalaperture For accurate work (objects in focus over a wide range) Use more light to permit narrower aperture

16. Optical effects Real lens Image plane • Depth of focus • Finite depth of field • Image is ‘in focus’ over finite region only • Due to deviations of real (thick) lens from ideal shape • Depth of field increased by decreasing aperture • Always work with a small aperture! • f-stop • Ratio of focal length to aperture • ‘f-stop’ = f / a • ‘f’ = 1.4 – aperture wide open • f’ = 22 – aperture closed down a Mechanicalaperture Depth of field f=22 f=1.4 For accurate work (objects in focus over a wide range) Use more light to permit narrower aperture

17. Lambertian Scatterers • ‘Perfect scattering’ • Intensity of scattered light is the same in all directions • Observers at all angles see the same intensity • Characteristics of a scattering surface • Optically ‘rough’ • Surface roughness ≈  •  = wavelength of incident light • 700-400 nm for visible light

18. Reflectors • Perfect reflector • 100% of incident intensity reflected • Angle of incidence = angle of reflection • All reflected intensity seen at one angle • Properties of a reflecting surface • Optically smooth • Surface roughness   i r i r

19. Real Surfaces Specular reflection • Reflector • Most of incident intensity reflected • Bright ‘spot’ seen over a small angle • Properties of a reflecting surface • Optically smooth • Surface roughness   • Specular reflections are the result of partly reflecting surfaces • Bright spots in images from • Water surfaces • Polished glass, plastic, … r i Smooth surface

20. Real Surfaces • Scattering surface • Reasonably uniform distribution of intensity with angle • Intensity of the same patch varies with angle • Intensity mis-match between L and R images • Systematic mis-match between views of the same patch • One is always less intense than the other • Shape of reflectance curve varies with surfaces • Random mis-match between different patches • Reflectance curves can be l-dependent also • Different views of the same patch appear to have different colours

21. Real Surfaces • Grating effects • In extreme cases • Regular surface variations with dimensions  • Grating effects • Spectrum generation • Small variations in view angle  large colour variations • Holograms • Systematic mis-match between different patches

22. Ordering constraint • If M appears to the left of N in the left image,it should appear to the left of N in the right image • Violated by ‘poles’

23. Stereo Correspondence is an ill-posed problem! or There are always multiple solutions!

24. Corresponding signals and possible surface profiles View direction right - scanline signal • Single surface reconstruction • Extreme disjoint variant left scan-line signal corresponding Areas (possible matches)

25. Effect of Noise but … What happens if we use ‘noise-free’ images?  Precise ‘ground truth’is available L Image - ‘corridor’ set Synthetic (ray traced)

26. Noise-free Image Matching • Mismatch is the matching error • R image pixels shifted by known d • (from ground truth) • Zero difference with L pixel? • Even ‘noise-free’ images have matching errors! Examine one scan line – line 152 Intensity MismatchIL(x)-IR(x-dx) Disparity(from ground truth)

27. Real Image Matching Tsukuba – line 173 Intensity ? MismatchIL(x)-IR(x-dx) Occlusion Disparity(from ground truth)

28. Noise-free Image Matching Examine one scan line – line 152 Intensity MismatchIL(x)-IR(x-dx) Edge effects No change in d,  no occlusion! Disparity(from ground truth)

29. Pixel-wise correspondences – ‘Tsukuba’ pair (line 173) Distribution of signal differences Grey-coded signal differences

30. Dealing with the noiseWindow-based correlation Epipolar line Window in reference image Possible windows in other image Algorithm For each pixel in the reference image For each possible disparity, d Calculate signal difference between windows centred on (x,y)L and (x-d,y)R Choose best matching window Output disparity

31. Matching in the presence of noise • Correlation algorithms • Match over a window • Correlation functions: • SSD (sum squared differences)  (IL(x+j.y+k)-IR(x+i-d,y+k))2 • SAD (sum absolute differences) |IL(x+j.y+k)-IR(x+i-d,y+k)| • Average matching errors • Handles random intensity fluctuations • Electronic, quantization, … noise • Doesn’t handle • Occlusions • Pixelisation • Intensity anomalies in one window compared to the other • Gain and dark noise variations • Perspective • Tries to assign all pixels in a window to the same disparity • Non-Lambertian surfaces & specular effects

32. Improving correlation • Normalized correlation functions • Normalize pixel intensities to mean within a window • Some tolerance for gain and dark noise changes • Often not worth the additional computational effort! • Better: Correct the images at source! • Align amplifiers, calibrate electronics, … • Other variations of cost function • Several hundred papers in the literature!

33. Correlation algorithms • Generally poor matching • Mismatches (small windows) or • Blurred edges due to large windows • Simple (almost trivial!) code • Adaptive windows improve performance 

34. Correlation-Based Methods • Matching Performance • Success of correlation methods depends on whether the image window in one image exhibits a distinctive structure that occurs rarely in the search region of the other image. • How to choose the size of the window, wh? • Too small a window • may not capture enough image structure and • may be too noise sensitive • many false matches • Too large a window • makes matching less sensitive to noise (desired) but also • decreases precision(blurs disparity map) • An adaptive searching window has been proposed

35. Correlation Methods Input – Ground truth 3x3 window Too noisy! 7x7 window Sharp edges are blurred! Adaptive window Sharp edges and less noise

36. Correlation algorithms • Generally poor matching • Mismatches (small windows) or • Blurred edges due to large windows • Simple (almost trivial!) code for each pixel position,w for each disparity, d Cwd = S(ILw+j-IRw+j-d)2 Choose min(Cwd) over d  d for position w • Fast enough?

37. Correlation algorithms • Not fast enough for small images • For nn image, d disparity values and hw window – O (n2dhw) • For small (300x300) images and low depth accuracy (5%  d = 20) • If n = 3x102, d = 20, h=w=9, then t(n) = c 81  2 x 10  9 x 104 ≈ 1.5 x 107 • For a 3GHz processor, cycles / pixel ≈ 3 109 / 1.5 x 107 ≈ 200 • ??? Enough to • Compute indices, • Fetch two pixels, • Subtract • Square or absolute difference • Find minimum from d costs • Repeat for adaptive window? but ……

38. Correlation algorithms • Moderate resolution images • For moderate (1Mpixel) images and reasonable depth accuracy (1%  d = 100) • If n = 103, d = 100, h=w=9, then t(n) = c 81  102 106 ≈ 1010 • For a 3GHz processor, cycles / pixel ≈ 3 109 / 1010 ≈ 0.3 • Inherently parallel Cwd = S(ILw+j-IRw+j-d)2 • Hardware acceleration needed!! • MMX (Intel Graphics Processing Extensions) helps • Optimized for simple arithmetic on pixels but • MMX pipeline needs to be filled efficiently • Max 16 operations in parallel • Excellent for 4  4 windows  • Usually 9  9 needed!

39. Correlation algorithms • Easy to implement in hardware • Match for each disparity at the same time • Add minimum circuit • Simple circuit .. • But O (hw) computations for each disparity • Relatively large circuitry

40. SAD Algorithm Block Diagram • Note:High ‘fan-in’ for SAD calculator blockStrains FPGA routing resourcesLarge buffers needed for scanlinesConsequence of window size

41. Correlation methods - summary • Simplest code • Poorest performance • Adaptive windows don’t help much! • Medium speed • Window size is an important factor • Simple hardware realization but • Expensive in resource use • Handle random noise only • Window is just treated as vector of pixels • No spatial information used • Occlusions ignored

42. Dynamic Programming Stereo • Attempts to find the ‘best path’ (sequence of disparity values) • Can recognize occlusions! • Averages noise over a scanline • Essentially local • Always moves ‘forward’ in a scanline • Solution generated by backtracking through predecessor array • Doesn’t adjust values in backtrack • Uses the ordering constraint • Readily adapted to • allow for gain and offset changes • Perspective distortion • ‘Stubborn’ • Incorporates a penalty for occlusions • Tends to make ‘streaks’ in disparity images ..\resources\DP_Gonzalez.pdf • Can be improved by using neighbouring scan lines • Requires fewer scan line buffers than correlation window

43. Dynamic Programming Result Note the horizontal streaks! It’s like a bureaucrat: once a DP algorithm ‘decides’ to adopt a disparity value – it doesn’t want to change its mind! Adding inter-scanline constraints (using a neighbouring scanline)generally improves this!

44. Dynamic Programming Performance • Better matching than correlation methods • ‘Global’ along scan lines • Recognizes occlusions • Uses ordering constraint • Uses spatial information • Time complexity O(n2d) • Faster than correlation methods in software • Uses memory (typical of DP algorithms) • For n=103, d=102, t = c 108 • c = ~30 cycles / pixel on 3GHz machine • Not enough for real time in software!

45. Dynamic Programming • Basic idea • Given a very large problem • Solve small problems first, • Save solutions to the small problems, • Use them to solve larger problem • Continue until large problem is solved • Dynamic programming uses storage space (for solutions to the small problems) to gain speed

46. Dynamic programming stereo • Small problems are the costs of a ‘path’ to a certain pixel • Costs are based on • Difference between pixel intensities • Cost for an occlusion • Sometimes - Discontinuity cost • Assume that disparity changes (depth changes) are relatively rare • Alternatively – assume most surfaces are smooth

47. Monocular Right Pixels Dynamic programming stereo Left scan line Start Right scan line This path would indicate all matches at zero disparity Monocular Left pixels Optimum path marks matching pixels Diagonal path – smooth surface – no change in disparity Jumps – changes in disparity End 47

48. Monocular Left Pixels Dynamic programming stereo Left scan line Start Right scan line Maximum disparity Monocular Right pixels DP algorithm calculates cumulative costs for all possible paths – using costs for previous pixel to estimate costs for the current pixel Compute costsfor matches in this region End 48

49. Monocular Left Pixels Dynamic programming stereo Left scan line Start Right scan line Maximum disparity As cost for each pixel at each disparity is determined, save the predecessor Use the predecessors to back track Need w predecessors – Biggest resource cost! Predecessor array stores disparity of best previous point Monocular Right pixels Compute costsfor matches in this region End 49