Image Noise

1. Image Noise John Morris Department of Computer Science, Tamaki Campus The University of Auckland

2. Stereo Image Noise Sources • Signal noise • Electromagnetic interference eg cross-talk • Quantum behaviour of electronic devices eg resistor shot-noise • Quantization: digitization of real-valued signals • Geometric sources • Discrete pixel sensors with finite area • Occlusions • Perspective distortion • Opto-Electronic sources • Sensitivity variations between cameras • Different ‘dark noise’ levels • Real lenses • Depth-of-focus • Single camera sources • Stereo (2-camera) sources Note that we use the term ‘noise’ for all problem sources!

3. Electronic Noise • Antennae (Receivers) • Wires act as antennae for EM waves • ‘Wire’ includes discrete wiresbut also • Tracks on circuit boards • Interconnects on chips • Transmitters • Any wire with a changing current emits EM waves

4. Electronic Noise • Digital circuits  very rapid transitions (switching events) • High frequency signals • Crosstalk • One wire is influenced by neighbouring wires Ideal digital signal Real digital signal ‘Instaneous’ rise or fall ≡ infinite frequency perfect radiator ‘Fast’ rise or fall  high frequency very good radiator Signal driven into purple wire EM coupling Signal picked up on green wire

5. Electronic noise • Quantum effects • Resistor ‘shot’ noise • Resistive element is composed of discrete atoms • Always in motion for all T > 0oK (absolute zero) • Noise as effective resistance changes • Moving atoms ‘collide’ with electrons moving to form the current • Random fluctuations in current or • Noise as effective resistance changes • Similar effects in all current carrying or producing devices • Transistors • Capacitors • Inductors, etc e- e-

6. Electronic noise • Digitization noise • Analogue signal • Taking all possible values • At least at a macroscopic level! • Digital signal • Represented by a range of integers • 0 .. 255 (8 bit signal) • 0 .. 4095 (12 bit signal) • -2048 .. 2047 (12 bit signed signal) • A to D converter • Decides to which integer value to map a real value • Discretization • Values which differed (in real domain) become the same (in integer domain)

7. Geometric noise • ‘Pixelisation’ of images • Sensor is divided into discrete regions – pixels • ‘Edges’ in images don’t conveniently fall onto pixel boundaries Blue object Real image has blurred edges Red object

8. Geometric noise Stereo Problem • Occlusions • Points visible from one camera only • Points which it is impossible to match • Perspective distortion • Field of view in one camera differs from other • Left and right images contain different numbers of pixels • Impossible to match all pixels correctly

9. Opto-electronic noise Stereo Problem • Cameras have different gain settings • Amplifiers are not ‘matched’ perfectly • Sensors have different ‘dark current’ characteristics • All sensors produce some electrons (current) with no light • Quantum ‘tunneling’ out of the sensor device Different slopes Gains differ Current Different offsets Dark currents differ Light Intensity

10. Effect of Noise but … What happens if we use ‘noise-free’ images?  Precise ‘ground truth’is available L Image - ‘corridor’ set Synthetic (ray traced)

11. Noise-free Image Matching Examine one scan line – line 152 Intensity MismatchIL(x)-IR(x-dx) Disparity(from ground truth)

12. Real Image Matching Tsukuba – line 173 Intensity MismatchIL(x)-IR(x-dx) Disparity(from ground truth)

13. Pixel-wise correspondences – ‘Tsukuba’ pair (line 173) Distribution of signal differences Grey-coded signal differences

14. Previous work: pros & cons Conventional approach: energy minimisation combining image dissimilarity, surface curvature and occlusions • Exact minimisation with dynamic programming: • global 1D optimum matching under ordering constraints; can account for local photometric (offset or contrast) deviations and occlusions; fast processing; • no inter-scan-line constraints; random deviations on textureless regions; error propagation along scan-lines • Approximate minimisation with Min-Cut techniques: • 2D surface curvature constraints (MRF); a provably close approximate solution of an NP-hard problem; can account for local occlusions; • (-)cannot account for local or global photometric deviations; high computational complexity • Heuristic approximate minimisation with Belief Propagation • 2D surface curvature constraints (MRF); can account for local occlusions; • cannot account for local or global photometric deviations; high computational complexity

15. Conventional approaches: basic problems No account for intrinsic ill-posed nature of stereo problems Search for a single surface giving the best correspondence between stereo imagesbut the single surface assumption is too restrictive in practice Heuristic or empirical weights of energy terms dramatically affect matching accuracy Large images and large disparity ranges lead to high computational cost of min-cut or belief propagation algorithms

16. Actual disjoint surface profiles and piecewise-constant corresponding signals right - scanline signal • Single surface reconstruction: • Extreme disjoint variant: left scan-line signal signal-based corresponding areas

17. Pixel-wise correspondences for a “Tsukuba” stereo pair (scan-line y = 173) Distribution of signal differences Grey-coded signal differences

18. Concurrent Stereo Matching: Main ideas • Human ‘stroke-wise’ analysis of a 3D scene • Eyes browse from low to high frequency regions, from sharp points to smooth areas rather than scan line-by-line (Torralba, 2003) • Appropriate (likely) correspondence rather than best matching • Separation of noise estimation and signal matching from selection of surfaces and occlusion handling Stereo matching should avoid the ‘best match’ or signal difference minimisation almost universally used nowin favour of a likely match based on a local signal noise model

19. Modular structure of CSM Step 1: Estimate the image noise model(allow it to be spatially variant) Segment based on noise Select candidate 3D volumes Step 2: Fit constrained surfaces to the candidate volumes Could use K-Mean, SUSAN, etc Could be surface optimisation

20. Noise Map Technique A: use a fast, efficient stereo matching technique (SDPS) to produce a disparity map – use mismatches as noise estimates Scaled (Amplitudex6) Noise Map White regions have higher noise - almost always appears in occluded regions

21. Noise-Driven Segmentation Colour Mean Shift Segmentation Colour-position clustering in a 5D feature space: 3D-colour model L*u*v and 2D-lattice coordinates The noise map is considered to be the extra, sixth dimension • Convert an image into data tokens • Choose initial search window locations • Compute the ‘mean shift’ window location for each initial position • Merge windows that end up on the same ‘peak’ or mode • Cluster data over the merged windows After noise-driven segmentation: occluded regions are segmented into small isolated blocks

22. CSM: candidate 3D volumes and surface fitting d= 3 d= 4 d= 5 d= 6 d= 7 d=8 d=9 d=10 d=11 d=12 d=13 d=14 Black regions contain likely matching points in the ‘slice’ for each disparity, dSurface fitting shrinks or expands each segmented region from slice to slice (based on counts of candidate points)

23. CSM: candidate 3D volumes and surface fitting Ideal disparity slices d=5 d=6 d=8 d=9 d=10 d=11 d=12 d=13 d=14 Disparity map CSM surface fitting d=10 d=5 d=6 d=8 d=9 d=11 d=12 d=13 d=14 CSM Disparity map

24. Symmetric CSM: candidate 3D volumes and surface fitting d=3 d=5 d=6 d=4 d=8 d=7 d=9 d=10 d=11 d=12 d=13 d=14

25. Symmetric CSM: candidate 3D volumes and surface fitting Ideal disparity slices d=10 d=5 d=6 d=8 d=9 d=11 d=12 d=13 d=14 Disparity map SCSM surface fitting d=5 d=6 d=8 d=9 d=10 d=11 d=12 d=13 d=14 SCSM Disparity map

26. Algorithm Comparison Symmetric DP stereo Graph cut Symmetric BP SCSM CSM Ground Truth

27. Middlebury Benchmark (MB) rank among 40 algorithms * MB-SBPO – symmetric belief propagation algorithm (best-performing Middlebury benchmark)

28. Conclusions • Stereo matching is an ill-posed problem, reconstruction of actual 3D optical surfaces is impractical • More reasonable goal: mimic human binocular stereo vision • Conventional constrained best matching does not explicitly account for a multiplicity of equivalent matches, for noise in both images of a stereo pair and for local contrast or offset image distortions • Concurrent stereo matching gives promising results because it separates the problem into • search for all the candidate volumes with equivalent good matches (allowing for the estimated noise) and • search for surfaces fitting to the volumes • Even the simplest implementation of the new approach competes with the best-performing conventional algorithms • Sloping surfaces challenge our CSM algorithm – watch this space!

29. IVCNZ’2006 • For a conference with a different style, consider IVCNZ’2006(Image and Vision Computing, New Zealand) • Great Barrier Island, New Zealand • Gateway to the Hauraki Gulf and Auckland • 40 mins by light plane from Auckland,3 hours by ferry • Full range of accommodation options: • Hotel style, cabins, … , even tents! Book early and you can sail there

30. Great Barrier Island

31. Stereo: Correspondence Problem • Stereo Pair • Images from identical cameras separated by some distanceto produce two distinct views of a scene Corresponding Regions Left Image Right Image xL xR 1 Disparity = xL - xR  z