Exposure Fusion for Time-Of-Flight Imaging

1. Exposure Fusion for Time-Of-Flight Imaging Uwe Hahne, Marc Alexa

2. Depth imaging [Audi] Applications [Shotton2011]

3. Depth imaging [PMDTec] [Microsoft] Devices [PTGrey]

4. General problem Video #1

5. Our results Video #2

6. Contribution • +No calibration • + Real-time capable • + Error reduction • - Only for time-of-flight imaging...

7. [PMDTec]

8. [PMDTec] Popt t

9. [PMDTec] Popt 50 ns (ω = 20 MHz) t Amplitude intensity Phase shift  distance

10. [PMDTec] Integration time Popt between 50 and 3000 µs t

11. Problem Noisy background Noisy foreground Short integration time (50 µs) Long integration time (1250 µs)

12. Sampling Popt t S3 S1 S2 S0 S0

13. Sampling Popt Short integration time  low SNR t S3 S1 S2 S0 S0

14. Sampling Popt Long integration time  saturation t S3 S1 S2 S0 S0

15. Whatisthe optimal integration time?

16. Existing approaches 1. Adaptation (May2006) • Updating the integration time based on data from previous frames. • Only a global solution Video #3 • 2. Calibration (Lindner, Schiller, Kolb,...) • Capture samples of known geometry and correct the measurements. •  Laborious and device specific [Lindner2007]

17. Existing approaches 1. Adaptation (May2006) • Updating the integration time based on data from previous frames. • Only a global solution • 2. Calibration (Lindner, Schiller, Kolb,...) • Capture samples of known geometry and correct the measurements. •  Laborious and device specific [Lindner2007]

18. Our approach How is this problem solved in photography?  High Dynamic Range (HDR) Imaging

19. HDR imaging Radiance map 101011101001010110101010101010010010101001001011000010100101000101001101011101001010110101010101010010010101011101001010110101010101010010 Over-exposed Details Missing details Details Under-exposed [Images are courtesy of Jacques Joffre]

20. Depth imaging Under-exposed Over-exposed

21. Algorithm • Capture a series of depth images with varying integration times. • Compute weights using quality measures. • Fuse images together as affine combination of weighted depth maps.

22. Processoverview Depth maps Laplace pyramids Weight maps Fused pyramid Resulting fusion

23. Processoverview Depth maps Laplace pyramids Weight maps Fused pyramid Resulting fusion

24. Quality measures Contrast Well exposedness Entropy Weight maps Surface

25. Quality measures Well exposedness Emphasizes those pixels where the amplitude is in a “well” range and hence removes over- and underexposure. [Mertens2007]

26. Quality measures Contrast Good contrast in the amplitude image indicates absence of flying pixels. [Mertens2007]

27. Quality measures Entropy The entropy is a measure for the amount of information. [Goshtasby2005]

28. Quality measures Surface σ = Gaussian blurred squared depth map μ = Gaussian blurred depth map A measure for surface smoothness which indicates less noise. [Malpica2009]

29. Quality measures Contrast Well exposedness Entropy Weight maps Surface

30. Process overview Depth maps Laplace pyramids Weight maps Fused pyramid Resulting fusion

31. Evaluation • Depth map quality • What is the reference?

32. Reference • We compared our results to single exposures with the integration time t’. • The integration time t’ is the global optimum found by the method from [May2006]. • We use N = 4 exposures (i = 0..N-1) with integration times .

33. Evaluation • Depth map quality • What is the reference? • What does quality mean? • Less temporal noise  Stability over time test

34. Stability over time Capture a static image over time and determine the standard deviation for each pixel. Overall reduction of mean standard deviation by 25%. Comparison F > S F < S F ≈ S Fused result (F) Single exposure (S) mm

35. Evaluation • Depth map quality • What is the reference? • What does quality mean? • Less temporal noise  Stability over time test • Better processing results  Run ICP and compute 3D reconstruction error

36. 3D reconstruction

37. Evaluation • Depth map quality • What does quality mean? • Less temporal noise  Stability over time test • Better processing results  Run ICP and compute 3D reconstruction error • Impact of each quality measure • Computation times

38. Computation

39. Evaluation • Depth map quality • What does quality mean? • Less temporal noise  Stability over time test • Better processing results  Run ICP and compute 3D reconstruction error • Impact of each quality measure • Computation times • Precision  Plane fit in planar regions

40. We captured test scenes with planar regions. Planar regions

41. We ran our fusion algorithm ... Planar regions Depth maps Laplace pyramids Weight maps Fused pyramid Resulting fusion

42. Planar regions Contrast Well expo. Entropy Weight maps Surface ...and compared the fusion results of all combinations of quality measures.

43. Planar regions We identified planar regions...

44. Planar regions ...and fitted planes into the 3D data.

45. Planar regions

46. Planar regions 1 cm Surface Only contrast does work alone

47. Planar regions Surface Combinations are valuable Combinations are valuable

48. Planar regions Surface All four quality measures lead to the smallest error

49. Conclusion A new method for time-of-flight imaging: • + No calibration • + Real-time capable • + Reduced error

50. Thank you • Acknowledgements • Martin Profittlich (PMDTec) for TOF camera support. • Bernd Bickel and Tim Weyrich for helpful discussions.