Applications of Light Polarization in Vision Lecture #18

1. Applications of Light Polarization in Vision Lecture #18 Thanks to Yoav Schechner et al, Nayar et al, Larry Wolff, Ikeuchi et al

2. Michael Oprescu,www.photo.net Separating Reflected and Transmitted Scenes Reconstructing Shape of Transparent Objects Removing Specularities Removing Haze and Underwater Scattering Effects

3. Separation of Diffuse and Specular Reflections Diffuse surfaces : No (or minimal) Polarization All light depolarized due to many random scattering events inside object. Specular Surfaces: Strong Polarization (even though partially polarized) Smooth/Rough Surfaces: The degree of polarization decreases with roughness.

4. Active Illumination • Completely remove specular reflections using polarized light • when the filters are 90 degrees apart. • Commonly used in industrial settings.

5. Passive Illumination • Most illumination from sources (sun, sky, lamps) is unpolarized. • Merely using a polarizer will not remove specular reflections completely.

6. I_min is not equal to I_d (diffuse component)

7. polarizer o 180 Polarization vector determination 3 general measurements suffice camera Polarization Measurements

8. Determining the Polarization Cosine Curve Using Vector Notation: Three measurements suffice to determine the cosine curve.

9. Degree of Polarization • Varies between 0 and 1. • If zero, then there is no polarization  Only diffuse component present. • If one, only specular component present. • If degree of polarization does not change as polarizer is rotated, • then there is no guarantee that specular component is completely removed • (I_sc may still be present).

10. Fresnel Ratio • I_sc and I_sv depend on refractive • index and angle of incidence. • I_sc and I_sv are related to fresnel • coefficients: is fresnel coefficient perpendicular to plane of incidence is fresnel coefficient parallel to plane of incidence

11. Fresnel Ratio Metals Dielectrics Brewster angle • Hard to separate diffuse and specular parts for metals. • Easier for dielectrics (good for non-normal incidences).

12. Dichromatic Model for Removing Specularities Completely • Specularities are only reduced in intensity using polarization. • They are removed completely only for the Brewster angle of incidence. • Nayar et al. use additional color constraints in dichromatic model • to remove reflections completely. • Assume a local patch where the highlight and • its surrounding area have the same diffuse component.

13. Michael Oprescu,www.photo.net Semi-Reflections • Both Reflected and Transmitted light are polarized. • But they are polarized differently. • They depend on the orientation of the transparent layer. • Reflections are removed completely only at • Brewster Angle of Incidence.

14. window Transparent Layers camera

15. Semi-Reflections transmitted reflected scene scene camera window

16. r transmitted reflected scene scene polarizer 0.8 0.6 r Optical coding 0.4 camera r r window 0.2 80 20 40 60 Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99

17. r t transmitted reflected scene scene polarizer t=1-r 0.8 0.6 t r Optical coding 0.4 camera r r window 0.2 80 20 40 60 Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99

18. reflected transmitted observed scene scene image I I r t window Experiment Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99

19. I observed reflected transmitted image scene scene I I t r [ ] I = r + t window I I / 2 t r Optical coding Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99

20. I reflected transmitted observed scene scene image I I r t window I t r + [ ] = I I / 2 r t Optical coding Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99

21. 2 unknowns: , Solve for I I o Window at 27 _ _ Reflected r 2 r 2 2 2 _ _ r r r r _ I I I I T T R R Transmitted 2 r 2 r _ _ I I r r r r _ [ ] I = r + t I r t + [ ] = I I I I / / 2 2 r t r t Digital decoding Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99 2 Linear equations

22. Transmitted Reflected I I T R negative crosstalk positive crosstalk o o o 37 27 17 0.4 The inclination of an invisible surface 0 _ _ 0.8 0.4 correlation j 10 20 30 40 50

23. Imaging through Haze clear day moderate haze very hazy www.hazecam.net Previous work Recover: Pure image processing Grewe & Brooks ’98, Kopeika ’98 Oakley & Satherley ’98 • Object + haze layers • Scene structure Polarization filtering • Info about the aerosols Physics based Shurcliff & Ballard ’64 Nayar & Narasimhan ’99 Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

24. direct Airlight transmission A T object radiance R scattering camera Imaging through Haze Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

25. direct Airlight transmission T A object radiance R scattering • z is a function of (x,y) • Multiplicative & additive models • Color 1 1 - similar dependence camera 0 0 z z

26. Polarization and Haze A A A A A A A A A A + _ direct transmission polarizer Plane of rays determines airlight components polarized p=1 =0 Airlight degree of polarization A unpolarized p=0 = > camera Along the line of sight, polarization state is distance invariant @ all orientations Assume: The object is unpolarized Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

27. Trivial case Life is tough… I I Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar … still, there is a dominant polarization

28. I = T/2 + A Experiment Best polarized image Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

29. I = T/2 + A Experiment Worst polarized image Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

30. airlight A camera transmission object T radiance R 2 input images: Recovery = + T / 2 A I = + ( )/p T / 2 - I I + - - ( I I ) ( I I ) / p - b z depth = - e 1 radiance = R A - βz e ¥ - transmission βz = T R e airlight - = æ ö βz A A 1 - e ç ÷ ¥ è ø _ A A for known polarization degree º p p , A A A + ¥ Model A I

31. saturated airlight A ¥ airlight polarization p Recovery = + T / 2 A I = + ( )/p T / 2 - I I + - - ( I I ) ( I I ) / p - b z depth = - e 1 radiance = R A - βz e ¥ - βz = T R e - = æ ö βz A A 1 - e ç ÷ ¥ è ø A A for known º p p , A A A + ¥ Model camera 2 input images: A I transmission airlight _ polarization degree

32. I Best polarized image Dehazing Experiment Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

33. R Dehazed image Dehazing Experiment Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

34. log component images Airlight saturation polarization p ( ) - I I - b z = - e 1 pA ¥ Range map depth

35. I Dehazing Experiment Best polarized image Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

36. R Dehazed image Dehazing Experiment Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

37. Range map Instant Dehazing: Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

38. veiling lightB • zis a function of (x,y) 1 1 • Color + - object with blur 0 0 z z signal S

39. Lythgoe & Hemmings, 1967 (Nature) : “Many invertebrates are able to distinguish the plane of polarized light. Does this enable them to see further underwater?” Lythgoe, 1972 (Handbook Sensory Physiol) : “…there is a strong possibility that it [polarization] could be useful for improving the visibility of distant objects, especially under water.” Hypothesis, 4 Decades Old

40. Lythgoe, 1972 (Handbook Sensory Physiol) : “…there is a strong possibility that it [polarization] could be useful for improving the visibility of distant objects, especially under water.” Hypothesis, 4 Decades Old Lythgoe & Hemmings, 1967 (Nature) : “Many invertebrates are able to distinguish the plane of polarized light. Does this enable them to see further underwater?” “…when the [polarizing] screen was oriented to exclude the maximum spacelight … fishes stood out in greater contrast against their background.” “… simple polarizing screen will be less versatile than the system found in Octopus, where there is the intra-ocular ability to distinguish light polarized in one plane from that polarized in another.”

41. B B • Veiling light is partially polarized Polarization of Veiling Light Y. Schechner & N. Karpel, polarization-based recovery

42. 24 Image Components veiling signal lightB S L scattering Veiling light = Spacelight = Path radiance = Backscatter Schechner, Karpel, underwater vision

43. Rough surfaces : naturally depolarize • Specular reflection : weaker than in air • Multiple scattering • Signal decreases with distance / veiling-light increases At large distance: signal polarization has a negligible effect (Supported by Shashar, Sabbah & Cronin 2004) Signal Polarization Y. Schechner & N. Karpel, polarization-based recovery

44. 25 Polarization Photography

45. min max max max min min max min I I I I I I I I Past Polarization-Based Methods Raw images Degree of polarization Polarization-difference imaging

46. camera Recovery Model 2input images:

47. camera Recovery 2input images:

48. Aqua-polaricam Y. Schechner & N. Karpel, underwater imaging

49. Experiments

50. min I Experiment Eilat, 26m underwater Best polarization image Y. Schechner & N. Karpel, underwater imaging