1 / 45

Active Lighting for Appearance Decomposition

Active Lighting for Appearance Decomposition. Todd Zickler DEAS, Harvard University. I = f ( shape,. reflectance ). illumination,. ?. f -1 ( I ) =. Appearance. Research Overview. COLOR IMAGE FILTERING. 3D RECONSTRUCTION. APPEARANCE CAPTURE. PHOTOMETRIC INVARIANTS. ?.

rafael
Download Presentation

Active Lighting for Appearance Decomposition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Active Lighting for Appearance Decomposition Todd Zickler DEAS, Harvard University

  2. I = f (shape, reflectance) illumination, ? f -1( I ) = Appearance Appearance Decomposition

  3. Research Overview COLOR IMAGE FILTERING 3D RECONSTRUCTION APPEARANCE CAPTURE PHOTOMETRIC INVARIANTS Appearance Decomposition

  4. ? f -1( I ) = I = f (shape, reflectance, illumination) Getting 3D Shape:Image-based Reconstruction Appearance Decomposition

  5. Reflectance: BRDF Bi-directional Reflectance Distribution Function Appearance Decomposition

  6. LAMBERTIAN: IDEALLY DIFFUSE Conventional 3D Reconstruction:Restrictive Assumptions Appearance Decomposition

  7. Example: Conventional Stereo Il Ir ASSUMPTION: Il = Ir Appearance Decomposition

  8. Example: Conventional Stereo Il Ir ASSUMPTION: Il = Ir Appearance Decomposition

  9. Conventional 3D Reconstruction:Restrictive Assumptions Variational Stereo [Faugeras and Keriven, 1998] Shape from shading [Tsai and Shaw, 1994] Multiple-window stereo [Fusiello et al., 1997] Space Carving [Kutulakos and Seitz, 1998] Appearance Decomposition

  10. Reflectance: BRDF Appearance Decomposition

  11. Reflectance: BRDF Appearance Decomposition

  12. Helmholtz Reciprocity [Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977] Appearance Decomposition

  13. Stereo vs. Helmholtz Stereo STEREO HELMHOLTZ STEREO Appearance Decomposition

  14. Stereo vs. Helmholtz Stereo STEREO HELMHOLTZ STEREO Appearance Decomposition

  15. Stereo vs. Helmholtz Stereo STEREO HELMHOLTZ STEREO Appearance Decomposition

  16. Il Ir • Specularities “fixed” to surface • Relation between Il and Ir independent of BRDF Reciprocal Images Appearance Decomposition

  17. p ^ ^ vr vr ^ ^ n n ol or ^ ^ vl vl = Reciprocity Constraint p ol or Appearance Decomposition

  18. p p ^ ^ vr vr ^ ^ n n ol or ol or • Arbitrary reflectance • Surface normal ^ ^ vl vl = Reciprocity Constraint Appearance Decomposition

  19. Reciprocal Acquisition CAMERA LIGHT SOURCE Appearance Decomposition

  20. Recovered Normals [Zickler et al. 2002] Appearance Decomposition

  21. Recovered Surface [Zickler et al., ECCV 2002] Appearance Decomposition

  22. In Practice • Arbitrary Reflectance • Off-the-shelf components • Direct surface normals • Images aligned with recovered shape • Self-calibrating (coming…) Appearance Decomposition

  23. Ongoing Work: Auto-calibration [Zickler et al., CVPR 2003, CVPR 2006,…] Appearance Decomposition

  24. Research Overview COLOR IMAGE FILTERING 3D RECONSTRUCTION APPEARANCE CAPTURE PHOTOMETRIC INVARIANTS Appearance Decomposition

  25. DIFFUSE SPECULAR = + Reflectance Decomposition [Phong 1975; Shafer, 1985] Appearance Decomposition

  26. Reflectance Decomposition [Shafer, 1985] Appearance Decomposition

  27. = + LAMBERTIAN: IDEALLY DIFFUSE Reflectance Decomposition: Simplifies the Vision Problem = + Appearance Decomposition

  28. Reflectance Decomposition: A Difficult Inverse Problem DIFFUSE SPECULAR = + = + [Bajscy et al., 1996; Criminisi et al., 2005; Lee and Bajscy, 1992; Lin et al., 2002; Lin and Shum, 2001; Miyazaki et al., 2003; Nayar et al., 1997; Ragheb and Hancock, 2001; Sato and Ikeutchi, 1994; Tan and Ikeutchi, 2005; Wolfe and Boult, 1991,…] Appearance Decomposition

  29. Known Illuminant: Still Ill-posed B S IRGB D? G R Appearance Decomposition

  30. Known Illuminant: Still Ill-posed B S IRGB D? G R Appearance Decomposition

  31. Observation:Explicit Decomposition not Required B S IRGB r1 G r2 J • INVARIANT TOSPECULAR REFLECTIONS • BEHAVES ‘LAMBERTIAN’ R Appearance Decomposition

  32. B S IRGB r1 G r2 J R Observation:Explicit Decomposition not Required IRGB || J || [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

  33. Generalization: Mixed Illumination SINGLE ILLUMINANT MIXED ILLUMINATION B B S1 S S2 IRGB IRGB r1 G G r2 J r1 J R R [Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006] Appearance Decomposition

  34. Generalization: Mixed Illumination Appearance Decomposition

  35. Example: Binocular Stereo Conventional Grayscale(R+G+B)/3 Specular Invariant, ||J||(blue illuminant) Specular Invariant, ||J||(blue & yellow illuminants) One image from input stereo pair Recovered depth [Algorithm: Boykov, Veksler and Zabih, CVPR 1998] Appearance Decomposition

  36. Example: Optical Flow Conventional Grayscale(R-+G+B)/3 Specular Invariant, ||J||(blue illuminant) Specular Invariant, ||J||(blue & yellow illuminants) Ground truth flow [Algorithm: Black and Anandan, 1993] Appearance Decomposition

  37. Example: Photometric Stereo J behaves ‘Lambertian’  Linear function of surface normal [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

  38. Example: Photometric Stereo J behaves ‘Lambertian’  Linear function of surface normal [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

  39. Example: Photometric Stereo J behaves ‘Lambertian’  Linear function of surface normal [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

  40. Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

  41. Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

  42. ψ Generalized Hue B S IRGB r1 G r2 J R Appearance Decomposition

  43. Example: Material-based Segmentation Conventional Grayscale Specular Invariant ||J|| Input image Generalized Hue y Conventional Hue [Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006] Appearance Decomposition

  44. Active Lighting for Image-guided Surgery? Endoscopic imagery: • Illuminant(s) is/are controlled and known • Non-Lambertian surfaces • Lack of texture Active lighting can provide: • Precise shape (surface normals) for a broad class of (non-Lambertian) surfaces • Specular and/or shading invariance (e.g., optical flow, tracking, segmentation) • Minimal hardware requirements Appearance Decomposition

  45. Acknowledgements Satya Mallick, UCSD Peter Belhumeur, Columbia University David Kriegman, UCSD Sebastian Enrique, Columbia University Ravi Ramamoorthi, Columbia University zickler@eecs.harvard.edu http://www.eecs.harvard.edu/~zickler Appearance Decomposition

More Related