Vision & Graphics Research in UCSD CSE

1. Vision & Graphics Researchin UCSD CSE David Kriegman Computer Science & Engineering University of California, San Diego

2. The Pixel Lab Serge Belongie Henrik Wann Jensen David Kriegman Vincent Rabaud Andrew Rabinovich John Rapp Geoffrey RomerSteve Rotenberg Josh Wills Arash KeshmirianTobias KlugAnders Wang Kristensen Kuang-chih Lee Jongwoo Lim Satya Mallick Ben Ochoa Sameer Agarwal Kristin Branson Piotr Dollar Craig DonnerJeff Ho Wojciech Jarosz

3. Computer Vision Research Computer Vision Research Threads • Segmentation • Recognition • Reconstruction • Motion Applications Theory

4. A failed outline I. Theoretical Contributions: A. Object tracking B. Measurement and segmentation of motion C. Unsupervised learning (clustering) of objects from images D. Object recognition and categorization • Helmholtz reciprocity stereopsis • Structure from motion G. Illumination and reflectance modeling H. Optical flow through refractive objects II. Vision meets Graphics A. Refractive Optical flow for video compositing B. Image-based rendering C. Efficient rendering using environment maps D. Modeling of rough dielectric surfaces E. Texture synthesis on surfaces III. Applications of Vision A. Face recognition in images and video B. Person tracking for video monitoring and ubiquitous vision systems C. Visual monitoring of animal health and welfare (Smart Vivarium) D. Tissue microarray analysis for cancer detection. E. Protein structure reconstruction from cryo-electron micrographs

5. Face Recognition The challenge caused by lighting variability Same Person or Different People

7. Same Person or Different People

9. Same Person or Different People

11. The Illumination Cone Illumination Cone x x x 1 2 n Theorem::The set of images of any object in fixed posed, but under all lighting conditions, is a convex cone in the image space. (Belhumeur and Kriegman, IJCV, 98) 2-light source image Single light source images lie on cone boundary N-dimensional Image Space

12. Face Detection: First Online Step

13. 3-D Face Modeling for Recognition Training Images 3-D Generative Model [Georghiades, Belhumeur, Kriegman]

14. Face Database 0-12o 12-25o 25-50o 50-77o 64 Lighting Conditions 9 Poses => 576 Images per Person Variable lighting

15. Face Recognition Error Rates Our Method

16. What Went Where? Segmenting images into regions with different motion

17. What Went Where: Motion Segmentation [J. Wills, S. Agarwal, S. Belongie ]

18. Motion Segmentation • Compute point correspondences by comparing filter response vectors • Partition these correspondences into groups using RANSAC and estimate the represented motion layers • Densely assign pixels to one of the detected motion layers using a Markov Random Field method [J. Wills, S. Agarwal, S. Belongie ]

19. Helmholtz Stereo Reconstructing shape of surfaces with arbitrary BRDF

20. Helmholtz Stereo: Arbitrary BRDF ^ n (in,in) LEFT RIGHT (out,out) Non-Matte surfaces: A challenge for conventional stereo Bi-directional Reflectance Distribution Function (in, in ; out, out) [ Zickler, Kriegman, Belhumeur ]

21. Helmholtz Reciprocity & Stereo in, in ^ ^ n n out, out (in, in ; out, out) = (out, out ; in, in) out, out in, in [Helmholtz, 1910], [Minnaert, 1941], [ Nicodemus et al, 1977]

22. Using Multiple Helmholtz Stereo Pairs     il1wl1T–ir1wr1T il2wl2T–ir2wr2T ^ ^ n n = 0   wl1 wr1 … } A • Multiple views (at least three pairs) yield a matrix constraint equation. • For correct match, matrix A must be Rank 2. • Surface normal lies in null space of A [ Zickler, Kriegman, Belhumeur ]

23. Helmholtz Stereo [ Zickler, Kriegman, Belhumeur ]

24. Experimental Aparatus [ Zickler, Kriegman, Belhumeur ]

25. Subtle Surface Geometry: Wrinkles

26. Helmholtz Stereo: Depth & Normals

27. Reconstructed Geometry [ Zickler, Kriegman, Belhumeur ]

28. Comparison to other methods [ Zickler, Kriegman, Belhumeur ]

29. II. Vision meets Graphics A. Refractive optical flow for compositing. B. Modeling of rough dielectric surfaces C. Efficient rendering using environment maps D. Image-based rendering E. Texture synthesis on surfaces

30. Refractive Optical Flow We use motion of the background to recover how light rays are transformed from the background to the foreground as they travel through a refractive medium. [ Agarawal, Mallick, Belongie, Kriegman ]

31. The Setup Theory Experiment [ Agarawal, Mallick, Belongie, Kriegman ]

32. Results Real Recovered [ Agarawal, Mallick, Belongie, Kriegman ]

33. Refractive Optical Flow Movies With Water Without Water [ Zickler, Kriegman, Belhumeur ]

34. Henrik’s Slides

35. Rendering with Environment Maps [ Agarwal, Jensen, Belongie ]

36. Area Importance [ Agarwal, Jensen, Belongie ]

37. Illumination Importance [ Agarwal, Jensen, Belongie ]

38. Structured Importance Sampling • A novel importance metric that combines area and illumination importance. • A new hierarchical stratification algorithm. [Hochbaum Shmoys] • Fast • Stable • Quality guarantees. [ Agarwal, Jensen, Belongie ]

39. Lambertian BRDF Structured Importance Area LightGen Illumination [ Agarwal, Jensen, Belongie ]

40. Glossy BRDF Illumination importance sampling 10 100 300 1 Structured importance sampling [ Agarwal, Jensen, Belongie ]

41. Texture Synthesis on Surfaces Shape Model Texture Sample [ Magda, Kriegman ]

42. Making it fast Fast methods for selecting Texture Triangles Blending to hide seams Textons Run Time: 1-2 seconds Bunny Model: 8192 faces 1.3Ghz Athlon [ Magda, Kriegman ]

48. Image-based modeling & Rendering Goal: To render, under arbitrary pose & lighting images of both natural and man-made objects. Rendered Images

49. Model objects with many images

50. Real Objects in Synthetic Scenes