Amit Agrawal, Ramesh Raskar and Rama Chellappa Presented by: Ramesh Raskar * University of Maryland, MD, USA

1. What is the Range of Surface Reconstructions from a Gradient Field? Amit Agrawal, Ramesh Raskar and Rama Chellappa Presented by: Ramesh Raskar* University of Maryland, MD, USA * Mitsubishi Electric Research Labs (MERL), MA, USA Source Code: http://www.cfar.umd.edu/~aagrawal

2. Overview • Goal: Reconstruct height field from a gradient field • Challenges • Handle noise and outliers • Tradeoff: Preserve features or smooth • Provide knobs for control ? X Gradient Y Gradient Height Field

3. Choices of Reconstructions Noisy Gradients Height Field Least Sq Solution Robust Solution MSE = 10.81 MSE = 2.26

4. A Range of Reconstructions 1. Generalized Solution Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Continuous weights Binary weights Affine Transformation Scaling 2. Linear transformation of gradient vectors

5. Source of Gradients: Image Manipulation 2D Integration Grad X New Grad X Gradient Processing New Grad Y Grad Y Forward Difference Modified Gradients(Non-integrableGradients) Image

6. X Gradient Y Gradient Source of Gradients: Photometric Stereo Mozart Images under varying illumination (Non-integrable Gradients) 2D Integration Robust Solution MSE=2339.2 MSE=373.72

7. Gradient Field Integration • Photometric Stereo • Phase unwrapping, Mesh smoothing • Retinex (Horn’74) • HDR compression (Fattal’02) • Poisson image editing (Perez’04) • Poisson matting (Sun’04) • Image fusion (Raskar’04), Multi-flash Camera (Raskar’04) • Image Stitching (Levin’04) • Flash/No-flash (Agrawal’05) • ..

8. Space of Solutions Non-Integrable Gradient Field Robust solutions Residual Field Least Squares solution Subspace of Integrable Gradient Fields Space of All Gradient Fields

9. Example Application: Photometric Stereo • How to obtain heights from gradient field? • Preserve features during integration • Handle noise and outliers Z p q ? X Gradient Y Gradient Height Field

10. Integrability • Gradient field of solution Z should be integrable • Curl(Zx,Zy) (loop integrals) should be zero • Noisy estimated gradient field is non-integrable • Curl(p,q) ≠ 0

11. Common approaches to handle Integrability • Least Square: • Minimize error between • Estimated gradients (p,q) and Gradients of solution Z • [Horn et al. IJCV’90], [Simchony et al. PAMI’90] • Solution: Poisson equation Divergence Laplacian

12. Integrability: Reconstruction using Basis Functions • Project non-integrable gradients onto basis functions • Fourier: Frankot-Chellappa (PAMI’88) • Cosine: Georghiades (PAMI’01) • Redundant basis: Kovesi (ICCV’05)

13. Limitations of Least Squares Approaches • Lack of robustness • Smooth solution rather than feature preserving • Cannot handle outliers Height Field Least Sq Solution Robust Solution MSE = 10.81 MSE = 2.26

14. Key Ideas • All gradients are not required for integration • Replace gradients by functions of gradients

15. Approach • Transform input and output gradients • Poisson equation Unknown (Output) Gradients of Height Field Z Estimated(Input) Gradients

16. Approach • Transform input and output gradients • Poisson equation • New Generalized Equation Functions of Output Gradients Functions of Input Gradients

17. Approach • Transform input and output gradients • Poisson equation • New Generalized Equation • Knobs for control: functions f1, f2, f3, f4 fi

18. Least Squares Poisson Solution Isotropic Anisotropic Alpha-Surface M-estimator, Regularization Diffusion Poisson Solver Continuous weights Binary weights Affine Transformation Scaling No Change f1(Zx,Zy) f2(Zx,Zy) f3(p,q) f4(p,q) Poisson solver Zx Zy p q

19. A Range of Solutions by Transforming Gradients Anisotropic Isotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Continuous weights Binary weights Affine Transformation Scaling fi Linear Transformation of Gradient Vectors using fi

20. A Range of Solutions by Transforming Gradients Anisotropic Isotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Continuous weights Binary weights Affine Transformation Scaling fi f1(Zx,Zy) f2(Zx,Zy) f3(p,q) f4(p,q) Poisson solver Zx Zy p q 1. -surface bxZx byZy bxp byq 2. M-estimators wxZx wyZy wxp wyq 3. Regularization wxZx wyZy p q 4. Diffusion d11Zx+d12Zy d12Zx+d22Zy d11p+d12q d12p+d22q

21. Anisotropic Isotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Continuous weights Binary weights Affine Transformation Scaling A Range of Solutions 1. Robust Estimation by ignoring outliers in gradients

22. 1. -Surface: Binary Weights • Classifying gradients as inliers/outliers • Based on tolerance  • Graph Analogy • 2D grid as a planar graph • Nodes correspond to height values • Edges correspond to gradient values • Given edges compute nodes • [Agrawal et al ICCV 2005]

23. All Gradients are not required • Minimal set is the spanning tree of the graph • All nodes can be reached via spanning tree • N2 -1 edges in spanning tree for N2 nodes • Dimensionality of gradient field ~= 2N2 • Dimensionality of solution space = N2 - 1

24.  - Surface • Start with minimum spanning tree • Minimal set of (N2-1) edges • Iterate • Integrate height field using (inlier) edges • Compute gradient edges of the solution • Find inlier edges using tolerance 

25.  Tolerance d d< d

26. Choice of   = 0Minimum Spanning TreeUnique SolutionRobust to outliers  >>Overconstrained: All GradientsLeast Squares SolutionSmooth

27. Anisotropic Isotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Continuous weights Binary weights Affine Transformation Scaling A Range of Solutions 2. Robust Estimation by weighting gradients

28. 2. Continuous Weights Solution • M-estimators • Formulated as iterative re-weighted least squares • wx, wy are weights applied to gradients Least squares Huber Influence function

29. 3. Regularization • Add edge-preserving term using function Ф • Solved iteratively • Estimate weights wx, wy using Z • Update Z using weights • We use

30. Anisotropic Isotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Continuous weights Binary weights AffineTransformation Scaling A Range of Solutions 4. Robust Estimation by affine transformation of gradients

31. 4. Affine transformation of Gradient Field • General Transform of Gradients Vectors • Matrix D2x2 is a field of tensors • Estimated using the input gradient field (p,q) • Similar to edge-preserving diffusion tensor • Image Restoration [Weickert’96]

32. 4. Affine transformation of Gradient Field • Transformed gradient vectors • Non-iterative, sparse linear system f1(Zx,Zy) f2(Zx,Zy) f3(p,q) f4(p,q) d11Zx+d12Zy d12Zx+d22Zy d11p+d12q d12p+d22q

33. Mozart: Sharp Features MSE=219.72 MSE=2339.2 MSE=1316.6 MSE=359.12 MSE=806.85 MSE=373.72

34. Vase: Smooth Object MSE=294.5 MSE=239.6 MSE=22.2 MSE=164.98 MSE=15.14 MSE=2.78

35. Flowerpot

36. Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Continuous weights Binary weights Affine Transformation Scaling Generalized Solution using Gradient Vector Transformation Ground Truth Least Squares Feature Preserving

37. Isotropic Anisotropic Poisson Solver Alpha-Surface M-estimator, Regularization Diffusion Continuous weights Binary weights Affine Transformation Scaling Reconstruction from Gradients • Gradient Field integration as robust estimation • Generalized equation • Isotropic case: Least-square solution • Anisotropic transforms: Range of feature preserving solutions • Knobs for meaningful control • Future Directions • Automatic choice of anisotropic function • Nth order error functions • Using control/seed points information Source Code: http://www.cfar.umd.edu/~aagrawal

38. Extra Slides

39. Toy Example

40. H W Space of All solutions • For a H*W grid, all solutions lie in a subspace of dimension HW-1 • Equations • W-1 loops for each row, H-1 loops for each column • Total number of independent equations • N = (H-1)(W-1) = HW – H –W + 1 • Unknowns • W-1 x gradients in H rows, H-1 y gradients in W columns • Total unknowns • M = (W-1)*H + (H-1)*W = 2HW – H - W

41. Linear System • Stack all equation to form Ax = b • b contains negative of the curl values • x contains all the residual gradient values (pε,qε) • A is a sparse matrix with each row having 4 non-zero elements • Under-constrained system. A has null space of dimension M-N = H*W-1 • xp = particular solution Axp = b • Xh = homogeneous solution Axh= 0

42. p q Images X Gradient Y Gradient Example Application: Photometric Stereo • Multiple images, varying illumination • Obtain surface gradient field from images • Lambertian reflectance model

43. Mozart: Sharp Features MSE=219.72 MSE=2339.2 MSE=1316.6 MSE=359.12 MSE=806.85 MSE=373.72

44. Vase: Smooth Object MSE=294.5 MSE=239.6 MSE=22.2 MSE=164.98 MSE=15.14 MSE=2.78

45. Flowerpot

46. Choice of   = 0Minimum Spanning TreeUnique SolutionRobust to outliers  >>Overconstrained: All GradientsLeast Squares SolutionSmooth Toy Example

47. Choosing spanning tree • Assign weights to edges based on gradients • Find minimum spanning tree (MST) • Compute  based on variance

48. Face Input Images Estimated Heights with -tolerance

49. 3. Regularization • Add edge-preserving term using function Ф • Solved iteratively • Estimate weights wx, wy using Z • Update Z using weights • We use