Bryan Willimon , Steven Hickson, Ian Walker, and Stan Birchfield Clemson University

1. An Energy Minimization Approach to 3D Non-Rigid Deformable Surface Estimation Using RGBD Data Bryan Willimon, Steven Hickson, Ian Walker, and Stan Birchfield Clemson University IROS 2012 - Vilamoura, Portugal

2. Overview • We propose an algorithm that uses energy minimization to estimate the current configuration of a highly non-rigid object. • Our approach relies on a 3D nonlinear energy minimization framework to solve for the configuration using a semi-implicit scheme adapted from Fua and colleagues (Pilet et al. 2005).

3. Previous Work on Pose Estimation for Robotics • Elbrechter et al. (IROS 2011) use a soft-body-physics model with visual tracking to manipulate a piece of paper. • Bersch et al. (IROS 2011) describe a method to bring a T-shirt into a desired configuration by alternately grasping the item with two hands, using a fold detection algorithm. Both approaches require predefined fiducial markers.

4. Energy Minimization Approach • The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms: • Smoothness term • Correspondence term • Depth term • Boundary term

5. Energy Minimization Approach • The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms:

6. Energy Minimization Approach • Mesh Initialization

7. Energy Minimization Approach • Smoothness term • Correspondence term • Depth term • Boundary term

8. Energy Minimization Approach • Smoothness term

9. Energy Minimization Approach • Correspondence term

10. Energy Minimization Approach • Depth term Front View Top View

11. Energy Minimization Approach • Boundary term Without Boundary With Boundary

12. Energy Minimization Approach • Minimize energy equation

13. Experimental Results • We captured RGBD video sequences of shirts and posters to test our proposed method’s ability to handle different non-rigid objects in a variety of scenarios. • Four experiments were conducted: • Illustrating the contribution of the depth term • Illustrating the contribution of the boundary term • Partial self-occlusion • Textureless shirt sequence

14. Experimental Results • Illustrating the contribution of the depth term

15. Experimental Results • Illustrating the contribution of the boundary term

16. Experimental Results • Partial self-occlusion

17. Experimental Results • Textureless shirt sequence

18. Experimental Results Video

19. Conclusion • We have presented an algorithm to estimate the 3D configuration of a highly non-rigid object through a video sequence using feature point correspondence, depth, and boundary information. • We plan to extend this research to handle a two-sided 3D triangular mesh that covers both the front and the back of the object.

20. Questions?