Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair)

1. Sensing Highly Non-Rigid Objects with RGBD Sensors for Robotic Systems Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair) Ian Walker (co-advisor) Adam Hoover Damon Woodard

2. Robotic systems are divided into two groups • Industrial • Recycling • Mail • Food • Laundry • Domestic • Outdoor • Lawn care • Pool cleaning • Gutter cleaning • Window cleaning • Indoor • Floor cleaning • Pet assistance • Picking up objects • Laundry

3. Why Domestic Robots? • robotic systems are able to accomplish chores around the house • Domestic robotic systems are getting more attentionin the news • Domestic • Outdoor • Lawn care • Pool cleaning • Gutter cleaning • Window cleaning • Indoor • Floor cleaning • Pet assistance • Picking up objects • Laundry

4. Laundry is an Important Problem • Domestic • Outdoor • Lawn care • Pool cleaning • Gutter cleaning • Window cleaning • Indoor • Floor cleaning • Pet assistance • Picking up objects • L Laundry

5. Laundry is an Important Problem • Industries and research institutes are making attempts to solve the process Laundry NEDO Laundry Handling System PR2 at UC Berkeley

6. Difficulties in the Laundry Problem • What are current problems that make a laundry system difficult to automate? • Highly deformable objects • Infinitely large number of configurations • Lots of possible grasp points • The laundry problem is still in a research stage

7. Research Path • Clothing Classification • B. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011. • B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013. • B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013. • Unfolding Clothing • B. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011. • Pose Estimation • B. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012. • B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). • B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA2013. Focus of thesis

8. Previous Work on Classification for Robotics • Kita et al. use a humanoid system to recognize the state of clothes using a cloth model with 22 out of 27 attempts correctly classified • Cusumano-Towner et al. were aimed at classifying the category of seven articles with a success rate of 20 out of 30 trials Our system does not use predefined cloth models or dual manipulators

9. Classification Framework • Isolation • Graph-based Segmentation • Stereo Matching • Determining Grasp Point • Classification • Hanging Position • Binary Silhouettes • Visual-based shape and appearance information • Lying Position • Low level features • Characteristics • Selection Masks

10. Classification in a … Hanging Position Lying Position

11. Experimental Results • With 6 categories, 5 items per category, and 20 images per item, the dataset collected by the extraction / isolation procedure consists of 600 images • This dataset was labeled in a supervised learning manner so that the corresponding category of each image was known • Two experiments were conducted: • Extraction and isolation process • Interaction vs. Non-interaction

12. Experimental Results • Extraction and isolation process: • The image taken by one of the downward-facing stereo cameras • The result of graph-based segmentation • The object found along with its grasp point (red dot) • The image taken by the side-facing camera • The binary silhouettes of the front and side views of the isolated object.

13. Experimental Results • Interaction vs. Non-interaction: • The process of interacting with each article of clothing provided the system with multiple views using various grasp locations, allowing the system to collect 20 total images of each object. • Two articles were compared by examining the 400 match scores between their pairs of images (20 images per article).

14. Classification in a … Hanging Position Lying Position

15. Classification in a Lying Position

16. Experimental Results • The proposed L-C-S-H approach is applied to a laundry scenario. • Two different scenarios involved using: • 3 categories {shirts, dresses, socks} • 7 categories {shirts, dresses, socks, cloths, pants, shorts, jackets} • Each scenario is run through three experiments: • Baseline System (L-H) • L-C-H approach • L-C-S-H approach

17. Experimental Results • Scenario 1 → 3 categories {shirts, dresses, socks}

18. Experimental Results • Scenario 2 → 7 categories {shirts, dresses, socks, cloths, pants, shorts, jackets}

19. Research Path • Clothing Classification • B. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011. • B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013. • B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013. • Unfolding Clothing • B. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011. • Pose Estimation • B. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012. • B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). • B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA2013. Focus of thesis

20. Previous Work on Unfolding for Robotics • Cusano-Towner et al. were aimed at flattening a piece of crumpled clothing by implementing a disambiguation phase and a reconfiguration phase. Our system does not use a dual manipulator or a predefined model of the clothing

21. Model to Unfold Laundry into a Flat Canonical Position • First Phase • Remove any major wrinkles and / or folds • Pulling the cloth at individual corners every d degrees • Second Phase • Define cloth model • Calculate various components needed for the cloth model

22. Experimental Results • The proposed approach was applied to a 3D simulated cloth to determine the results of the first and second phase. • Two experiments were conducted: • Experimental Test of Algorithm • Test to Fully Flatten the Cloth

23. Experimental Results • Experimental Test of Algorithm : • This experiment tested the first phase of the proposed algorithm and monitored the process from eight iterations of pulling the cloth. • The models continually change configurations in a manner that flattens and unfolds larger areas of the cloth as the iterations increase. • Eventually, the cloth is mostly flattened out to a more recognizable shape in the final iteration.

24. Experimental Results • Test to Fully Flatten the Cloth : • This experiment tested the proposed algorithm in determining if this approach would completely flatten a piece of clothing. • The test used the first and second phase of the algorithm to grasp the cloth at various locations and moved the cloth at various orientations until the cloth obtained a flattened percentage greater than 95%.

25. Research Path • Clothing Classification • B. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011. • B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013. • B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013. • Unfolding Clothing • B. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011. • Pose Estimation • B. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012. • B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). • B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA2013. Focus of thesis

26. Previous Work on Pose Estimation for Robotics • Elbrechter et al. (IROS 2011) use a soft-body-physics model with visual tracking to manipulate a piece of paper. • Bersch et al. (IROS 2011) describe a method to bring a T-shirt into a desired configuration by alternately grasping the item with two hands, using a fold detection algorithm. Both approaches require predefined fiducial markers.

27. Energy Minimization Approach • The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms: • Smoothness term • Correspondence term • Depth term • Boundary term

28. Energy Minimization Approach • The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms:

29. Energy Minimization Approach • Smoothness term • Correspondence term • Depth term • Boundary term

30. Energy Minimization Approach • Smoothness term

31. Energy Minimization Approach • Smoothness term

32. Energy Minimization Approach • Correspondence term

33. Energy Minimization Approach • Depth term Front View Top View

34. Energy Minimization Approach • Boundary term

35. Energy Minimization Approach • Minimize energy equation

36. Segmentation and Initialization • Foreground / background segmentation

37. Segmentation and Initialization • Mesh Model Generator

38. Segmentation and Initialization • Reinitialization

39. Experimental Results • We captured RGBD video sequences of shirts, pants, shorts, and posters to test our proposed method’s ability to handle different non-rigid objects in a variety of scenarios. • Four experiments were conducted: • Estimating pose of clothing • Estimating pose of posters • Reinitializing mesh after in-plane rotation • Quantitative Results

40. Shirt moving out-of-plane • Shirt moving in-plane • Shirt partially occluded Experimental Results • Shirt moving in-plane and out-of-plane • Estimating pose of clothing • 7 shirts • 1 pair of pants • 2 pairs of shorts • Shirt changing scale • Shirt translating from side to side • Shirt moving in-plane • Pair of pants moving out-of-plane • Shorts moving out-of-plane • Shorts moving out-of-plane

41. Experimental Results • Estimating pose of posters • Poster with little texture moving out-of-plane • Poster with a lot of texture moving out-of-plane

42. Experimental Results • Reinitializing mesh after in-plane rotation

43. Experimental Results • Quantitative Results

44. Conclusion • Clothing Classification • Extraction / Isolation • A novel approach in which a pile of laundry is sifted by an autonomous robot system in order to separate each item. • Hanging position • Using interaction to provide multiple views of an object and capture more visual data • The results show that, on average, classification rates using robot interaction are 59% higher than those that do not use interaction. • Lying Position • Multi-layer approach involving a mixture of global and local features • Characteristics and selection masks achieve, on average, an improvement of • 27.47% for three categories • 17.90% for four categories • 10.35% for seven categories

45. Conclusion • Unfolding Clothing • An approach to interactive perception in which a piece of laundry is flattened out into a canonical position by pulling at various locations of the cloth. • The algorithm is shown to flatten a simulated cloth by 95.57% of its total area • Pose Estimation • A new and novel algorithm that estimates the 3D configuration of a deformable object through an RGBD video sequence • An energy model is used to create a non-linear energy function and the information is computed using a semi-implicit scheme • Energy terms • Smoothness • Feature point correspondence • Depth • Boundary

46. References • Copies of: • Publications • Code • Datasets • Videos • are located at www.bryanwillimon.com