1 / 41

Learning 3D mesh segmentation and labeling

Learning 3D mesh segmentation and labeling. Head. Torso. Upper arm. Lower arm. Hand. Upper leg. Lower leg. Foot. Evangelos Kalogerakis, Aaron Hertzmann, Karan Singh University of Toronto. Goal: mesh segmentation and labeling. Labeled Mesh. Input Mesh. Head. Neck. Torso. Leg. Tail.

lucerom
Download Presentation

Learning 3D mesh segmentation and labeling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Learning 3D mesh segmentation and labeling Head Torso Upper arm Lower arm Hand Upper leg Lower leg Foot Evangelos Kalogerakis, Aaron Hertzmann, Karan Singh University of Toronto

  2. Goal: mesh segmentation and labeling Labeled Mesh Input Mesh Head Neck Torso Leg Tail Ear Training Meshes

  3. Related work: mesh segmentation [Mangan and Whitaker 1999, Shlafman et al. 2002, Katz and Tal 2003, Liu and Zhang 2004, Katz et al. 2005, Simari et al. 2006, Attene et al. 2006, Lin et al. 2007, Kraevoy et al. 2007, Pekelny and Gotsman 2008, Golovinskiy and Funkhouser 2008, Li et al. 2008, Lai et al. 2008, Lavoue and Wolf 2008, Huang et al. 2009, Shapira et al. 2010] Surveys: [Attene et al. 2006, Shamir 2008, Chen et al. 2009]

  4. Related work: mesh segmentation Shape Diameter [Shapira et al. 10] Randomized Cuts [Golovinskiy and Funkhouser 08] Random Walks [Lai et al. 08] Normalized Cuts [Golovinskiy and Funkhouser 08]

  5. Is human-level segmentation even possible without higher-level cues? [X. Chen et al. SIGGRAPH 09]

  6. Is human-level segmentation even possible without higher-level cues? [X. Chen et al. SIGGRAPH 09]

  7. Image segmentation and labeling [Konishi and Yuille 00, Duygulu et al. 02, He et al. 04, Kumar and Hebert 03, Anguelov et al. 05, Tu et al.05, Schnitman et al. 06, Lim and Suter 07, Munoz et al. 08,…] Textonboost [Shotton et al. ECCV 06]

  8. Related work: mesh segmentation & labeling Consistent segmentation of 3D meshes [Golovinskiy and Funkhouser 09] Multi-objective segmentation and labeling [Simari et al. 09]

  9. Learning mesh segmentation and labeling Learn from examples Significantly better results than state-of-the-art No manual parameter tuning Can learn different styles of segmentation Several applications of part labeling

  10. Labeling problem statement Head c2 Neck c3 c1 Torso Leg c4 Tail Ear C= { head,neck,torso,leg,tail,ear}

  11. Conditional Random Field for Labeling Head Neck Torso Leg Tail Ear Labeled Mesh Input Mesh Unary term

  12. Conditional Random Field for Labeling Head Neck Torso Leg Tail Ear Labeled Mesh Input Mesh Face features

  13. Conditional Random Field for Labeling Head Neck Torso Leg Tail Ear Labeled Mesh Input Mesh Face Area

  14. Conditional Random Field for Labeling Head Neck Torso Leg Tail Ear Labeled Mesh Input Mesh Pairwise Term

  15. Conditional Random Field for Labeling Head Neck Torso Leg Tail Ear Labeled Mesh Input Mesh Edge Features

  16. Conditional Random Field for Labeling Head Neck Torso Leg Tail Ear Labeled Mesh Input Mesh Edge Length

  17. Conditional Random Field for Labeling Head Neck Torso Leg Tail Ear Labeled Mesh Input Mesh Unary term

  18. Feature vector surface curvature singular values from PCA shape diameter distances from medial surface average geodesic distances shape contexts spin images contextual label features x

  19. Learning a classifier Head x2 Neck Torso Leg Tail Ear x1

  20. Learning a classifier We use the Jointboost classifier [Torralba et al. 2007] Head x2 Neck Torso ? Leg Tail Ear x1

  21. Unary term

  22. Unary Term Most-likely labels Classifier entropy

  23. Our approach Head Neck Torso Leg Tail Ear Labeled Mesh Input Mesh Pairwise Term

  24. Pairwise Term Geometry-dependent term

  25. Pairwise Term Label compatibility term Head Neck Ear Torso Leg Tail Head Neck Ear Torso Leg Tail

  26. Full CRF result Head Neck Torso Leg Tail Unary term classifier Full CRF result Ear

  27. Learning Learn unary classifier and G(y) with Joint Boosting [Torralba et al. 2007] Hold-out validation for the rest of parameters

  28. Dataset used in experiments We label 380 meshes from the Princeton Segmentation Benchmark Each of the 19 categories is treated separately [Chen et al. 2009] Antenna Head Thorax Leg Abdomen

  29. Quantitative Evaluation Labeling • 6% error by surface area • No previous automatic method Segmentation • Our result: 9.5% Rand Index error • State-of-the art: 16%[Golovinskiy and Funkhouser 08] • With 6 training meshes: 12% • With 3 training meshes: 15%

  30. Labeling results

  31. Segmentation Comparisons Shape Diameter [Shapira et al. 10] Randomized Cuts [Golovinskiy and Funkhouser 08] Our approach

  32. Segmentation Comparisons Shape Diameter [Shapira et al. 10] Randomized Cuts [Golovinskiy and Funkhouser 08] Our approach

  33. Learning different segmentation styles Head Neck Torso Leg Tail Ear Training Meshes Test Meshes Head Front Torso Middle Torso Back Torso Front Leg Back Leg Tail

  34. Generalization to different categories Head Wing Body Tail Head Neck Torso Leg

  35. Failure cases Face Hair Torso Handle Neck Leg Nose Cup

  36. Limitations Adjacent segments with the same label are merged Head Torso Upper arm Lower arm Hand Upper leg Lower leg Foot

  37. Limitations Results depend on having sufficient training data Handle Cup Top Spout 19 training meshes 3 training meshes

  38. Limitations Many features are sensitive to topology Head Torso Upper arm Lower arm Hand Upper leg Lower leg Foot

  39. Applications: Character Texturing, Rigging Ear Head Torso Back Upper arm Lower arm Hand Upper leg Lower leg Foot Tail

  40. Summary • Use prior knowledge for 3D mesh segmentation and labeling • Based on a Conditional Random Field model • Parameters are learned from examples • Applicable to a broad range of meshes • Significant improvements over the state-of-the-art

  41. Thank you! Acknowledgements: Xiaobai Chen, Aleksey Golovinskiy, Thomas Funkhouser,Szymon Rusinkiewicz , Olga Veksler,Daniela Giorgi, AIM@SHAPE, David Fleet, Olga Vesselova, John Hancock Our project web page: http://www.dgp.toronto.edu/~kalo/papers/LabelMeshes/

More Related