Projective Analysis for 3D Shape Segmentation

1. Projective Analysis for 3D Shape Segmentation Yunhai Wang1MinglunGong1,2Tianhua Wang1,3Hao (Richard) Zhang 4 Daniel Cohen-Or 5Baoquan Chen1,6 1Shenzhen Institutes of Advanced Technology 3Jilin University 2 Memorial University of Newfoundland 4 Simon Fraser University 5 Tel-Aviv University 6 Shandong University

2. Segmentation of 3D shapes • One of the most fundamental tasks in shape analysis • Low-level cues (minimal rule; convexity) alone insufficient

3. Knowledge-driven approach Unsupervised co-analysis [Sidi et al. 2011] Learning segmentation [Kalograkis et al. 10] Keys to success: amount & quality of labelled or unlabelled 3Ddata Joint segmentation [Huang et al. 2011] Active co-analysis [Wang et al. 2012]

4. 3D data challenge: amount • How many 3D models of strollers, golf carts, gazebos, …? • Not enough 3D models = insufficient knowledge • Labeling 3D shapes is also a non-trivial task 380labeled meshes over 19object categories

5. Many many more images About14 million images across almost 22,000object categories Labeling images is quite a bit easier than labeling 3D shapes

6. 3D data challenge: quality Self-intersecting; non-manifold Incomplete Real-world 3D models (e.g., those from Tremble Warehouse) are often imperfect

7. Projective shape analysis (PSA) • Treat a 3D shape as a set of projected binary images • Label these images by learning from vast amount of image data • Then propagate the image labels to the 3D shape • Alleviate various data artifacts in 3D, e.g., self-intersections

8. Contributions • Joint image-shape analysis via projective analysis for semantic 3D segmentation • Utilize vast amount of available image data • Allowing us to analyze imperfect 3D shapes

9. Contributions • Bi-class Symmetric Hausdorff distance = BiSH • Designed for matching 1D binary images • More sensitive to topology changes (holes) • Caters to our needs: part-aware label transfer

10. Related works onshape-image hybrid processing Many works on 2D-3D fusion, e.g., for reconstruction [Li et al.11] Image-guided 3D modeling [Xuet al.11]

11. Related works onprojective shape analysis Image-space simplification error [Lindstrom and Turk 10] Light field descriptor for 3D shape retrieval [Chen et al.03] We deal with the higher-level and more delicate task of semantic 3D segmentation

12. Outline • PSA for 3D shape segmentation • Results and conclusion • Region-based binary shape matching

13. Data preparation: image labeling Labeling involves GrabCut and some user assistance

14. Projecting input 3D shape • Assume all objects are upright oriented; they mostly are! • Project an input 3D shape from multiple pre-set viewpoints

15. Retrieve labeled images • For each projection of the input 3D shape, retrieve top matches from the set of labelled images

16. Select projections for label transfer • Select top (non-adjacent) projectionswith the smallest average matching costs for label transfer

17. Label transfer Later … • Label transfer is done per corresponding horizontal slabs • Pixel correspondence straightforward

18. Confidence map • Three terms based on image-level, slab-level, and pixel-level similarity: more similar = higher confidence • Label transfer is weighted by a confidence value per pixel

19. Backprojection (2D-to-3D) • One primitive projects to multiple pixels in multiple images • Per-pixel confidence gathered over multiple retrieved images • Probabilistic map over input 3D shape: computed by integrating per-pixel confidence values over each shape primitive

20. Graph cuts optimization • Final labeling of 3D shape: multi-label alpha expansion graph cuts based on the probabilistic map

21. Outline • PSA for 3D shape segmentation • Results and conclusion • Region-based binary shape matching

22. The matching problem … … Projections of input 3D shape Database of (labeled) images • Goal: find shapes most suitable for label transfer and FAST! • Not a global visual similarity based retrieval • Want part-aware label transfer but cannot reliably segment Classical descriptors, e.g., shape context, interior distance shape context (IDSC), GIST, Zenike moments, Fourier descriptors, etc., do not quite fulfill our needs • Possibly complex topology (lots of holes), not just a contour • All upright orientated: to be exploited • Characteristics of the data to be matched

23. Scan-line view Takes advantage of upright orientation

24. Scan-lines to slabs Classical choice for distance: symmetric Hausdorff (SH) But not sensitive to topology changes; not part-aware • Cluster scan-lines into smaller number of slabs --- efficiency! • Hierarchical clustering by a distance between adjacent slabs

25. Extend SH to consider both B and W! C SH(C,B)=2, SH(Cc, Bc)=2 B A SH(A,B)=2, SH(Ac, Bc)=10 B SH for only one class may not be topology-sensitive A bi-class SH distance is!

26. Bi-class symmetric Hausdorff (BiSH) C SH(C,B)=2, SH(Cc, Bc)=2 BiSH(C,B) = 2 B A SH(A,B)=2, SH(Ac, Bc)=10 BiSH(A,B) = 10 B

27. BiSH vs. SH BiSH is more part-aware: new slabs near part boundaries SH BiSH

28. Piecewise linear warping • Slabs are scaled/warped vertically for better alignment • Another measure to encourage part-aware label transfer Warp Recolor Slabs of labeled image warped to better align with slabs in projected image Slabs recolored: many-to-oneslab matching possible

29. Slab matching and image similarity • Dissimilarity between slabs: BiSH scaled by slab height • Slab matching allows linear warp: optimized by a dynamic time warping (DTW) algorithm • Dissimilarity between images: sum over slab dissimilarity after warped slab matching

30. Outline • PSA for 3D shape segmentation • Results and conclusion • Region-based binary shape matching

31. vs. learning mesh segmentation • Same inputs, training data (we project), and experimental setting • Models in [K 2010]: manifold, complete, no self-intersections • PSA allows us to handle any category and imperfect shapes

32. Data • All input 3D shapes tested have self-intersections as well as other data artifacts • 11 object categories; about 2600 labeled images

33. Complex topology Pavilion (465 pieces) Bicycle (704 pieces)

34. Incomplete model: point clouds

35. Timing • Number of selected projections: 5 – 10 • Number of retrieved images per projection: 2 • Matching two images (512 x 512) takes 0.06 seconds • Label transfer (2D-to-2D then to 3D): about 1 minute for a 20K-triangle mesh

36. Conclusions • Demonstrated potential in labeling 3D models: imperfect, complex topology, over any category • Projective shape analysis (PSA): semantic 3D segmentation by learning from labeled 2D images

37. Main advantages • Utilize the rich availability and ease of processing of photos for 3D shape analysis • No strong requirements on quality of 3D model

38. Limitations • Inherent to data-driven: knowledge has to be in data • Relying on spatial and not feature-space analysis • Assuming upright; not designed for articulated shapes • Inherent limitation of 2D projections: they do not fully capture 3D info

39. Future work • Additional cues from images and projections, e.g., color, depth, etc. • Apply PSA for other knowledge-driven analyses • Labeling 2D images is still tedious: unsupervised projective analysis

40. Thank you! More results and data can be found from http://web.siat.ac.cn/~yunhai/psa.html