Structure Recovery by Part Assembly

1. Structure Recovery by Part Assembly Chao-Hui Shen1Hongbo Fu2 Kang Chen1 Shi-Min Hu1 1Tsinghua University 2City University of Hong Kong

2. Background • Consumer level scanning devices • Capture both RGB and depth • Reconstruction is challenging • Low resolution • Noise • Missing data • …

3. Example-based Scan Completion • Global-to-local and top-down [Kraevoyand Sheffer2005; Paulyet al. 2005] • Rely on the availability of suitable template model • However … shape retrieval No suitable model!

4. Assembly-based 3D Modeling • Data-drive suggestion and interaction [Chaudhuri and Koltun 2010; Chaudhuri et al. 2011] • Retrieve suitable parts to match user intent • Aim to support open-ended 3D modeling • Quite different goal from ours • Automatic shape synthesis by part composition [Kalogerakis et al. 2012; Jain et al. 2012; Xu et al. 2012] • Result in database that grows exponentially • Significantly enlarge the existing database • But make storage and retrieval challenging

5. Our solution: Recover the Structure by Part Assembly • Structure recovery instead of geometry reconstruction • Do NOT prepare a large database • Retrieve and assemble suitable parts on the fly

6. Problem Setup Session: Acquiring and Synthesizing Indoor Scenes An Interactive Approach to Semantic Modeling of Indoor Scenes with an RGBD Camera [Shao et al. 2012] A Search-Classify Approach for Cluttered Indoor Scene Understanding [Nan et al. 2012] Acquiring 3D Indoor Environments with Variability and Repetition [Kim et al. 2012] Goal: Recover high-level structure Assembly close to geometry Point cloud + Image (Single view) …… …… Pre-segmented Repository Models (Parts + Labels) Input Output

7. Observations • Directly searching is computationally prohibitive • Need a quick way to explore meaningful structures guided by: • Spatial layout of the parts in the repository models • Acquired data

8. Observations • Complementary characteristics of point cloud & image 3D, more accurate cues for geometry & structure Lack depth information Capture the complete object Incomplete and noisy

9. Algorithm Overview …… Candidate Parts Selection Structure Composition Part Conjoining

10. Algorithm Overview …… Candidate Parts Selection Structure Composition Part Conjoining

11. Candidate Parts Selection • Goal: select a small set of candidates for each category • Achieved by retrieving parts that fit well some regions

12. Straightforward Solution • Search for the best-fit parts over the entire domain • Disregards the semantics associated with each part and the interaction between different parts Unlikely to produce good results! X X X X X X X X

13. Key Fact • Man-made objects lie in a low dimensional space • Defined with respect to the relative sizes and positions of shape parts [Ovsjanikov et al. 2011] • Employ 3D repository model as a global context • Globally align the models with the input scan first Search in a 3D offset window around the part

14. Part Matching Scheme Geometric fidelity score edge map 3D 2D (part contour) Geometric contribution score Total matching score (2D field) 3D offset window

15. Candidate Parts • Select top K parts with highest score for each category …… Seat …… Back …… Arm …… Front leg …… …… …… …… …… …… ……

16. Algorithm Overview …… Candidate Parts Selection Structure Composition Part Conjoining

17. Structure Composition • Goal: compose the underlying structure by identifying a subset of candidate parts

18. Constraints for Promising Compositions    having high score no isolated parts minimized intersection    Geometric fidelity Proximity Overlap

19. Search and Evaluate • Search for promising compositions under constraints • Globally Evaluate the compositions …… optimal composition total geometry contribution average geometry fidelity of parts total geometry fidelity

20. Algorithm Overview …… Candidate Parts Selection Structure Composition Part Conjoining

21. Part Conjoining • Problem: the parts are loosely placed together • Goal: generate a consistent & complete model

22. Identification of Contact Points • Refer to their parent models [Jain et al. 2012]

23. Matching of Contact Points • Greedily match nearby contact points • Generate auxiliary contact points when necessary auxiliary contact points

24. Global Optimization • Adjust the sizes {} and positions {} of parts • Make matched point as close as possible • Contact enforcement • Shape preserving • Global optimization j i identity scale transformed contact points

25. Results: Chairs • 70 repository models, 11 part categories

26. Results: Tables • 61 repository models, 4 part categories

27. Results: Bicycles • 38 repository models, 9 part categories

28. Results: Airplanes • 70 repository models, 6 part categories

29. Results: Creating New Structures

30. Results: Impact of Dataset Randomly picking some repository models input data

31. Summary • A bottom-up structure recovery approach • Effectively reuse limited repository models • Automatically compose new structure • Handle single-view inputs by the Kinect system • Future work • Multi-view inputs • Include style/functional constraints • Recover Indoor scenes

32. Thank you! Project Page: http://cg.cs.tsinghua.edu.cn/StructureRecovery