Modeling as Structural Variation

1. Hao (Richard) Zhang Simon Fraser University (SFU), Canada Modeling as Structural Variation Course: Structure-Aware Shape Processing

2. Modeling: 3D content creation Inspiration? Inspiration a readily usable digital 3D model

3. Inspiration = real-world data Realistic reconstruction [Nan et al., SIGGRAPH 2010]

4. Creation of novel 3D shapes Inspiration = design, sketch, photo, other examples Focus on creative modeling High demand in VFX, games, simulation, VR, … sketch

5. 2D-to-3D: an ill-posed problem Shape from shading, sketch-based modeling, … Creation from scratch is hard: job for skilled artists 3D content creation is hard One of the most talked about at SIG’10 panel Main reason why graphics is not as ubiquitous as we wanted it to be  SIG’11 Award Talk

6. Models created are meant for subsequent use Desire to create readily usable 3D models Usable 3D content even harder

7. Models created are meant for subsequent use Desire to create readily usable 3D models Usability: higher-level info beyond low-level mesh Part or segmentation information Structural relations between parts Correspondence to relevant models, etc. Usable 3D content even harder Hard shape analysis problems!

8. Reuse existing 3D models and associated info Data- or model-driven approach: creation is driven by or based on existing (pre-analyzed) models Key: model reuse

9. Reuse existing 3D models and associated info Data- or model-driven approach: creation is driven by or based on existing (pre-analyzed) models Key: model reuse

10. New model created by varying existing model(s) Paradigm I: much like shape editing Modeling as variation Variationas modification of an existing model, e.g.,a warp or a deformation

11. New model created by varying existing model(s) Paradigm I: much like shape editing Paradigm II: shape synthesis Modeling as variation Variationas modification of an existing model, e.g.,a warp or a deformation • Variation by part composition, from multiple models

12. Available models are assumed to be structurally or functionally valid or proper New model should maintain that validity, but what is the essential structures to preserve? Conflicting goals = challenge = “fit & diverse”: Fit: structure preservation from existing models Diverse: encourage significant deviation from existing models Structure-aware but not too aware!

13. iWires: [Gal et al. 2009] Style transfer on part proportions: [Xu et al. 2010] Component-wise controllers: [Zheng et al. 2010] Photo-inspired modeling: [Xu et al. 2011] Structural retargeting: [Lin et al. 2011, Bokeloh et al. 2012, Bao et al. 2012, Yeh et al. 2013, Zhang et al. 2013] Modeling by deformation

14. Wires as control handles [Singh & Fiume 1999] Edits preserve structural relations among wires iWire: analyze-and-edit [Gal et al. 2009]

15. Cuboids and generalized cylinders as control handles Use the analyze-and-edit paradigm like iWires Edits preserve structural relations among controllers, mainly symmetry and proximity Component-wise controllers [Zheng et al. 2010]

16. Key diff: not editing; modeling inspired by a set Co-analysis results in style-content table (Peter talk) Style ( = part proportion) transfer Content Style 1 Style Style 2 Style 3 [Xu et al. 2010]

17. Shape creation: filling the style-content table Style transfer [Xu et al. 2010]

18. Not editing; modeling inspired by a single photograph Warp an existing 3D model to fit object silhouette Structure preservation ensures a coherent 3D model Photo-inspired modeling [Xu et al. 2011]

19. Photo-inspired modeling • Use the controllers from [Zheng et al. 2010] photo [Xu et al. 2011]

20. Photo-inspired modeling • Use the controllers from [Zheng et al. 2010] photo Retrieved candidate model [Xu et al. 2011]

21. Photo-inspired modeling • Use the controllers from [Zheng et al. 2010] Result of deformation to fit silhouette photo Retrieved candidate model [Xu et al. 2011]

22. Structure preservation at work • Use the controllers from [Zheng et al. 2010] by symmetry [Xu et al. 2011]

23. Structure preservation at work • Use the controllers from [Zheng et al. 2010] by symmetry by proximity [Xu et al. 2011]

24. Structure preservation at work • Use the controllers from [Zheng et al. 2010] by symmetry by proximity additional optimization [Xu et al. 2011]

25. Structure preservation at work • Use the controllers from [Zheng et al. 2010] by symmetry by proximity output additional optimization [Xu et al. 2011]

26. Key analysis: regularity detection and organization Repetition of 1D or 2D regular patterns is easy Retarget amid irregularity is hard Structural retargeting Key words: analyze-and-stretch; pattern repetition rather than geometric stretch!

27. [Lin et al. 2011]: Retarget irregular 3D facades (1D) sequence by sequence Hierarchical organization obtained manually [Bao et al. 2012]: Retarget 2D facades by user-specified constraints Retarget with semi-auto analysis [Lin et al. 2011]

28. Hierarchical decomposition of irregular 2D facades Optimal decomposition via SYMAX (earlier talk) Retarget facade by altering the generative model Automatic analysis [Zhang et al. 2013]

29. Modeling by example [Funkhouser et al. 2004] Probabilistic synthesis: [Kalogerakis et al. 2012] Fit & diverse: [Xu et al. 2012] Structure recovery by part assembly: [Shen et al. 2012] Replacing functional sub-strucures: [Zheng et al. 2013] Modeling by part composition

30. New models composed by parts retrieved from an existing data repository Key: retrieve relevant parts by geometric similarity of parts Many variants to date Modeling by example [Funkhouser et al. 2004]

31. Learn a generative, probabilistic, and component-based structure from a pre-segmented set of shapes Observable variables include number of parts, descriptors for part geometry, adjacency relations, etc. Latent variables are the “style”, e.g., part proportion A new model is drawn from the prob. distribution Probabilistic synthesis [Kalogerakis et al. 2012]

32. Probabilistic synthesis Green models are from the training set; blue ones are synthesis results [Kalogerakis et al. 2012]

33. Different from previous works: instead of generating one model at a time, evolve sets of shapes together Inspired by the biological process of evolution Fit & diverse [Xu et al. 2012]

34. Off-springs by part mutation (warp) and cross-over (reassembly by fuzzy replaceability): leads to diversity A “design gallery”: user specifies “likes” or “dislikes”; defines fitness function Fit & diverse [Xu et al. 2012]

35. Modeling from single Kinect depth scan + RGB image Unlike [Xu et al. 2011], model is built by part assembly from multiple shapes, more versatile than just warp Structure recovery by part assembly [Shen et al. 2012]

36. Detect specific common sub-structures by sub-graph matching --- a step towards functionality analysis Replacement can cross different object categories Replacing functional sub-structures [Zheng et al. 2013]

37. Deformation applied mainly for editing or to model by fitting to something, e.g., a photo or sketch Part combination mainly for creating novel shapes More structural changes with part combination Part combination can alter topology, in a discrete way Deformation vs. part combination

38. Too much “fit” and not enough “diverse”: We are still doing a lot more “fit”, e.g., structure preservation, then creating truly diverse structures What is ultimately important is whether a new model is functionally valid/proper; we have yet been able to define it A low-level yet practical problem: “stitching” when recombining parts, is largely overlooked More later … Now and beyond