Temporally Coherent Completion of Dynamic Shapes

1. Temporally Coherent Completion of Dynamic Shapes AUTHORS:HAO LI,LINJIE LUO,DANIEL VLASIC PIETER PEERS,JOVAN POPOVIC,MARK PAULY,SZYMON RUSINKIEWICZ Presenter:Zoomin(Zhuming) Hao

2. Previous Work 1.Based on Template How to obtain the template? ①a separate rigid reconstruction step (e.g., [Li et al. 2008; de Aguiar et al. 2008; Vlasic et al. 2008])

3. Previous Work Robust Single-View Geometry and Motion Reconstruction[Li et al. 2009]

4. Previous Work 1.Based on Template How to obtain the template? ①a separate rigid reconstruction step (e.g., [Li et al. 2008; de Aguiar et al. 2008; Vlasic et al. 2008]) ②globally aggregating all surface samples through time (e.g., [Wand et al. 2009; Mitra et al. 2007; S¨ußmuth et al. 2008])

5. Previous Work Efficient Reconstruction of Nonrigid Shape and Motion fromReal-Time 3D Scanner Data [Wand et al. 2009] input ==> a sequence of point clouds sampled at different time instances automatically assembles them into a common shape that best fits all of the input data a deformation field is computed that approximates the motion of this shape to match all the data frames limitations：occurs if objects disappear in an acquisition hole and come out in a very different pose

6. Previous Work 1.Based on Template How to obtain the template? ①a separate rigid reconstruction step (e.g., [Li et al. 2008; de Aguiar et al. 2008; Vlasic et al. 2008]) ②globally aggregating all surface samples through time (e.g., [Wand et al. 2009; Mitra et al. 2007; S¨ußmuth et al. 2008]) Disadvantage？ fix the topology geometric details are limited to those in the template

7. Previous Work 2.Based on the assumption: Dynamic performance consists of rigid parts [Pakelny and Gotsman2008] manual segmentation,an optimal rigid motion is computed for each part [Chang and Zwicker 2009] limits to subjects that exhibit articulated motion [Zheng et al.2010] automatically extract a consensus skeleton to derive a consistent temporal topology

8. Previous Work Articulated Mesh Animation from Multi-view Silhouettes [Vlasic et al. 2008] Consensus skeleton for nonrigid space-time registration [Zheng et al.2010] input==>a sequence of point clouds acquired over time extract per-frame skeletons consolidate theminto a skeleton structure (consistent across time andaccounts for all the frames) Limitations：It assumes that the underlying shape is clearly articulatedwhich is not always the case for subjects wearing loose clothing

9. System Overview

10. Framework -- 1Pairwise Correspondences Coarse-scale Correspondences： non-rigid ICP algorithm[Li et al.2009]

11. Framework -- 1Pairwise Correspondences Fine-scale Correspondences： Improvement based on two observations: 1.far-away points can bias the local alignment(local-support) 2.stability of ICP matching algorithm depends on the local geometry Three-step Algorithm provided by this paper: 1.Sampling 2.Matching: non-rigid locally weighted ICP algorithm[Brown and Rusinkiewicz 2007] employ a CSRBF for point selection near feature point 3.Warping

12. Framework -- 1Shape Accumulation fi'(merged) and fi+1(original)==>Corrsepondences warp merge from first frame to the last frame from last frame to the first frame fi+1‘ interleaved registration/merging scheme in a forward&backward fashion

13. System Overview

14. Framework -- 2Hole Filling visual Hull prior [Vlasic et al. 2009] + weighted Poisson surface reconstruction [Kazhdan et al. 2006] Surface Fairing: Minimizing bending energy of the patch’s vertices using bi-Laplacian [Botsch and Sorkine 2008]

15. System Overview

16. Framework -- 3Temporal Filtering • Warp two neighboring frames to current frame based on the pairwise correspondences • Combine them using Poisson reconstruction with different weight for different region Poisson reconstruction warp to warp to

17. System Overview

18. Framework -- 4Detail Resynthesis 1. Resynthesize high frequency detail [Nehab et al.2005] 2. Acquire normal maps [Vlasic et al.2009]

19. Conclusions Contribution: • A framework to automatically fill holes with temporal coherent patches without relying on a geometrical template. • some little improvements on previous algorithms Limitation: • Topology of our meshes will always match the(changing and sometimes incorrect)topology of the visual hull.(Ideally, we need to extract a single consistent topology) 2. Temporal correspondences are valid between nearby frames only 3. each frame should cover most part of the object surface ( limited to multi-view scans),the unobserved regions have no geometric details in them. Future work: • To take physical properties into account

20. Thank You!