Quad-Tree Motion Modeling with Leaf Merging

1. Quad-Tree Motion Modeling with Leaf Merging Reji Mathew and David S. Taubman CSVT 2010

2. Outline • Introduction • Quad-tree representation • Quad-tree motion modeling • Motion vector prediction strategies • Pruning algorithm • Merging principle • Motion signaling • R-D performance results • Hierarchical and polynomial motion modeling • Scalable motion modeling • Conclusion

3. Quad-tree Representation • Image modeling • Image to be recursively divided into smaller regions, each region represented by a suitable model. • Sub-optimal: dependency between neighboring leaf nodes with different parents is not exploited

4. Quad-tree Representation • Image modeling • Rate-distortion optimization, allowing a Lagrangian cost function(D+λR) to be minimized using tree pruning with leaf merging step. [1] R. Shukla, P. Dragotti, M. Do, and M. Vetterli, “Rate-distortion optimized tree structure compression algorithms for piecewise polynomial images,” IEEE Trans. Image Process., vol. 14, no. 3, pp. 343–359, Mar. 2005.

5. Quad-tree Motion Modeling • Motion model • forward-only, backward only or bi-directional motion with two reference frames. • Motion vector prediction strategies • Hierarchical motion coding • H.264 spatial motion vector prediction strategy Motion models

6. Quad-tree Motion Modeling • Pruning Algorithm • Produce a quad-tree structure that minimizes the Lagrangian cost objective Df + λRf • Given a parent node p, the four children ci , 1 ≤ i ≤ 4, are pruned away if • When pruning occurs, andOtherwise, and =Rp in hierarchical coding =0 at all times in spatial coding R-D optimally pruned quad-tree: Tree pruning yields a globally minimal value for Df + λRffor hierarchical coding; while it is somewhat greedy for spatially predictive coding.

7. Quad-tree Motion Modeling • Merging principle • possibility of jointly coding and optimizing neighboring nodes that belong to different parents. • Merge target contains nieghboring node located at a higher level or at the same level. • Merging is allowed to take place only if it reduces the overall Lagrangian cost. The same parent

8. Quad-tree Motion Modeling • Motion signaling • Anchor node: • Hierarchical: the only member node of the region that is not signaled as being merged • Spatial: the first node in the region that is encountered during decoding.(the top-left block)

9. Quad-tree Motion Modeling • R-D performance results 25% 35% 35% 45% once merging is included the performance of hierarchical motion representation can be brought close to that achieved by spatial prediction with merging.

10. Hierarchical and Polynomial Motion Modeling • Further improve the performance of hierarchical motion representation by polynomial motion models. • Formation of larger regions during merging process • Smoother motion representations • Motion models • The parameters of the motion model are obtained by a weighted least squares fitting procedure. • Pruning phase • Merging phase : mv belonging to node b at level k : motion corresponding to translation, linear and affine flows

11. Hierarchical and Polynomial Motion Modeling • Motion compensation • Generate a set of MVs for each descendants at level K (4*4 block) • R-D performancewith motion models depend on the motion model and the central location of block b’

12. Scalability Motion Modeling • Scalability objective • Modified Lagrangian cost function • When terminating decoding at an intermediate resolution level, motion compensation is performed using leaf nodes that may already be available; in those cases where leaf nodes are not available, information contained in branch nodes is utilized. : The total distortion of all nodes for which motion compensation is performed Level k : : The costs for each level k of the quad-tree : The weights assigned to each level, and terminate

13. Scalability Motion Modeling • Scalability performance • α0 =α1=α2=0.1, α3=0.7

14. Scalability Motion Modeling • Residual coding • JPEG2000: full resolution motion compensated residual frames • Total rate for coding motion and residual frames

15. Scalability Motion Modeling • Wavelet-based video encoding results • integrate the quad-tree motion model with the wavelet-based scalable interactive video (SIV) codec[9] [9] A. Secker and D. S. Taubman, “Lifting-based invertible motion adaptive transform framework for highly scalable video compression,” IEEE Trans. Image Process., vol. 12, no. 12, pp. 1530–1542, Dec. 2003.

16. Conclusion • The merging step can be incorporated into quad-tree motion representationsfor a range of motion modeling contexts. • R-D performance that can be gained by introducing merging for the two cases of hierarchical and spatially predictive motion coding (such as that employed by H.264). • Report on the benefits of polynomial modeling and hierarchical coding, once merging has been incorporated into the conventional quad-tree approach.