1 / 23

Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting

Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting. Chunbo Zhu , Xiaoyan Sun , Feng Wu , and Houqiang Li. ICME 2008. Outline. Introduction Framework Previous work Encoder Decoder Experiment Result Conclusion. Introduction. Texture synthesis

clementine-giles
Download Presentation

Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Video Coding with Spatio-temporal Texture Synthesis and Edge-based inpainting Chunbo Zhu , Xiaoyan Sun , Feng Wu , and Houqiang Li ICME 2008

  2. Outline • Introduction • Framework • Previous work • Encoder • Decoder • Experiment Result • Conclusion

  3. Introduction • Texture synthesis • Image/video inpainting

  4. Introduction Removed blocks Edge map Original image Encoder Decoder Structure propagation Texture synthesis JPEG [1] D. Liu, X. Sun, F. Wu, S. Li, and Y.-Q. Zhang, “Image compression with edge-based inpainting,” IEEE Transactions on Circuits and Systems for Video Technology, Oct. 2007.

  5. Introduction • Texture regions can be well synthesized. • Selectively removed during encoding. • Restored by texture synthesis and edge-based inpainting in the decoder. • Assistant information can be extracted to guide restoration. • Region types, edge information, motion parameters. Keep spatio-temporal consistency

  6. Framework—previous work … I B B B P [2] C. Zhu, X. Sun, F. Wu, and H. Li, “Video coding with spatio-temporal texture synthesis,” IEEE International Conference on Multimedia and Expo, 2007.

  7. Framework—encoder Same as previous work Local Motion Region … I B B B P Global Motion Region

  8. Framework—encoder

  9. Block Categorization Edge detection Block categorization Structure blocks Exemplar selection Texture blocks examplars Check connective degree removed blocks do not constitute a large region

  10. Framework—encoder

  11. Motion Threading • Spatial and temporal consistency • Block-based backward motion estimation • Motion threads in texture regions are pruned B2 B3 B4 B1 B5 B0 [3] L. Luo, F. Wu, S. Li, Z. Xiong, and Z.Q. Zhuang, “Advanced motion threading for 3D wavelet video coding,” Signal Processing: Image Communication, Vol. 19, Issue 7, pp. 601-616, Aug. 2004.

  12. Exemplar Selection • All the motion threads are 3-D exemplar candidates • Spatio-temporal variation(SVT) Choose the threads with higher variation N : length of a thread which consists of N blocks Bi(i=1…N) :spatio-temporal 6-neighboring blocks of each block : variance of a block E() : mean pixel value of a block Output a sequence of binary masks indicating which blocks are removed

  13. Framework—encoder

  14. Sprite Generation • GMR • pixel motion can be described by a parametric geometrical model • Sprite(background mosaic) • A sprite for a GOP (x, y) and (x’,y’) are coordinates of pixels in original frame and reference image [4] Y. Lu, F. Wu, S. Li, and Y.-Q. Zhang, “Efficient background video coding with static sprite generation and arbitrary-shape spatial prediction techniques,” IEEE Transactions on Circuits and Systems for Video Technology, May 2003.

  15. Framework—encoder

  16. Region removal in GMR • Mapping into original frame inverse warping • if one 8x8 block in an original frame contains more than half of the removable pixels, it is selected as a removable block and then skipped when encoding. Otherwise, it is selected as an exemplar and coded with H.264 encoder. removable blocks unremovable blocks Region Classification Mapping Exemplar Selection Edge Detection RS SPT RF RS: Removable blocks in sprite RF: Removable blocks in original frame

  17. Framework—decoder Mask1: region map Mask2: removing map

  18. Decoder—Spatio-temporal texture synthesis in LMR • Texture synthesis module • Arbitrary-shaped regions • Perform frame by frame • Square patch • Confidence map • Similarity of candidate patch and target patch Squared difference of known pixels

  19. Framework—decoder

  20. Decoder—Edge-based inpainting in GMR GMR GMR parameter Warping Blending SPTH Edge inpainting SPT edges Texture Synthesis SPT

  21. Experimental Results • Implemented into JM 10.2 of H.264 • YUV 4:2:0 sequence format with CIF resolution(352x288) • rate distortion optimization (RDO) and CABAC were turned on

  22. Experimental Results

  23. Conclusion • In this paper, we present a video coding scheme in which some regions are removed in the encoder and restored in the decoder by spatio-temporal texture synthesis and edge-based image inpainting. • To keep temporal consistency, different types of motion have been considered in region removal and restoration for both textural and structural regions.

More Related