Ai-Mei Huang And Truong Nguyen Image processing, 2006 IEEE international conference on

1. Motion vector processing based on residual energy information formotion compensated frame interpolation Ai-Mei Huang And Truong Nguyen Image processing, 2006 IEEE international conference on 

2. introduction • Low bandwidth requirements, i.e. video telephony • by skipping frames • low frame rate video usually results in motion jerkiness • Motion-compensated frame interpolation (MCFI) • Directly uses the received MVs • Suffer from annoying artifacts such as blockiness and ghost effect. • Motion re-estimation is not suited for mobile devices Video coding/compression Video in encoder Bit stream Reconstructed frame decoder

3. Motion compensated frame interpolation • In MCFI methods, a skipped frame is often bi-directionally interpolated • from its two neighboring reconstructed frames . • by using the received MVs of the second frame. MV is not always reliable!

4. The correlation between motion vectorreliability and residual energy. • Macroblock(MB) contain areas of different motion. • Encoder favors the MV that can represent most of the region • The prediction residual will be generated and encoded. • Correct those unreliable MVs for frame interpolation • Avoid using those unreliable MVs to correct other MVs.

5. The proposed method • Based on residual energy associated with each motion vector • iterative process, stops when the MAP is no longer changed, or reaches a pre-defined maximum iteration, max_ite.

6. (1)motion vector classification based on residual energy (base unit) 16*16 MB=>four 8*8 blocks • Classify MVs to 3 Groups. • Calculate every 8x8 block’s residual energy Em,n • By taking the sum of the absolute value of each reconstructed prediction error for each pixel. • Em,n<threshold  MV will be classified as reliable and place into first group G1. • Em,n>=threshold  MV will be classified as unreliable and place into first group G3. • Unreliable MV’s neighboring MVs within the same MB will be classified as possibly unreliable into the second group G2

7. (2) motion vector correction • Works on those unreliable MVs Correcting from neighbor reliable MV • A residual-energy constrained median filter (RECMF) is used in this process and defined as • S contains the neighboring MVs centered at v*m,n. • Select a new MV only from its neighboring reliable MVs. • vm,n itself will be excluded from the candidates. • we prevent unreliable MVs to be used to correct other unreliable MVs.

8. (3)motion vector similarity check • Ensure v*m,n is not identical or similar to vm,n • Remain unreliable • Use angle variance by taking inner product of the two vectors. • < threshold => two MVs is similar ,fail in the check => bm,n still in G3 • Pass in the check => bm,n take in G1

9. (4)motion vector re-sampling and smoothing • In [7], each 8 x 8 block is further broken into four 4 x4 sub-blocks. • The MVs of these four sub-blocks can be obtained simultaneously by minimizing a smoothness measure , which is defined in the following. • For example: [7] G. Dane and T. Q. Nguyen, "Smooth motion vector resampling for standard compatible video post-processing," Proc. Asilomar Conf: Signals, Systems and Computers, 2004.

11. SIMULATIONS • two video sequences, FOREMAN and SILENT, • CIF frame resolution • 30 frame per second (fps). • both encoded using H.263 but even frames are skipped by the encoder. • fix quantization parameter (QP) values. • Avg. bitrate • FOREMAN is 240.3 Kbps • SILENT is184.59 Kbps