Enhanced Motion Compensated Frame Interpolation Using Object Layer Inference

1. Enhanced Motion Compensated FrameInterpolation Using Object Layer Inference T.-S. Wang, K.-S. Choi, H.-S. Jang and S.-J. Ko Electronics Letters Sponsored by Institution of Engineering and Technology IET Journal

2. Outline • Introduction • Problem Statement • Proposed MCI Method • Experimental Results • Conclusions

3. Introduction • In various MCFI techniques, even though reliable motion is estimated in ME, the interpolated frame can be degradedby annoying artifacts(blocky ,flicker). • This is mainly because MCFI can utilizewrongmotion information around the occlusion region and the motion ambiguity (MA) region.

4. In [4], MV having the lowest SAD is selected to solve MA on the assumption that the FG object produces the lowest SAD. In [5], the median of pixel values associated with the multiple MV candidates is exploited for the interpolated pixel value. Problem Statement • In [4][5](Wang’s & Haan’smethods ), conventional methods produce the annoying artifacts since they cannot actually discriminate the FG and BG layers. • We utilize the occlusion estimation method [6] to identify the occlusion and to remove the inaccurate MVs caused by the occlusion. [4] Wang, D.: ‘Comparison of motion-compensated algorithms for frame interpolation’, Opt. Eng., 2003, 42, pp. 586–590 [5] Haan, G.D., and Ojo, O.A.: ‘Robust motion-compensated video up-conversion’, IEEE Trans. Consume. Electron., 1997, 43, pp. 1045–1056 [6] Ince, S., and Konrad, J.: ‘Geometry-based estimation of occlusion from video frame pairs’. Proc. IEEE Int. Conf. Philadelphia, PA, USA, March 2005, Vol. II, pp. 933–936

5. Problem Statement (cont.) • BG and FG objects can pass through a single location in the interpolated frame. • To interpolate the MA region correctly, the motion of the FG object should be correctly chosen. Inaccurate MVs caused by the occlusion is removed by [6].

6. Proposed MCI Method • MA occurswhenever the magnitude of relative motion between the FG and BG layer, ||vFG − vBG||, is larger than the size of the FGobject. • The FGobject always moves away from the occlusion belonging to the BG layer(U). Ox : Set of pixel positions of uncovered (U) in fn+1 Sx : Set of the positions which the multiple MVs causing MA pass through in fn+1 Ox Sx

7. Proposed MCI Method (cont.) • First find a position within the occlusion, xocOx, such that ||xoc − xi || is minimal, xiSx. • Then, the element of Sx, farthest from xoc, is associated with the FG object. • For the occlusion regions (no MV passes), the MV of the BG is easily estimated using the neighboring MV. Ox Sx

8. Proposed MCI Method (cont.2) • However, such a straightforward approach to the distance calculation is computationally expensive. • Using a simple morphological operator as follows: • 1. The Ox area is first do the dilation by repeating with the 3×3 structuring element. At each iteration, the dilated pixels are labeled with the number of the iteration. • 2. When the MA occurs, Sx is obtained and the distance label is examined. The pixel with the largest label value is utilized to interpolate the frame(FG). • 3. Step 2 is repeated until the overlapped area is completely interpolated.

9. Proposed MCI Method (cont.2) Ox Sx

10. Experimental Results

11. Experimental Results

12. Conclusions • We propose a novel layer inference-based MCI method using the occlusion information. • Experimental results confirm that the proposed method achieves much higher visual quality of the interpolated frame without annoying artifacts in both the MA and occlusion regions