Media Re-expression for Stereo Cinema

1. Media Re-expressionfor Stereo Cinema FélixRaimbault, François Pitié and Anil Kokaram

2. Overview 1/2 • Motion Magnification for Stereo Videos • Stereo Video Inpainting

3. Overview 2/2 • Automatic Cartoonization • Stereo Video Segmentation Segmentation of a video volume [Collomosse et al. 2005]

4. Stereo Video Inpainting

5. Context • Initial motivation: fill-in revealed areas for stereo motion magnification • Correct artefacts arising during shooting in stereo • Remove unwanted objects • Fill-in dis-occluded areas for disparity remapping play video

6. Exemplar-based Framework: Priority • Priority: [Criminisi, A. et al. 2004] • process first pixels with more available information nearby • try to reconstruct first areas with “structure” (edges and depth discontinuity) target frame edge map initial priority order of filling reconstruction

7. Exemplar-based Framework: Matching • Patch-matching strategy:[Efros, A. and Leung, T. 1999] • find similar neighbourhood to current missing pixel • SSIM-based distance: compare patches structure ... ^ target patch around x I(x) replaced by I(x) ^ S(x) best match at site x

8. Patch Tracking • [Raimbault, F. and Kokaram, A. SPIE 2011] • use long-term data • motion vector reconstruction inside the hole • use data across views • disparity vector reconstruction inside the hole

9. Smoothness • Spatial smoothness: (to be submitted to WIAMIS’12) • “coherent patch sewing” • estimate average distance of selected patches as a criteria to prune patch copying target frame missing data pixel by pixel coherence sewing

10. Smoothness • Stereo-Temporal smoothness:(to be submitted to WIAMIS’12) • preferentially select patch in previously reconstructed frame • stereo-spatio-temporal patch-matching v’ v t-1 t t+1 • de-activate smoothness for outliers (bad motion and disparity estimates)

11. Luminance Correction • [Raimbault, F. and Kokaram, A. SPIE 2011]: • colour discrepancy • due to sampling from frames far away in time from current frame (lighting can change) • colour correction needed • 1-tap linear predictive model: • weighted least squares solution

12. Results • video “spywalk” • reconstruction of twist in the lash of the girl’s bag • average SSIM: 0.9997 slightly better than Rig Removal: 0.9989 • small hole -> block matching is enough to estimate offsets • video “water_drop” • “fresh” data input from other view in our technique whereas reconstruction with Rig Removal degrades • video “walking_girl” • block matching and pixel accuracy -> not enough • Published in SPIE’11– Accepted for publication in JEI’12 play video

13. Issues • Issues: • greedy => lack of global coherence • more accurate motion vectors needed • parameters choice can be complicated (patch size, search size) • lack of temporal smoothness • To be explored: • experiment with global optimisation (QPBO) • use trajectories (Viterbi tracker) • patch stitching • impose constraints by first doing stereo-video segmentation

14. Plan • Internship for Sony Research in Stuttgart • temporal stabilisation of videos • Stereo Video Segmentation • based on [Baugh, G. and Kokaram, A. 2010] • Return to Stereo Video Inpainting • feature point trajectories [Baugh, G. and Kokaram, A. 2009]