Stereoscopic video generation from monoscopic video

1. Stereoscopic video generation from monoscopic video Xu Xuyuan(Evan) 2010/5/20

3. 3DTV in the market • Most of the TV manufacturers release the 3DTV sets. • Sony 3DTV using Shutter glass technology

4. Motivation of 2D to 3D • Large number of monoscopic videos existing in many databases • Stereo video market demand • Enable to capture 3D using the 2D camera

5. Introduction • 3D shutter glass display system • 2D-3D generation process • Depth map generation • Depth Image Based rendering • Conclusion • Recommendation

6. 3D display system--shutter glass 3D display system • Device • A monitor with frequency 120Hz or above • A shutter glasses

7. 3D display system --shutter glass 3D display system • Methodology • Used the monitor to shift the image concurrently.

8. Overview of 2D-to-3D Video Conversion Process DIBR --Depth Image Based Rendering Input Video 3D Image Warping Hole filling Modification of DIBR Depth map generation Output 3D video Motion estimation Filtering Color Segmentation Fusion

9. Overview of 2D-to-3D Video Conversion Process Input Video Depth map generation Motion estimation

10. Depth map generation based on depth from motion • Depth Cue -- Motion Parallax • Near objects move faster than the distant objects • Requirement of source video • Have movement • Depth map • A grey scale image • Each pixel represent by 8 bits

11. Block Motion estimation Searching Frame Reference Frame

12. Block-Matching Based Motion Estimation • Used Y (luminance) value for the motion vector search • sum of absolute difference • (SAD) • Four kind of search: • Full Search • Spiral Search • Diamond Search • Hexagonal search

13. Depth Generation from Motion Vectors • D(i,j) is the depth value for pixel (i,j) • MV(i,j)x is the X motion vectors values • MV(i,j)y is the Y motion vectors values • C is a custom define scale parameter

14. Compare of motion estimation Spiral search Input Image of size: 416*240 Hexagonal search Full Search Diamond search

15. Overview of 2D-to-3D Video Conversion Process Input Video Depth map generation Motion estimation Color Segmentation

16. Color Segmentation • 1. Step color quantization • Set a quantization step. • 2. Color merging • Find out the best two merging color in the image and merge them into one.

17. Node Structure prev next Color – R, G B Before Merging Number of pixels m j p i k q q After Merging j m p i and j are merging into k (i is updated into k and j is delete) Color merging • Merging ColorC = |colorA*numA+colorB*numB|/(numA+numB) • Error = (|colorA- colorC|*numA+ |colorB- colorC|*numB)/(numA+numB)

18. Segmentation While (has unassigned value for the segmentation map) { index = Get next unassigned index point from the segmentation map. Find the continue area begin with the index in sMap with the value equal to qMap[index] Assign a defined segmentation value to sMap } qMap: quantization map sMap: segmentation map

19. Overview of 2D-to-3D Video Conversion Process Input Video Depth map generation Motion estimation Color Segmentation Fusion

20. Fusion While (has unassigned value for final depth map) { index = Get next unassigned index point from final depth map Find out the corresponding area in segmentation map and decide the depth value from average, median and mode. If number of the frequency depth value from mode is greater than half Final depth value = value from mode else { If there exists at least two depth value from three fusion method are similar Final depth value = interpolation of these depth value else Final depth value = value from average fusion } }

21. Depth map fusion • Final depth map generate from • Depth map from motion estimation • Color segmentation Fusion

22. Overview of 2D-to-3D Video Conversion Process Input Video Depth map generation Motion estimation Filtering Color Segmentation Fusion

23. Depth Map Post-Processing: • Gaussian filtering

24. Example

25. Hole filling

26. Overview of 2D-to-3D Video Conversion Process DIBR Input Video 3D Image Warping Depth map generation Motion estimation Filtering Color Segmentation Fusion

27. DIBR-3D Image Warping • Formula:

28. Overview of 2D-to-3D Video Conversion Process DIBR Input Video 3D Image Warping Hole filling Depth map generation Motion estimation Filtering Color Segmentation Fusion

29. 3D image Warping Objects Zdetph Intersection Screen Left eye Right eye

30. DIBR-Hole Filling • Why hole filling? • Linear Interpolation to fill the hole.

31. Overview of 2D-to-3D Video Conversion Process DIBR Input Video 3D Image Warping Hole filling Modification of DIBR Depth map generation Motion estimation Filtering Color Segmentation Fusion

32. Modification of DIBR method New depth Zdetph Depth =255 Screen Screen Depth(i,j)=255-Depth(i,j). Left eye Right eye

33. Modification of DIBR method

34. 2D-to-3D Video Conversion Process DIBR --Depth Image Based Rendering Input Video 3D Image Warping Hole filling Modification of DIBR Depth map generation Output 3D video Motion estimation Filtering Color Segmentation Fusion

35. Overview of 2D-to-3D Video Conversion Process DIBR Input Video 3D Image Warping Hole filling Modification of DIBR Depth map generation Output 3D video Motion estimation Filtering Fusion Color Segmentation Depth Map

36. Conclusion • Depth from motion can achieve really time generation. • Achieve a good 3D effect for the source video with static background • The output video compatible for most of 3D display system • GUI – More user friendly interface

37. Further Recommendation • Camera motion correction • Multi-view 3D generation

38. Thank You!