1 / 68

Three-Dimensional Video Postproduction and Processing

Three-Dimensional Video Postproduction and Processing. Ibraheem Alhashim - July 10 th 2013. CMPT 880. Overview. History + Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook. Overview. History + Fundamentals

libba
Download Presentation

Three-Dimensional Video Postproduction and Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Three-Dimensional VideoPostproduction and Processing Ibraheem Alhashim - July 10th 2013 CMPT 880

  2. Overview • History + Fundamentals • 3D display technologies • Basic processing • View synthesis • 2D to 3D conversion • 3D media for mobile devices • Outlook

  3. Overview • History + Fundamentals • 3D display technologies • Basic processing • View synthesis • 2D to 3D conversion • 3D media for mobile devices • Outlook

  4. A bit of history • Imaging technology The Jazz Singer 1927 Underwood 1901 Joseph Niépce1826  JC d'Almeida 1858 James Maxwell 1855 Stereoscope 1860 The Power of Love (1922 film)

  5. A bit of history Becky Sharp (1935) 1950s

  6. A bit of history • Timeline of 3D movies 1950s 1980s 2010s Worldwide: $2,782,275,172

  7. 3D Feature Films • Highest-grossing films 2012: nine of the top 15 were in 3D [businessinsider.com] • Industry forecast > 20% of TVs by 2015

  8. Other uses • WW2 maps • Virtual reality • 3D Electron Microscopy • Video games + virtual cinema 1945 1968 1980 2013 2013 2013

  9. 3D Video • Usually marketed as objects popping off screen

  10. 3D Video • In reality.. It’s the same old concept • Present slightly different image per eye • The brain combines them and perceives depth • Trick human visual system • Stereo 3D content production: • technical, psychological, and creative skills

  11. 3D Video - Issues • Not as straightforward as 2D production • Several considerations for a good 3D experience • Balance between 3D effect and overall experience • Minimize viewing discomfort • Stereoscopic comfort zone • Scene depth adaptation • Control of global and local disparity • Video composition

  12. Stereoscopic Comfort Zone • Comfort zone • Stereographer • “bring the whole real world inside this virtual space”

  13. Control of Absolute Disparity • Convergence is controlled by shifting the views

  14. Scene Depth Adaptation • Keep in mind disparity range • (screen size and resolution) • Carefully plan a scene’s 3D effect • Consider transitions and provide resting periods • Post-production depth adaptation • Manual changes per display

  15. 3D Display Adaptation • From cinema to TV • Depth composition has to modified (stereographer) • Depth information allows for virtual view interpolation

  16. Local Disparity Adaptation • Objects should be within stereoscopic window • Intentional depth changes • Physical / multiple camera rigs • Synthetic / depth-editing • Objects at the border • Can cause retinal rivalry • Should be cropped by virtually shifting screen plane closer • However, not applicable to live broadcast

  17. Live auto-correction • Automatic correction & manipulation of stereo live broadcast • Live sport events (big player) • Close objects could abruptly appear • Open research problem • Some kind of novel view synthesis

  18. Video composition • Mixing and Composition of 3D Material, Real and Animated Content • Graphics overlay / subtitles • Cannot be simply pasted over other footage • Leverage knowledge about depth range of footage

  19. 3D Video - Issues • Summary • 3D Production has an “art” component • Different medium requires different parameters • Content makers / directors need to think about 3D issues • Stereographer try to balance 3D effect with overall viewing experience

  20. 3D video • Fundamentals • 3D display technologies • Basic processing • View synthesis • 2D to 3D conversion • 3D media for mobile devices • Outlook

  21. 3D Display Technologies • “Offer immersive experience” • 3D Glasses (cinema + TV) • Head-mounted displays • Volumetric and holographic displays • Autostereoscopic displays

  22. Autostereoscopicdisplays • Best choice for mobile devices • Backward compatible & closer to viewer expectation • Most common • Parallax barrier • Lenticular sheet

  23. Autostereoscopic displays • Crosstalk (ghosting) is most important parameter • “information meant for one eye intrudes into the other eye’s view”

  24. Autostereoscopic displays • Issues • Generally less available depth range • More ghosting artifacts • Also, depth information is essential • Synthesize different views

  25. 3D video • Fundamentals • 3D display technologies • Basic processing • View synthesis • 2D to 3D conversion • 3D media for mobile devices • Outlook

  26. Basic processing • Signal processing to avoid visual artifacts • Any small visual discrepancy could cause discomfort • Three main categories • Correction of Geometrical Distortions • Color Matching • Adjustment of Stereo Geometry

  27. Correction of Geometrical Distortions • Camera rigs might not be perfectly aligned • Real lenses impose radial distortions by nature • Other lens parameters might not sync • E.g. geometrical lens distortions or chromatic aberration

  28. Color Matching • Color discrepancy can lead to eyestrain and visual fatigue • Manual calibration need to be done on cameras • Automatic methods exist (histogram filtering)

  29. Color Matching • Modern professional postproduction tools incorporate stereo color matching and grading

  30. Adjustment of Stereo Geometry • Convergence need to be selected and balanced carefully to achieve good stereo content • Usually by shifting images horizontally in contrary directions, however, cropping & scaling is needed • “shift-crop-scale” • Demo http://stereo.nypl.org/create

  31. Adjustment of Stereo Geometry • Stereo baseline is fixed during shooting • Several hardware solutions help camera team analyze the disparity range • Also help visualize result of shift-crop-scale

  32. 3D video • Fundamentals • 3D display technologies • Basic processing • View synthesis • 2D to 3D conversion • 3D media for mobile devices • Outlook

  33. 3D Depth information • Depth information is needed for • novel output images in post-production • adjusting the view parallax (different screens) • many different uses in postproduction

  34. Depth info • Extracting depth information • (time of flight camera / SfS) • Structure-from Stereo (SfS) • Advanced computer vision problem • Stereo matching • Local – block matching, optical flow est. • Global – graph cuts, simulated annealing

  35. Structure-from Stereo • Global methods are more accurate • Slow + don’t work well on video / motion • Local methods are more widely used • Window-based methods • Some system are in real-time • Blocky output

  36. Example of Depth-based method • Apply hybrid recursive matching (HRM) • Follow by cross-trilateral median filtering (ACTMF) • Semi-automatic

  37. Post processing depth • Align depth discontinues to object borders • Remove noise and mismatches • Fill occlusions • Approaches • Use image segmentation • Neighborhood filtering

  38. View synthesis • Synthesize new virtual stereo views by image-based rendering • Input – depth + color • Output – image with new view • Two types depth-based + warping-based

  39. Depth-based • Computer vision techniques • Image-based rendering (IBR) • Depth-based rendering (DIBR) • Layered-depth images (LDI) • Intermediate view reconstruction (IVR)

  40. Depth-Image-Based Rendering • Need pixel-by-pixel depth maps • Recent focus • Handle depth discontinues • Better depth boundaries

  41. Warping-based • Methods that deform the image content directly • Compress or stretch by nonlinear warp function • Do not need camera calibration, segmentation, fill holes • Worst case, visible wobbling artifact

  42. View synthesis • Summary • Depth maps are computed using computer vision techniques (still active) • Generate new views by image-based rendering or warping • Warping methods can potentially have less visual problems DEMO

  43. 3D video • Fundamentals • 3D display technologies • Basic processing • View synthesis • 2D to 3D conversion • 3D media for mobile devices • Outlook

  44. 2D to 3D • 3D to 2D is trivial • Hot topic for 3DTV and 3D cinema • Methods so far are • Manual (computer assisted) • Automatic

  45. Depth cues • Human visual system • Monocular cues • Binocular cues

  46. Depth cues • Monocular depth cues- things can be seen by one eye (2D Camera) • Focus / defocus, perspective, relative size • Light and shading and texture • Motion parallax • Binocular – 2 eyes or 3D camera rigs • accommodation, convergence, and binocular discrepancy

  47. Manual 2D to 3D • Applicable to prerecorded video • Utilize depth cues to generate a stereoscopic view for each frame • Time consuming and costly • Cost vs. quality

  48. Manual 2D to 3D • Three major steps • Rotoscoping/ segmentation • Depth assignment • Inpainting • Few companies provide process as a service

  49. Depth Assignment • Shifting different parts of scene to simulate 3D • To avoid cardboard effect hire a “3D compositor” • Create displacement maps for each pixel • Use 3D primitives, spheres or cubes • Use DIBR or 3D warping to synthesize view • cannot handle transparencies well

  50. Occlusion filling • Also known as “in-painting” • One of the most challenging parts in 2D/3D conversion

More Related