1 / 37

Virtual Viewpoint Reality NTT: Visit 1/7/99

Virtual Viewpoint Reality NTT: Visit 1/7/99. Overview of VVR Meeting. Motivation from MIT ... Discuss current and related work Video Activity Monitoring and Recognition 3D Modeling Demonstrations Related NTT Efforts Discussion of collaboration Future work Lunch. Motivating Scenario.

nevin
Download Presentation

Virtual Viewpoint Reality NTT: Visit 1/7/99

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Virtual Viewpoint RealityNTT: Visit1/7/99 Virtual Viewpoint Reality

  2. Overview of VVR Meeting • Motivation from MIT ... • Discuss current and related work • Video Activity Monitoring and Recognition • 3D Modeling • Demonstrations • Related NTT Efforts • Discussion of collaboration • Future work • Lunch Virtual Viewpoint Reality

  3. Motivating Scenario • Construct a system that will allow a user to observe any viewpoint of a sporting event. • From behind the goal • Along the path of the ball • As a participating player • Provide high level commentary/statistics • Analyze plays • Flag goals/fouls/offsides/strikes Virtual Viewpoint Reality

  4. Given a number of fixed cameras… Can we simulate any other? Virtual Viewpoint Reality

  5. A Virtual Reality Spectator Environment • Build an exciting, fun, high-profile system • Sports: Soccer, Hockey, Tennis, Basketball • Drama, Dance, Ballet • Leverage MIT technology in: • Vision/Video Analysis • Tracking, Calibration, Action Recognition • Image/Video Databases • Graphics • Build a system that provides data available nowhere else… • Record/Study Human movements and actions • Motion Capture / Motion Generation Virtual Viewpoint Reality

  6. Factor 1: Window of Opportunity • 20-50 cameras in a stadium • Soon there will be many more • HDTV is digital • Flexible, very high bandwidth transmissions • Future Televisions will be Computers • Plenty of extra computation available • 3D Graphics hardware will be integrated • Economics of sports • Dollar investments by broadcasters is huge (Billions) • Computation is getting cheaper Virtual Viewpoint Reality

  7. Factor 2: Research • Calibration • How to automatically calibrate 100 moving cameras? • Tracking • How to detect and represent 30 moving entities? • Resolution • Assuming moveable/zoomable cameras: How to direct cameras towards the important events? • Action Understanding • Can we automatically detect significant events - fouls, goals, defensive/offensive plays? • Can we direct the user towards points of interest? • Can we learn from user feedback? Virtual Viewpoint Reality

  8. Factor 3: Research • Learning / Statistics • Estimating the shape of complex objects like human beings is hard. How can we effectively use prior models? • Can we develop statistical models for human motions? • For the actions of an entire team? • Graphics • What are the most efficient/effective representations for the immersive video stream? • What is the best scheme for rendering it? • How to combine conflicting information into a single graphical image? Virtual Viewpoint Reality

  9. Factor 4: Enabling Other Applications • Cyberware Room • A room that records the shape of everything in it. • Every action and motion. • Provide Unprecedented Information • Study human motion • Build a model to synthesize motions (Movies) • Study sports activities • Provide constructive feedback • Study ballet and dance • Critique? • Study drama and acting Virtual Viewpoint Reality

  10. Factor 5: NTT Interest and Involvement • NTT has expertise: • Networking and information transmission • Computer Vision • Human Interfaces • We would like your feedback here! Virtual Viewpoint Reality

  11. Overview of VVR Meeting • Motivation from MIT ... • Discuss current and related work (MIT) • Video Activity Monitoring and Recognition • 3D Modeling • Demonstrations • Related NTT Efforts • Discussion of collaboration • Future work • Lunch Virtual Viewpoint Reality

  12. Progress on 3D Reconstruction • Simple intersection of silhouettes • Efficient but limited. • Tomographic reconstruction • Based on medical reconstructions. • Probabilistic Voxel Analysis (Poxels) • Handles transparency. Virtual Viewpoint Reality

  13. Simple Technical Approach • 1 Integration/Calibration of Multiple Cameras • 2: Segmentation of Actors from Field • Yields silhouettes -> FRUSTA • 3: Build Coarse 3D Models • Intersection of FRUSTA • 4: Refine Coarse 3D Models • Wide baseline stereo Virtual Viewpoint Reality

  14. Idea in 2D Virtual Viewpoint Reality

  15. Idea in 2D: Segment Virtual Viewpoint Reality

  16. Idea in 2D: Segment Virtual Viewpoint Reality

  17. Idea in 2D: Intersection Virtual Viewpoint Reality

  18. Coarse Shape Virtual Viewpoint Reality

  19. Real Data: Tweety • Data acquired on a turntable • 180 views are available… not all are used. Virtual Viewpoint Reality

  20. Intersection of Frusta • Intersection of 18 frusta • Computations are very fast • perhaps real-time Virtual Viewpoint Reality

  21. Agreement provides additional information Virtual Viewpoint Reality

  22. Tomographic Reconstruction • Motivated by medical imaging • CT - Computed Tomography • Measurements are line integrals in a volume • Reconstruction is by back-projection & deconvolution Virtual Viewpoint Reality

  23. Acquiring Multiple Images (2D) Virtual Viewpoint Reality

  24. Backprojecting Rays Virtual Viewpoint Reality

  25. Back-projection of image intensities Virtual Viewpoint Reality

  26. Volume Render... • Captures shape very well • Intensities are not perfect Virtual Viewpoint Reality

  27. Iterative refinement: 1 Confidence measurement Confidence Virtual Viewpoint Reality

  28. Iterative refinement: 2 Constraint Application { Normalize along ray to obtain estimate probability that voxel is visible from camera Virtual Viewpoint Reality

  29. Iterative refinement: 3 Constraint Application PDF Distance from sensor Virtual Viewpoint Reality

  30. Iterative refinement: 4 Compute oriented differential cumulative density function: dCDF = CDF - PDF CDF is computed by integration of PDF along line of sight. Estimation CDF dCDF PDF Distance from sensor Virtual Viewpoint Reality

  31. Iterative refinement: 5 Estimation The inverse dCDF, idCDF = 1 - dCDF is an estimate of how much information each camera has about each voxel. dCDF idCDF Distance from sensor Virtual Viewpoint Reality

  32. Iterative refinement: 6 inverse differential cumulative distribution function idCDF Visibility of voxel from camera Distance from sensor Virtual Viewpoint Reality

  33. Iterative refinement: 7 The idCDF is used to update confidences, given expectation of occlusion Estimation disagreement expected due to occlusion Importance of mismatch Distance from sensor Virtual Viewpoint Reality

  34. Iterative refinement: 8 Or given the expectation of transparency Estimation Aggravated disagreement due to expected transparency Importance of mismatch Distance from sensor Virtual Viewpoint Reality

  35. Constrain Measure Estimate 9 Iterative refinement Virtual Viewpoint Reality

  36. Results… Virtual Viewpoint Reality

  37. Overview of VVR Meeting • Motivation from MIT ... • Discuss current and related work (MIT) • Video Activity Monitoring and Recognition • 3D Modeling • Demonstrations • Related NTT Efforts • Discussion of collaboration • Future work • Lunch Virtual Viewpoint Reality

More Related