1 / 46

Transport and Rendering Challenges of Multi-Stream, 3D Tele-Immersion Data Herman Towles,

Transport and Rendering Challenges of Multi-Stream, 3D Tele-Immersion Data Herman Towles, Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott Larsen and Nathan Beddes Department of Computer Science University of North Carolina at Chapel Hill October 28, 2003. ‘’ Portal’ to Another Place.

keely
Download Presentation

Transport and Rendering Challenges of Multi-Stream, 3D Tele-Immersion Data Herman Towles,

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Transport and Rendering Challenges of Multi-Stream, 3D Tele-Immersion Data Herman Towles, Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott Larsen and Nathan Beddes Department of Computer Science University of North Carolina at Chapel Hill October 28, 2003

  2. ‘’Portal’ to Another Place Static, High-Resolution 3D Scene UNC - January 2000 Head-Tracked, Passive Stereo Projector Display

  3. Static Scene Acquisition • DeltaSphere™ Scanning Laser Rangefinder • Time of Flight • Resolution: ~3K x 2K for 150o azimuth and 90o elevation • Digital Camera mounted at identical COP for color image • Output • Range Image • Color Camera Image Images courtesy of 3rd Tech, Inc.

  4. NTII Collaborators • UNC-CH – Computer Graphics • Henry Fuchs et al. • UPenn – Computer Vision • Kostas Daniilidas et al. • Brown University – Collaborative Graphics • Andy van Dam et al. • Advanced Network & Services • Jaron Lanier & Amela Sadagic Goal: Develop a Live 3D ‘Portal’

  5. 3D Tele-Immersion – Phase 1 • 3D Real-Time Reconstruction • Correlation-based Stereo Algorithm • 1-3 fps at 320 x 240 (quad-PC/550) • 1 cubic meter working volume • Up to 5 streams of depth images • 3D Tracked Stereo Display • Life-size with gaze awareness • 3-PC Rendering Cluster • Composite Scene • Live 3D Remote Collaborator • Static, office background • Collaborative Graphics

  6. Monitor Camera Network Standard Videoconferencing Model • 1-Camera to 1-Monitor Paradigm • Single Stream of 2D Image Data View-Dependent Data !

  7. 3D Rendering 3D Camera 3D Camera 3D Camera Network User Tracker 3D Tele-Conferencing Model • N-Camera to 1-Monitor • Multiple Streams of 3D Image Data • Depth Image (16-bit 1/z) and RGB Image (24-bit) View-Independent Data w/ View-Dependent Rendering !

  8. 3D Rendering Acquisition Acquisition Reconstruction Reconstruction 3D Camera 3D Camera Network Network 3D Tele-Immersion- Phase 2 • Higher Quality • Improve Resolution, Frame Rate and Display • Larger Reconstruction Volume • 30-60 color cameras • Acquire full environment Terascale Computing System (Lemieux) at Pittsburgh Supercomputing Center (PSC)

  9. 2000-2002 Results • Show video

  10. Outline • Introduction • Data Transport Challenges & Solutions • Rendering Challenges & Solutions • Conclusions/Futures

  11. Data Transport Challenges • Reliable Transport • Low Latency • Synchronized Arrival of Multi-Streams • Large Bandwidth Requirements

  12. Compression Opportunities • Color Image Compression • Spatial Compression (JPEG) • Temporal Compression (MPEG) • Depth Image • 16-bit 1/z image • Elimination of Redundant Data • Overlapping FOVs from Multiple Cameras • Reduces BW and Rendering Requirements !!

  13. Overlapping Depth Images

  14. Redundant Data Elimination • Algorithm: Compare Every Depth Stream • Pros • Best compression • Eliminates all redundant Data • Cons • Jeopardizes Real-Time Performance • Not scalable with number of 3D cameras • Maximizes network BW requirements between ‘3D Cameras’

  15. Group-Based, Differential Stream Compression • Two Stream Differential Comparison • Maintains real-time performance • Scales with number of cameras • Static, Disjoint Stream Groups • Geometric coherency metric • One Reference Stream per Group • Main Reference Stream • Stream most closely aligned with remote viewpoint • Uncompressed

  16. Group 1 Group 2 5 Group 2 Group 3 Grouping & Differential Encoding 1 2 3 4 5 6 7 8 9

  17. Algorithm Performance • Using Synthetic Camera Arrays • 13 and 22 3D-camera arrays • Real-world 3D data • 5X Compression Ratio • 50-60% of Best Possible Point Reduction • 10z Performance at 640x480

  18. Multi-Stream Transport • Reliable is Good, so TCP/IP WRONG! • Latency • Unsynchronized Arrival of Streams • Synchronized at Cameras • Must be re-synchronize at renderer • TCP Flows Competing for BW

  19. Frame Arrival Time Variation

  20. Throughput Variation

  21. CP-RUDP • Reliable UDP (RUDP) • Coordination Protocol (CP) • Characteristics • A TCP-friendly UDP protocol • Aggregately acts like N TCP flows • Providing BW coordination and synchronization of multiple flows • Prevents send rates from exceeding network capacity Flows that are ahead are given less BW Flows that are behind are given more BW

  22. 3D Rendering 3D Camera 3D Camera 3D Camera Internet2 CP-RUDP Software Router CP-RUDP Software Router User Tracker Future CP-RUDP Testbed • Future Network Router Functionality • Emulated today on fast FreeBSD PC

  23. Outline • Introduction • Data Transport Challenges & Solutions • Rendering Challenges & Solutions • Conclusions/Futures

  24. Rendering Challenges • High Performance, Interactive Architecture • Quality Surface Representation • Scalable Performance

  25. High Performance, Interactive Architecture • Frame Re-synchronization • ‘N’ 3D-camera Streams • Timestamps on each data frame • Buffering -> More Latency BAD! • Multi-threaded Design for Interactivity • 30Hz Refresh Rate Desirable • <10 Hz Data Update Rate • 3-PC Rendering Cluster (Linux) • Left & right eye rendering nodes • 3rd Node – Network Aggregation Point

  26. Rendering Architecture

  27. Quality Surface Representation • Depth Images (1/z + RGB per pixel) • Real-Time Triangulation • Performance Limiting • Artifacts due to noisy outliers. Most Disturbing! • Real-Time Massive Point Cloud Rendering • Display List vs. VAR • GL_POINT vs. GL_POINT_SPRINT • Screen-Oriented vs. Object-Oriented Splats

  28. Fixed-Size Splats

  29. Screen vs. Object Space Splats

  30. Scalable Performance • 2x Gain w/3-PC Rendering Cluster • Ride the Wave (Moore’s Law & Beyond) Not Enough! • Divide and Conquer • Eliminate Data Replication Requirement • Match # Primitives to Performance • Depth Compositing • UNC PixelFlow, Stanford/Intel Lightning-2 or HP Sepia • DVI output – RGB and Z

  31. Conclusions/Futures • Much work remains ! • Performance and Fidelity • Investigating Temporal Compression of 3D Streams • Testing CP-RUDP Network Protocol • New Voxel-Based Solution • Elminates Redundant Data • Normals and Confidence Value per Voxel • GPU Programs for Object-oriented Quad Splats with Projective Texture

  32. Research Sponsors • National Science Foundation • Grant IIS-0121293 Junku Yuh, Program Director • DARPA • 3-D Tele-Immersion over NGI • Advanced Network & Services, Armonk, NY • National Tele-Immersion Initiative (NTII), 1998-2000

  33. Thank You UNC - January 2000

  34. Thank You UNC - January 2000

  35. Positive Technology Trends • Low-cost image sensors • Vigorous research into new light field and image-based scene reconstruction algorithms • Incredible performance in commodity PCs and 3D graphics • Ease of Developing Custom Hardware • New Displays – HiRes LCD and Plasma panels, Tiled-Projectors, OLEDs, Auto-stereoscopic • Increasing network bandwidth to the workplace and home

  36. Renderer Block Diagram

  37. VIRTUE: VIRtual Team User Environment • Information Society Technologies (IST) Programme of the European Commission http://bs.hhi.de/projects/VIRTUE.htm

  38. VIRTUE Partners • Heinrich-Hertz Institut (now Fraunhofer member) • Disparity Estimator, Real-time Components • University of Delft • 3D Algorithms, Image-Based Rendering • TNO Human Factors • Sony U.K. • System design, Compositor, Rendering, MPEG-2 • Harriot Watt University • 3D Video Processing • British Telecom • Project Prime, Video-based Head Tracking

  39. VIRTUE Technology • Realistic Wide View Synthesis for Dynamic Scenes • Object-based Segmentation • Disparity Engine runs 4x4 grid @ videorate • Facial feature (eye) tracking • Generating View-dependent Novel View • 2D Display (with motion parallax) rendered with virtual background Demonstrated at IBC – Fall 2002

  40. HP Labs ‘Coliseum’ System Image-based Visual Hull

  41. HP Labs ‘Coliseum’ System • Personal Immersive Conferencing • Based on ‘Visual Hull’ research of Matusik, McMillan et al. at MIT • Space carving algorithm • Foreground/Background • 5 camera (1394) array, Now runs on 1 PC • 224 x 224 @ 10 Hz • 300 x 300 display window • 3D Volume Composited with 3D VRML scene Demonstrated at ITP – Dec 2002

  42. Credits • Colleagues at • UPenn: Kostas Daniilidis (Co-PI), Nikhil Kelshikar, Xenofon Zabulis • PSC: John Urbanic, Kathy Benninger • Advanced Networks & Services: Jaron Lanier, Amela Sadagic • UNC: Henry Fuchs (PI), Greg Welch, Ketan Mayer-Patel • UNC RAs • Ruigang Yang, Wei-Chao Chen, Sang-Uok Kum, Sudipta Sinha, Scott Larsen, Vivek Sawant, Travis Sparks, David Ott, and Gabe Su

  43. Our Vision: ‘Xtreme Tele-Presence ‘Office of the Future’ Andrei State 1998

  44. Our Vision: ‘Xtreme Challenges • Distributed Graphics • Collaborative Datasets • User Interfaces • Presentation • 3D Stereo Display • View Dependent • Front Projection • Spatialized Audio • 3D Scene Acquisition • Cameras • 3D Reconstruction • Networking • QoS • BW-Latency-Jitter ‘Office of the Future’ Andrei State 1998

  45. Immersive Electronic Books • New Paradigm for Surgical Training • Allow Surgeons to Witness & Explore • Replay a surgical procedure from any novel viewpoint in 3D • Augmented with annotations & relevant medical metadata

  46. Remote Medical Consultation

More Related