230 likes | 413 Views
Treffen der ITG FG 3.1.2 Sci-Worx, Hannover, 2002. Streaming of Panoramas. Dr.-Ing. Peter Eisert Computer Vision & Graphics Image Processing Department Heinrich-Hertz-Institute, Berlin. Motivation. 3D Streaming / Mobile Rendering. Interactive Navigation in Virtual Worlds on Mobile Devices
E N D
Treffen der ITG FG 3.1.2 Sci-Worx, Hannover, 2002 Streaming of Panoramas Dr.-Ing. Peter Eisert Computer Vision & Graphics Image Processing Department Heinrich-Hertz-Institute, Berlin
Motivation 3D Streaming / Mobile Rendering • Interactive Navigation in Virtual Worlds on Mobile Devices • Virtual city guides (position dependent) • Tourism applications • E-commerce • Requirements / Constraints • 3D scene description located on a server(cylindrical panoramas, concentric mosaics, light fields) • Interactive navigation with low delay(Transmission of navigation decision / position of user) • Low data-rate of wireless channels • Restricted computational power and storage of PDAs
Overview • Image-based rendering • Cylindrical panoramas • Concentric mosaics • Light fields • Streaming of image-based datasets • First implementation: • MPEG-4 client/server system • Streaming of high resolution cylindrical panoramas • Demo
Image-based Rendering Real Picture Computer Graphics • Simple acquisition • Fast rendering • Natural impression • Modeling time consuming • Powerful rendering necessary • Artificial appearance
Cylindrical Panoramas • Simple acquisition • Rotating camera or mirror system • Vertical field of view limited • Navigation: Rotation and zoom • QuickTimeVR
Concentric Mosaics • 3D plenoptic function • Navigation: Rotation, translation in a plane and zoom • Camera motion constrained on planar concentric circles • Acquisition of n+1 vertical lines (slit images)at different radial positions horizontal parallax • Each pushbroom camera contributes one cylindrical panorama
Example of a Concentric Mosaic • a) Panorama acquired at the rotation center (CM0) • b) Panorama acquired at the outmost position (CMn) • c) Panorama as in b) but looking in the opposite direction a) b) c)
Simplified Acquisition • One single off-center camera rotating on an arm with radius Rn • a) Free movement inside a circle of radius • b) Free movement inside the ring Normal Viewing Direction Tangential Viewing Direction
Light Fields • Static 3D scene • Known viewing positions • 2D array of images • 4D plenoptic function(4D array of rays) • no time dependency • no wavelength • no scattering in air (bounding box) • Rendering • arbitrary viewing position and direction • new view interpolated from stored images • no geometry required
Light Field Representation ST array of UV images UV array of ST images
Interpolation • 8 x 8 images of resolution 256 x 256 • Aliasing!! • Many images needed large data sets
Example: Dragon (Stanford) • 4 slabs • 24 x 24 images each slab • resolution 256 x 256 pixels
3D Streaming • System for streaming image-based data • Virtual walk-throughs • E-commerce • Free navigation in 3D scenes • Visualization on portable devices • Limited band-width and storage • Efficient coding and transmission necessary
Interaction of all Components Photorealistic Visualization • Image-based representation • Large amount of data Free navigation Representation • Fast transmission of requested data • Prefetching necessary (RTT>100ms) • Intelligent selection of transmitted data • Efficient coding necessary • Scalability • Adapted to representation Coding Navigation Streaming • Efficient packet-based protokoll • RD optimization with delay constraints Bit-Rates: ~ 100 kbit/s - 2 Mbit/s
Current Implementation • Cylindrical, high resolution panoramas • MPEG-4 BIFS scene Representation • JPEG • JPEG-2000 Coding Navigation • MPEG-4 BIFS Player Streaming • MPEG-4 client/server architecture • TCP / UDP
Patches 360°-Panorama, BIFS Scene Cylindrical Panoramas • Navigation: Rotation / Zoom • Random access to image patches • Pre-fetching realized by visibility sensors
Current Viewing Area Visibility Sensor Visibility Sensors • Pre-fetching of neighboring patches • Removal of patches that are no longer used
Summary • Image-based rendering provides tools for photo-realistic visualization of 3D scenes • Cylindrical panoramas / concentric mosaics / light fields • Large datasets require efficient encoding and transmission • Interactive navigation on mobile devices • New applications • virtual guides • e-commerce • MPEG-4 client/server system for streaming high resolution panoramas