290 likes | 495 Views
Interactive Video Tours. MSR Interactive Visual Media Group //msrweb/vision/IBR Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Simon Winder, Antonio Criminisi. Being There: Video-Based Reality Project. A new computer-mediated interactive medium
E N D
Interactive Video Tours MSR Interactive Visual Media Group //msrweb/vision/IBR Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Simon Winder, Antonio Criminisi
Being There:Video-Based Reality Project A new computer-mediated interactive medium Goal:Combine theinteractivity of 3D games and virtual worlds with therealism and richness of film and video
Being There:Video-Based Reality Project How? Capture real-world video footage with an omni-directional camera and add interactive video/graphic elements Application: Video-Based Walkthroughs[IEEE CG&A May/June 2004]
Virtual Tours • Move camera along a rail (“dolly track”) and play back a 360 video • Applications: • Homes and architecture • Outdoor locations(tourist destinations)
Surround video acquisition system OmniCam (six-camera head)
OmniCam • Built by Point Grey Research (Ladybug) • Six camera head • Portable hard drives, fiber-optic link • Resolution per image: 1024 x 768 • FOV: ~100o x ~80o • Acquisition speed: 15 fps uncompressed
Acquisition platforms • Robotic cart • Wearable
Calibration: Lens distortion Original (distorted) image Output (corrected, perspective) image
Calibration: Vignetting Before correction After correction
Stitching (Only 4 of 6 images shown here)
Multiperspective plane sweep Without MPPS With MPPS
Feature tracking for calibration and ego-motion Points tracked in all frames in all 6 cameras Edges tracked in all frames in all 6 cameras
Stabilization Align frame- to-frame and distribute Dheading Before motion stabilization After motion stabilization
Augmented surround video A movie clip from the original surround data. The augmented movie clip. Surround augmented with video
A second scenario: Bellevue Botanical Gardens • Head-mounted outdoor scene acquisition • Map control • Localized audio for a richer experience • Complex path navigation (greater freedom of motion)
Data acquisition • Walk through the garden with the Ladybug camera mounted on helmet • Acquisition done in about 15 minutes • 44 Gb worth of video and audio data
Map control • The garden map • Drawing the • acquisition path • Mapping video frames • positions on map • Placing sound sources • (background and dynamic) • Output is a descriptive • XML file
Some audio sources • Current ego-position Audio processing and streaming • All audio sources are: • Attenuated (based on • distance from current • position), • Mixed together, • Streamed to sound card • (DirectX Audio). • Two types of sound: • background and dynamic
Bifurcation handling • Hypothesize, align, choose best pair as bifurcation • Choice of path depends on current heading and bifurcation point orientation Current heading
Bifurcation handling • Hypothesize, align, choose best pair as bifurcation • Choice of path depends on current heading and bifurcation point orientation Current heading
3rd scenario: High-End Home(architectural walkthrough) • High-Dynamic range (HDR) imaging • Complex indoor path navigation
High dynamic range imaging • Combine multiple exposures to obtain a high dynamic range (HDR) image. Underexposed Overexposed Combined
High dynamic range imaging Underexposed frames Overexposed frames All frames • HDR imaging is challenging for videos. • Image registration is necessary.
Features and statistics • Rapid scene acquisition • Map control • Localized audio for a richer experience • Complex path navigation • High data rates: • 44 Gb worth of video and audio data in 15 minutes
Current/future work • Scene Modeling • Motion estimation and map generation • More interactive 3D elements • Compression & selective decoding • Tiled video representation • Rendering • Better use of GPU • Interactive DVD deployment