1 / 67

Mosaic-Based 3D Scene Representation and Rendering

C. C V C L. Mosaic-Based 3D Scene Representation and Rendering. Allen Hanson Computer Vision Laboratory Computer Science Department University of Massachusetts Amherst hanson@cs.umass.edu. Zhigang Zhu Visual Computing Lab Department of Computer Science

kiara
Download Presentation

Mosaic-Based 3D Scene Representation and Rendering

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. C CVC L Mosaic-Based 3D Scene Representation and Rendering Allen Hanson Computer Vision Laboratory Computer Science Department University of Massachusetts Amherst hanson@cs.umass.edu Zhigang Zhu Visual Computing Lab Department of Computer Science City College and Graduate Center City University of New York zhu@cs.ccny.cuny.edu

  2. Some Real World Problems • City/Campus modeling (AFRL, NYSIA) - airborne video: far-view (1000 ft) • Environmental monitoring (NSF) - airborne video of Amazon rain forest: far-view (1000 ft) • Robot Navigation (DARPA, ARO)- ground video: medium range (~100ft) • Under-vehicle inspection (ACT Inc.) - car drives over cameras: near-view (< 8 in) Zhu & Hanson ICIP '05

  3. Objectives • Input: 2D array of cameras • multiple viewpoints; motion parallax • Output: Image-based representation • WFOV, Stereo, Occlusion Rep. Zhu & Hanson ICIP '05

  4. Related Work • 2D mosaics from a pure rotating camera • QuickTime VR, VideoBrush, Microsoft… • Stereo mosaics from off-center rotating camera(s) • two cameras (Huang & Hung 1998) • one camera ( Peleg et al PAMI 01, Shum & Szeliski ICCV99) • Panoramic imaging from a translating camera • Manifold projection (Peleg et al CVPR’97, PAMI2000) • multiple-center-of-projection (Rademacher & Bishop, SigGraph'98) • EPI-based 3D reconstruction (Zhu, XU & Lin, CVPR99) • linear pushbroom camera (R. Gupta & R. Hartley, PAMI 97) • Parallel projection ( Chai & Shum, CVPR’00) • Geo-referenced mosaic • Plane+Parallax+DEM (Kumar, Sawhney, et al , ICPR98, 2000) Zhu & Hanson ICIP '05

  5. Outline • Mosaic Representation • Research Issues • Real Applications • Air, Ground and Under-Vehicle • Summary Zhu & Hanson ICIP '05

  6. Mosaic Representations • Perspective Image • notes: (one viewpoint, occlusion in both forward and backward direction) • Orthogonal Image (Nadir view) • notes: only nadir (or frontal) view - cannot see sides • Images with Oblique Parallel Projection • Notes: multiple parallel - can see everything (generalization of orthogonal image) Zhu & Hanson ICIP '05

  7. O invisible Perspective Images • One viewpoint O • Narrow FOV • Multiple viewing directions Zhu & Hanson ICIP '05

  8. Orthogonal Images • Multiple viewpoints • Wide FOV • One viewing direction -Nadir view • Occlusion invisible invisible Zhu & Hanson ICIP '05

  9. Oblique Parallel Projections • Multiple viewpoints – Wide FOV • Multiple viewing directions • Parallel rays in each image • Various oblique angles: invisible visible invisible visible invisible invisible Nadir ‘Forward’ ‘Backward’ Zhu & Hanson ICIP '05

  10. Multiple parallel-perspective mosaics • Multi-disparity stereo: • Correspondence for 3D reconstruction • Mosaic-based rendering without 3D • View selection and rendering Zhu & Hanson ICIP '05

  11. Stereo Mosaics • Mosaics with two different oblique angles • Parallel projection • Adaptive baseline • Uniform depth resolution Zhu & Hanson ICIP '05

  12. Stereo Mosaics • Mosaics with two different oblique angles • Parallel projection • Adaptive baseline • Uniform depth resolution Zhu & Hanson ICIP '05

  13. Advantages • Various Oblique Parallel Projections • Occlusion representation • Adaptive Baselines • Uniform depth resolution • Large FOV stereo • High quality 3D from stereo mosaics • Image-based rendering without 3D Zhu & Hanson ICIP '05

  14. Main Research Issues • Real-World Problem • Uneven, sparse camera “array” • 6 DOF motion of camera • Two Main Research Issues • Orientation Estimation (see real applications) • GPS, INS, Bundle Adjustment • Parallel Ray Generation • Ray Interpolation - PRISM Zhu & Hanson ICIP '05

  15. PRISM: Ray Interpolation - 1D case • Regular Camera with arbitrary orientation Zhu & Hanson ICIP '05

  16. PRISM: Ray Interpolation - 1D case • Regular Camera with perspective projection Zhu & Hanson ICIP '05

  17. PRISM: Ray Interpolation - 1D case • Cameras with various known orientations Zhu & Hanson ICIP '05

  18. PRISM: Ray Interpolation - 1D case • Existing parallel rays with oblique angle b Zhu & Hanson ICIP '05

  19. PRISM: Ray Interpolation - 1D case • Interpolate rays between frames by local match Zhu & Hanson ICIP '05

  20. PRISM: Ray Interpolation - 1D case • … a dense parallel projection with angle b Zhu & Hanson ICIP '05

  21. Computation & Algorithm:distribute the computations in four steps • Geo-Referencing • Motion estimation: sparse tie points distributed in entire frames • Seamless Mosaicing (PRISM) • Process two narrow slices • Stereo Matching • stereo match only in two mosaics with adaptive baselines • 3D Mapping/Visualization • just a coordinate transformation and resampling Zhu & Hanson ICIP '05

  22. Real Applications • Aerial Video Surveillance • Under-Vehicle Inspection Zhu & Hanson ICIP '05

  23. Instrumentation Package on an Airplane Watson Attitude & Heading Reference System Profiling Laser Altimeter Canon XL1 DV camcorders Duct Tape Zhu & Hanson ICIP '05

  24. Objectives • Rapidly create large FOV image mosaics that are • geo-referenced, • with 3D (stereo) viewing and • with all dynamic targets identified • As a light UAV flies over an area Zhu & Hanson ICIP '05

  25. Multiple parallel-perspective mosaics • Multi-disparity stereo: • Correspondence for 3D reconstruction • Mosaic-based rendering without 3D • View selection and rendering Zhu & Hanson ICIP '05

  26. Virtual Fly-Through Digital City/Campus Zhu & Hanson ICIP '05

  27. Real Applications • Aerial Video Surveillance • Under-Vehicle Inspection Zhu & Hanson ICIP '05

  28. Camera Geometry of the UVIS • Car drives over a 1D array of cameras… Zhu & Hanson ICIP '05

  29. Y X • 32 cameras (images) per column – spatial “scan” • Car moves – temporal “scan” 1 Zhu & Hanson ICIP '05

  30. Y X Zhu & Hanson ICIP '05

  31. Y X 1 2 Zhu & Hanson ICIP '05

  32. Y X 1 2 3 Zhu & Hanson ICIP '05

  33. Y X 1 2 3 4 Zhu & Hanson ICIP '05

  34. Y X 1 2 3 4 5 Zhu & Hanson ICIP '05

  35. Y X 1 2 3 4 6 5 Zhu & Hanson ICIP '05

  36. Y X 1 2 3 4 6 5 7 Zhu & Hanson ICIP '05

  37. Y X 1 2 3 4 6 5 7 8 1 2 3 4 6 5 7 Zhu & Hanson ICIP '05

  38. Y X 1 2 3 4 6 5 7 8 9 1 2 3 4 6 5 7 Zhu & Hanson ICIP '05

  39. Y X 1 2 3 4 6 5 7 8 9 10 1 2 3 4 6 5 7 Zhu & Hanson ICIP '05

  40. Y X 1 2 3 4 6 5 7 8 9 10 11 1 2 3 4 6 5 7 Zhu & Hanson ICIP '05

  41. Y X 1 2 3 4 6 5 7 8 9 10 11 12 1 2 3 4 6 5 7 Zhu & Hanson ICIP '05

  42. Y X 1 2 3 4 6 5 7 8 9 10 11 12 1 2 3 4 6 5 7 • 32 cameras (images) per column – spatial “scan” • 48 columns (12 feet)- 4 inch separation in temporal “scan” direction • 1536 virtual cameras (images) with 2D scans • Different viewpoints! Zhu & Hanson ICIP '05

  43. UVIS: A Mosaicing example with 2D scan 1D array of 4 cameras ; car moves First pass: mosaicing along the columns (4 cameras) Second pass: mosaicing in the motion direction Zhu & Hanson ICIP '05

  44. One camera to N stereo mosaics • Very near range view • Distortion removal • Motion estimation • Stereo mosaicing • anomaly detection Zhu & Hanson ICIP '05

  45. UVIS: Dynamic Stereo Mosaics 6 mosaics with changing viewing directions Stereo mosaics: Each pair of 6 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo Zhu & Hanson ICIP '05

  46. UVIS: Dynamic Stereo Mosaics 6 mosaics with changing viewing directions Stereo mosaics: Each pair of 6 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo Zhu & Hanson ICIP '05

  47. UVIS: Dynamic Stereo Mosaics 6 mosaics with changing viewing directions Stereo mosaics: Each pair of 6 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo Zhu & Hanson ICIP '05

  48. UVIS: Dynamic Stereo Mosaics 6 mosaics with changing viewing directions Stereo mosaics: Each pair of 6 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo Zhu & Hanson ICIP '05

  49. UVIS: Dynamic Stereo Mosaics 6 mosaics with changing viewing directions Stereo mosaics: Each pair of 6 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo Zhu & Hanson ICIP '05

  50. Summary • Stereo Mosaics with Oblique Parallel Projections • Wide FOV • Occlusion Rep. • Stereo pairs • Research Issues • Orientation Estimation • Ray Interpolation ( Interframe match) • 3D reconstruction and Moving Target Detection • Real Applications • Airborne Surveillance • Ground Robot Navigation • Under-Vehicle Inspection Zhu & Hanson ICIP '05

More Related