1 / 31

Panorama Stitching and Augmented Reality

This project focuses on panorama stitching and augmented reality techniques, including local feature matching with large datasets, nearest-neighbor matching, k-d tree construction and query, bundle adjustment, multi-band blending, automatic straightening, gain compensation, and 3D structure and virtual object placement. Examples include identifying panoramas and objects in image sets, products in a supermarket, and locations for robot localization or augmented reality.

oliveras
Download Presentation

Panorama Stitching and Augmented Reality

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Panorama Stitching and Augmented Reality

  2. Local feature matching withlarge datasets • Examples: • Identify all panoramas and objects in an image set • Identify all products in a supermarket • Identify any location for robot localization or augmented reality

  3. Matching in large unordered datasets

  4. Matching in large unordered datasets

  5. Nearest-neighbor matching • Solve following problem for all feature vectors, x: • Nearest-neighbour matching is the major computational bottleneck • Linear search performs dn2 operations for n features and d dimensions • No exact methods are faster than linear search for d>10 • Approximate methods can be much faster, but at the cost of missing some correct matches. Failure rate gets worse for large datasets.

  6. 4 6 l1 7 8 5 9 10 3 2 1 11 2 5 4 11 8 1 3 9 10 6 7 K-d tree construction Simple 2D example l9 l1 l5 l6 l3 l2 l3 l2 l10 l4 l5 l7 l6 l8 l7 l4 l8 l10 l9 Slide credit: Anna Atramentov

  7. l1 q 2 5 4 11 8 1 3 9 10 6 7 K-d tree query 4 6 l9 l1 7 l5 l6 l3 8 l2 l3 l2 5 9 10 3 l10 l4 l5 l7 l6 l8 l7 2 1 l4 11 l8 l10 l9 Slide credit: Anna Atramentov

  8. Approximate k-d tree matching Key idea: • Search k-d tree bins in order of distance from query • Requires use of a priority queue

  9. Fraction of nearest neighbors found • 100,000 uniform points in 12 dimensions. Results: • Speedup by several orders of magnitude over linear search

  10. Panorama stitching (with Matthew Brown)

  11. Panorama stitching (with Matthew Brown)

  12. Bundle Adjustment • New images initialised with rotation, focal length of best matching image

  13. Bundle Adjustment • New images initialised with rotation, focal length of best matching image

  14. Multi-band Blending • Burt & Adelson 1983 • Blend frequency bands over range l

  15. 2-band Blending Low frequency (l > 2 pixels) High frequency (l < 2 pixels)

  16. Linear blending Multi-band blending Multi-band Blending

  17. Automatic Straightening

  18. Automatic Straightening • Heuristic: user does not twist camera relative to horizon • Up-vector perpendicular to plane of camera x vectors

  19. Automatic Straightening

  20. Gain Compensation • No gain compensation

  21. Gain Compensation • Gain compensation • Single gain parameter gi for each image

  22. Panoramas from handheld consumer cameras • Free working demo available: Autostitch • Commercial products: Serif, Kolor, others coming • Show in Java applet: Browser demo

  23. Autostitch usage in www.flickr.com • Over 20,000 panoramas posted by users of free Autostitch demo

  24. Public images from Flickr Surprise: Many users want borders to be visible

  25. Augmented Reality • Applications: • Film production (already in use) • Heads-up display for cars • Tourism • Medicine, architecture, training • What is needed: • Recognition of scene • Accurate sub-pixel 3-D pose • Real-time, low latency

  26. Augmented Reality (David Lowe & Iryna Gordon) • Solve for 3D structure from multiple images • Recognize scenes and insert 3D objects Shows one of 20 images taken with handheld camera

  27. System overview

  28. 0 iterations: error = 62.5 pixels 10 iterations: error = 4.2 pixels 50 iterations: error = 0.2 pixels 20 iterations: error = 1.7 pixels Bundle adjustment: an example 20 input images

  29. Incremental model construction • Problems: • computation time increases with the number of unknown parameters • trouble converging if the cameras are too far apart (> 90 degrees) • Solutions: • select a subset of about 4 images to construct an initial model • incrementally update the model by resectioning and triangulation • images processed in order determined by the spanning tree

  30. 3D Structure and Virtual Object Placement • Solve for cameras and 3D points: • Uses bundle adjustment (solution for camera parameters and 3D point locations) • Initialize all cameras at the same location and points at the same depths • Solve depth-reversal ambiguity by trying both options • Insert object into scene: Set location in one image, move along epipolar in other, adjust orientation

  31. Augmentation Example The virtual object is rendered with OpenGL, using the computed camera parameters.

More Related