1 / 22

Finding Perfect Rendezvous On the Go

Finding Perfect Rendezvous On the Go. Accurate Mobile Visual Localization and Its Applications to Routing. Tao Mei Media Computing Group, MSR Asia. Location: Core of LBS. Image-based localization. Why Image-based localization? GPS may fail (cloudy days, crowd urban scenes)

luke-boyer
Download Presentation

Finding Perfect Rendezvous On the Go

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Finding Perfect Rendezvous On the Go Accurate Mobile Visual Localization and Its Applications to Routing Tao Mei Media Computing Group, MSR Asia

  2. Location: Core of LBS

  3. Image-based localization • Why Image-based localization? • GPS may fail (cloudy days, crowd urban scenes) • Image contain more context information • Existing Methods • Localization by image retrieval - Only get locations of the top candidate images • View direction estimation given GPS - cannot deal with localization and cannot work without initial GPS • Localization by Registration to 3D models - Typically with a small scene ( 100~10K images)

  4. Our Approach • Benefits • Capable for City scale localization( Up to million images) • High accuracy ( localization by view registration to 3D point cloud) • Complete set of parameters ( Camera location, View direction and scene location) Large scale image retrieval 3D reconstruction

  5. Flow Chart

  6. Geo-Visual Clustering • Why we need Geo-Visual clustering? • Images maybe unordered • Public images prone to be noisy • Reconstruction in small clusters instead of the whole dataset will reduce unnecessary computational cost and suitable for parallel computing

  7. Geo-Visual Clustering Geographic Clusters Visual Clusters in a geo cluster Clustering using Affinity Propagation

  8. Building 3D scenes • Building 3D scenes via SfM for each visual cluster • Image clusters • Scene model of the cluster: • A set of 3D points: • A set of camera: • Each : rotation , translation , focal length and Image • Structure-from-Motion/Bundle adjustment • Minimize the re-projection errors

  9. Searching 3D scene models • Large scale image search • Image representation: • SIFT features -> Visual word -> Inverted files • Nearest images search: • BoW histogram intersection • GPS constraint ( 300m within the initial GPS of the query) • Geometric Verification( Spatial Coding, considering the relative spatial layout of the features) • Vote for 3D Scene models • According to the matched feature counts between query and image clusters indicates whether a feature point is matched to the scene

  10. Localization by view registration , • View registration • Find the 2D-to-3D correspondences by SIFT matching • Estimate the Projection(Camera) matrix of the query image • Calculate the camera location, view direction and scene location ,

  11. Localization by view registration • can be solved by: • Traditional 6-point algorithm, solve all , and • 4-point algorithm that can estimate focal length , and , with assumption that = 0 and , be the original of the image. • 3-point algorithm with known focal length (e.g. focal length from EXIF or transferred from other images) , so only and need to be estimated. The DOF is reduces to 6. • can be decomposed to: • 3 X 3 intrinsic matrix • 3 X 3 rotation matrix • 3 X 1 translation vector 1 1. K. Josephson and M. Byrod. Pose estimation with radial distortion and unknown focal length. In CVPR, 2009.

  12. Application UI design • Single user self-localization

  13. Application UI design • Collaborative localization for finding rendezvous

  14. Application UI design • Rendezvous for photographing Where to shoot such a photograph? Suggest the route to the scenery spot and other parameters( e.g. view direction)

  15. Experiments • Settings: two large datasets • 1.2 million San Francisco Street views(SF street view) • 0.6 million FlickrImages of five cities: Paris, Beijing, New York, Seattle and San Francisco( Flickr five cities) San Francisco Street view Flickr five cities

  16. Experiments • Objective Evaluations • 100 test queries for SF street view and 50 queries for Flickr five cities • Mean Average Precision(mAP) for image retrieval • Response time • User Study • Usability

  17. Image search mAP • Different Image retrieval strategies • Rough initial GPS will help the retrieval performance • Geometric verification(GV) is essential • Combination of GPS and GV

  18. Localization Errors Statistics Table 1. The error statistics of estimated geographical information of queries in “SF street view” (meters) Table 2. The error statistics of estimated geographical information of queries in “Flickr 5 cities” (meters)

  19. Localization Errors Distribution The distributions of estimated camera location and view direction of 60 successfully registered queries in SF street views

  20. System Response time

  21. Conclusion & Future Work • Contributions: • Accurate localization results in large scale datasets • Complete set of parameters estimation • A variety of mobile applications help the user • Future Work: • Enhance the retrieval and improve the robustness • Leverage compressed image representation to reduce the system latency

  22. Thanks!

More Related