Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization

1. Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization Gabriel Takacs, Vijay Chandrasekhar, Thanos Bismpigiannis, Bernd Girod Stanford University Radek Grzeszczuk, Natasha Gelfand, Wei-Chao Chen,Yingen Xiong, Kari Pulli Nokia Research Center, Palo Alto

2. Video Demonstration

3. Outline • System Overview • Image matching on the cell phone • Data Organization • Server groups images by location • Data Reduction • Server clusters, prunes and compresses descriptors • Image Matching Results • Qualitative and quantitative • System Timing • Low latency image matching on the cell phone

4. GPS Server System Overview

5. Prefetched Data Image Matching Query Image SURF-64 Descriptors Ratio Test Matching Affine RANSAC Database Images

6. Data Organization Kernel Loxel

7. FeatureCache Server ANN System Block Diagram Geo-TaggedImages Group Imagesby Loxel ExtractFeatures Cluster Features GeometricConsistency Check Match Images Prune Features Compress Descriptors Loxel-Based Feature Store Network ExtractFeatures ComputeFeature Matches DeviceLocation CameraImage GeometricConsistency Check Display Info forTop Ranked Image

8. Example Feature Cluster Feature Descriptor Clustering • Match all images in loxel • Form graph on features • Cut graph into clusters • Create representative meta-features meta-feature

9. meta-featuresingle feature Feature Descriptor Clustering Images of the same landmark

10. 4x Reduction Database Feature Pruning • Rank images by number of meta-features • Allocate equal budget for each landmark • Fill budget with meta-features by rank • Fill any remaining budget with single features

11. Feature Descriptor Pruning Budget: 200 100 500 All

12. Feature Compression • Quantization • Uniform, equal step-size • 6 bits per dimension • Entropy coding • Huffman tables • 12 different tables • 64-dimensional SURF • 256 bytes uncompressed • 37 bytes with compression Original Feature Compressed Feature Quantization Entropy Coding Sdx Sdy S|dx| S|dy|

13. ZuBuD Dataset (~1000 Images)

14. Stanford Dataset (~2500 Images)

15. Matching Results Query Rank 1 Rank 2 Rank 3 Rank 4

16. ~10 images / loxel ~10 images / loxel ~1000 images / loxel Matching Performance True Matches False Matches

17. Timing Analysis Nokia N95 332 MHz ARM 64 MB RAM 100 KByte JPEG over 60 Kbps Uplink Downloads Upload Upload Geometric Consistency Extract Features Extract Features Feature Matching All on Phone Extract Features on Phone All on Server

18. Conclusions • Image matching on mobile phone • Use loxels to reduce search space • 27x reduction in data sent to phone • Clustering • Pruning • Compression • ~3 seconds for image matching on N95

19. Questions