Geometry Directed Browser for Personal Photographs

1. Geometry Directed Browser for Personal Photographs Center for Visual Information Technology IIIT Hyderabad Aditya Deshpande, Siddharth Choudhary, P J Narayanan, Kaustav Kundu, Krishna Kumar Singh, Aditya Singh, Apurva Kumar

2. Digital Photography • - No hard copy - Capture photographs and relive later on display device • Photo-Browsers are tools to view digital photographs. E.g. Windows Photo Viewer, iPhoto, FSpot, KSquirrel etc. • Photo Browsing model has notevolved much. Photo-Browsing We use SfM and other 3D computer vision techniques to provide intuitive Geometry Directed Photo Browsing.

3. Face Detection & Tagging on Social Networking Sites. • [Zhang et al. MM’03], Automatic annotation of family albums. • [Davis et al. MM’05], Additional contextual data viz. time of capture, geo-tag, indoor/outdoor scene, co-occurring faces. Related Work Above techniques only improve photo-browsing experience of social engagements.

4. Apart from social engagements, a large chunk of user’s personal photographs consist of tourist places & monuments. • [Snavely et al. IJCV’08, SIGGRAPH’06] (Photosynth)- CPC Storage, local reconstruction to add new camera’s • Choudhary et al., Li et al., Sattler et al., Irschara et al. etc. - Localize new query images w/o exhaustive search. Our Goal We combine SfM-Reconstruction + Localization to provide intuitive browsing of user photos in 3D space of the monument.

5. Our target platform is an off-the-shelf laptop or a desktop. • User is expected to click around 5-50 photographs for a particular monument. • The system should localize these user photographs in areasonable time. • The system should provide a smooth visualization / transitions of all user photos and ~105 points of the monument. Assumptions

6. (1) Heavy SfM Reconstruction done offline in the cloud (3) User uploads personal photo’s through a camera / phone System Design (4) System registers user’sphotos to the point cloud and provides 3D visualization. (2) GDBPackage : reconstruction + addnl. information downloaded to local disk

7. GDBPackage System Block Diagram 1 2 RegistrationModule VisualizationModule Estimated Camera’s User Photos System is divided in two parts : 1. Registration / Localization Module2. Visualization Module

8. Trivial if photograph is taken from GPS enabled deviceand is geo-tagged! • What if no geo-tag information? • Two Localization Approaches :Image based search in a geo-tagged Image Dataset[Panda et al.] Geo-locate digital heritage site photos.Using structure information in SfM Dataset[Irschara et al. CVPR’09], match to nearby similar images.[Li et al. ECCV’10], visibility prioritized 3D-2D matches. [Sattler et al. ICCV’11, ECCV’12], visual words to find 2D-3D matches. Localizing User Photo’s

9. [Choudhary et al. ECCV’12]- Triangulate a seed point in the user photograph.- Further 3D-2D search is guided by visibility probabilities.- Find ~20 independent matches.- Use RANSAC to estimate camera parameters. Localization - Choudhary et al. Up Vector View Direction 3D Position Probability Guided 3D-2D correspondence

10. Data for Localization is stored in GDBPackage : (1) Cover Set (2) Visibility Matrix (3) Bi-Partite Visibility Graph • CPC images need not be stored, data requirements are minimal. • The method is fast and localizes images at the rate of 1sec/photo. Advantages of Localization Method

11. In some cases the images lack sufficient monument geometryfor localization to work :- Occluded by people.- Noisy images of nearby scenery/smaller monuments.- Zoomed in images of smaller monument structures etc. Non-Localizable Photographs Zoomed In View of Small Structure(Pantheon Dataset) Completely Occludedby People(Colosseum Dataset)

12. Photographs have time of capture stored in their EXIF-tags. • A non-localized image is placed at a position that is weightedaverage of its immediate known predecessor and immediateknown successor in time. • Similarly, linear interpolation is also done for the view-direction vector to get the complete camera pose. • The above method will not give the exact location, but placing it in temporal neighborhood suffices for display purposes. Non-Localizable Photographs

13. 3D Viewer • Mouse Navigation • Button Navigation • Add Screenshot • Delete Path • Generate Photo-Tour • 2D Viewer Visualization Module

14. Initial Mode : 3D Model and small preview (thumbnails) of • user photographs. • Select Mode : Animate to clicked photo and detailed view. • Linear quaternion interpolation of Rotation Matrix for smoothtransitions between images. • Smooth transitions give a feel of the geometric space of the monument. 3D Photo Browser : Geometry Directed Photo-Browsing

15. User can save the current viewpoint (“Add Screenshots”) • Once a set of viewpoints are saved, he can smoothly animateover viewpoints. (“Generate Photo-Tour / Animate Path”) • User can delete the viewpoints and generate a new photo-tour. • Photo-Tours are a good way to creatively view personalphoto’s taken at a tourist place. 3D Photo Browser : Generating Custom Photo Tours

16. Results (a) Localization Module (b) Visualization Module

17. Minimal System Requirements. • Intuitive 3D Visualization of User Photographs. • Pipeline for 3D personal photo-viewing from SfM reconstruction. • Port our system to a mobile phone and have a touch/gesture interface. Conclusion and Future Work 3D Photo-Viewing & Localization App

18. Thank you. Questions?More Results (a) Hampi Dataset (Stone Chariot) (b) Pantheon Dataset

19. Platform Details