Photo Tourism

1. Photo Tourism Exploring Photo Collections in 3D

2. Introduction • The internet has become a vast, ever-growing repository of visual information about our world. • Virtually all of the world’s famous landmarks and cities have been photographed many different times, both from the ground and from the air.

3. Millions of Images • There are billions of photographs on the Internet. • Representing an extremely large, rich, and nearly comprehensive visual record of virtually every famous place on Earth. Internet 2012 in numbers • 7 petabytes • How much photo content Facebook added every month. • 300 million • Number of new photos added every day to Facebook. • 5 billion • The total number of photos uploaded to Instagram since its start, reached in September 2012. • 58 • Number of photos uploaded every second to Instagram. • 1 • Apple iPhone 4S was the most popular camera on Flickr.

4. Opportunity • Enormous opportunities, both for research and for practical applications. • Mining the collections to create the ultimate virtual experience that could be extremely visually compelling, giving us the ability to walk around in a photorealistic 3-D version of the scene, to let us dial in any time of day or year, and to let us revisit different events.

5. Goal • To design a novel system for registering large sets of photos and exploring them in a 3D browser, that provides: • Accurate 3D reconstruction. • Geometric and semantic scene structure. • Interactive scene visualization system. • Interesting views, and segmenting. • Individual labeled objects.

6. Goal

7. The Concept • The system is based on the idea of using camera pose (location, orientation, and field of view). • First the system computes feature correspondences between images, using descriptors that are robust with respect to variations in pose, scale, and lighting. • Then runs an optimization to recover the camera parameters and 3D positions of those features.

8. Methods • SIFT • Vocabulary Trees. • SFM – Structure From Motion.

9. Methods • SIFT • Vocabulary Trees. • SFM – Structure From Motion.

10. SIFT

11. Methods • SIFT • Vocabulary Trees. • SFM – Structure From Motion.

12. Vocabulary Trees

13. Building The Tree

14. Building The Tree

15. Building The Tree

16. Building The Tree

17. Methods • SIFT • Vocabulary Trees. • SFM – Structure From Motion.

18. Structure From Motion • The problem: Given optical flow or point correspondences, compute 3-D motion (translation and rotation) and shape (depth).

19. Structure From Motion • Input: The positions of “P” points in “F” frames (F>=3). • Output: • point in the 3D world, for all the “P” points in the “F” frames. • and translation vectors from the 3D world to the 2D world (“F” frames). • .

20. Structure From Motion

21. Assumptions • The positions of “P” points in “F” frames (F>=3), which are not all coplanar, and have been tracked. • The entire sequence has been acquired before starting (batch mode). • Camera calibration not needed, if we accept 3D points up to a scale factor.

22. Feature Points

23. Orthographic Projection - point in the 3D world. Orthographic projection

24. How to find Translation? Rank of S is 3, because points in 3D space are not co-planar.

25. Rank Theorem The registered measurement matrix is at most of rank three. Because is a product of two matrices. The maximum rank of S is 3

26. Singular Valued Decomposition is diagonal, are orthogonal Approximate Rotation Matrix. Approximate Shape Matrix.

27. Note: • This decomposition is not unique. • is any 3X3 invertable matrix. • That is because we don’t care if the 3D model is scaled or rotated.

28. Where are we?

29. Additional Applications • Image Morphing. • Annotating Objects. • Classifying Photos.

30. Additional Applications • Image Morphing. • Annotating Objects. • Classifying Photos.

31. Image Morphing

33. Camera Transitions

34. Additional Applications • Image Morphing. • Annotating Objects. • Classifying Photos.

35. Annotating Objects • The user can select a region of an image and enter a text annotation. That annotation is then stored, along with the selected points, and appears as a semi-transparent rectangular box around the selected points. • When an annotation is created, it is automatically transferred to all other relevant photographs.

36. Annotating Objects • Sometimes the system also uses simple heuristics to determine if an annotated region is included.

37. Annotating Objects • If the annotation is visible in image and the apparent size is greater than 0.05 (to avoid barely visible annotations), and less than 0.8 (to avoid annotations that take up the entire image), the system transfers the annotation.

38. Additional Applications • Image Morphing. • Annotating Objects. • Classifying Photos.

39. Classifying Photos • The system also allows users to classify photos into different categories, such as day and night. • The systemcan then create controls for changing the appearance of a scene by toggling between categories of photos.

40. References • "Photo tourism: Exploring photo collections in  3D," Noah Snavely, Steven M. Seitz, Richard Szeliski, ACM Transactions on Graphics (SIGGRAPH Proceedings), 25(3), 2006, 835-846. • "Modeling the world from Internet photo collections," Noah Snavely, Steven M. Seitz, Richard Szeliski,International Journal of Computer Vision (to be published). • Scene Reconstruction and Visualization From Community Photo CollectionsNoah Snavely, Ian Simon, Michael Goesele, Richard Szeliski, and Steven M. Seitz. • Building Rome in a DaySameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard SzeliskiInternational Conference on Computer Vision, 2009, Kyoto, Japan. • “Scalable Recognition with a Vocabulary Tree”,David Nist´er and HenrikStew´enius, Center for Visualization and Virtual EnvironmentsDepartment of Computer Science, University of Kentucky. • “Multiview Structure from Motion in Trajectory Space”AamerZaheer, IjazAkhter, Mohammad HarisBaig, ShabbirMarzban, SohaibKhan, ICCV 2011 • “Nonrigid Structure From Motion”,Yaser Sheikh and Sohaib Khan, ECCV 2010 TUTORIAL. • “A Vocabulary Tree for Image Classification: Open Source Implementation, Validation and Characterization”, Master in Computer Vision and Artificial Intelligence, Report of the research project, Author: Sergi Rubio Manrique, Advisor: Ricardo Toledo.