Creating Photo Sequences Using Angled Graphs

1. Creating Photo Sequences Using Angled Graphs J. Armour MAS 531 Final Project Fall 2010

2. Main Idea Input: Unsorted Photo Collection • Large photo collections of the same event or the same object or place are common with the proliferation of camera phones, digital photography, and easy storage. • Imagine being able to take such a data base and automatically sort them and output a smooth transition movie of that object or event. Output: Smooth Image Sequence Calculate: Angled Graph

3. Additional Applications • Sporting event imagery. • Paparazzi photography. • Tourist or security camera photography of same location. • Data taken of the same scene with varying time of day, time of year, weather conditions, dynamic content, etc. • Use this method to decompose a video and remove camera jitter, but keep the appearance of smooth transitions frame to frame. • Use the graph to find circuits, creating infinite video loops (specific data sets needed with repeative motion, nearly identical backgroud. Example: flying bird) MIT Dome, Summer vs. Winter

4. Current Methods • Photo Tourism work uses online image collections along with structure from motion and planar proxy techniques to generate an browse-able photo collection. • Method first builds and understanding of a scene and then picks out sequences of images based on a series of selected camera poses. • SIFT, KLT, SURF, and GIST methods can all be used for matching images based on features. • Methods are computationally intensive and not robust to certain changes in imagery, for example changes in viewing angle > 20 degrees. • Isomap, PCA, and MDS methods, involving image manifolds, have also been used to work with image sorting. • Future work needed comparing angled graph method to these previous methods .

5. Overview of the Process • Step 1: Determine image feature vectors based on image histogram, gradients, colors, etc. • Step 2: Use feature vectors to calculate weighted graph of image distances. Given image m,n, have Fm, Fn , then: • Step 3: Calculate angles between adjacent edge of weighted graph. • Step 4: Convert angles and edge pairs to a new weighted graph, G* • Step 5: Threshold the G* to reduce the number of possible paths in the data, keeping only the “straightest” paths • Step 6: Calculate possible minimal paths in remaining data. • Step 7: Turn straightest path into an image sequence.

6. Contributions • We introduce a novel graph structure to summarize the relationship between sets of 3 images and the smoothness of their transitions. • We use this information to compute global optimal paths in the imagery starting from a given image, forming a image sequences. • Algorithm involves entirely image-based processing. • Does not reconstruct structure, shape, or geometry. • Method is near real-time. • Can handle non-geometric changes such as camera focus. Sample Weighted Graph G* • Extracts paths through images (camera poses, lighting conditions, etc.) without expensive pre-processing steps to build scene understanding. • Can handle a 360⁰ angular viewing of objects.

7. Further Thoughts • What Worked: Implemented the algorithm except step 6. • What Failed: Lack of understanding of shortest path algorithms and the correct way to find global minimum paths. Hoping to have sorted out possible coding errors by Tuesday. • Acknowledgements: • Project idea is 100% derived out of Camera Culture Group SIGGRAPH 2010 submission entitled “Personal to Shared Moments with Angled Graphs of Pictures”. Figure above is their initial vision for the project. • Previous work done by Prof. Gabriel Taubin, Kevin Chiu, Tom Cuypers, and Ramesh Raskar. • My Contribution: Implemented the idea in Matlab based on the rough draft paper, took more data sets for testing, implemented variation on initial

8. Limitations • Scalability could be a problem as method involves computation of n2 image distances, n3 angles from feature vectors of size m x 1 for each of the n images in a data set. • Included figure showing quadratic trend in computation time needed (without the shortest path step implemented) versus number of frames. • Shortest path calculations, in our case Floyd-Warshall, are generally O(n3). • We’ve thresholded the results to try to make up for this, but need to look into the actual cost of this step. • Further testing needed to understand robustness of the method to feature vector contents and weights.

9. Future Work • Modification of system (weights or feature vectors) to allow images from different cameras to be sorted. • Testing for robustness of system to different variations in data: ie lighting conditions, horizontal/vertical camera translation, camera rotation, zoom, focus, differing camera, etc. • Comparison of robustness of the system to that of previous methods. • Comparison of memory and computational cost of system to that of previous method. • Test system on data taken of same object at different angles but with static background. Example: Google Embryo data. • Look at intersections of paths, creating smooth transitions to data varying in different ways (eg. if using stanford face data, find head shaking left to right and smoothly transition into nodding up and down.)

10. Summary • We’ve proposed a new method for sorting images using angular information • This allows us to determine nearest-neighbor relationships. • We use these nearest-neighbors to create videos with smoothest possible image transition from a given data sets. • We suggested some possible applications and future work needed.