270 likes | 463 Views
Automatic Thumbnail Cropping and Its Effectiveness. Bongwon Suh, Haibin Ling, Benjamin B. Bederson, and David W. Jacobs Human Computer Interaction Lab. University of Maryland at College Park. UIST 2003 November 4th, 2003. Problem and Motivation.
E N D
Automatic Thumbnail Cropping and Its Effectiveness Bongwon Suh, Haibin Ling, Benjamin B. Bederson, and David W. Jacobs Human Computer Interaction Lab. University of Maryland at College Park UIST 2003 November 4th, 2003
Problem and Motivation • Thumbnails require significant amount of space on the screen • Small thumbnails are less useful • Make small thumbnails more recognizable and useful PhotoMesa (Zoomable Image Browser) Microsoft Windows Explorer Small screen devices Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Thumbnail Cropping • Image Shrinking (Plain Thumbnail) • We lose detailed information • Image Cropping • We lose a part of information • Select the portion of maximal informativeness • When cropping, keep the more informative part and cut less informative part • Preserve the recognizability of important objects in thumbnails Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Thumbnail Cropping Example • Crop first, then shrink the cropped images Original Image Shrinking (Subsampling) Cropped Image Generated Thumbnails Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
More Examples • Both sets use the same amount of screen space Plain Thumbnails Cropped Thumbnails Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Automatic Thumbnail Cropping • Automate Thumbnail Cropping Procedure • Which part is more informative? • Need to measure informativeness • Saliency based thumbnail cropping • Improve cropping by using dynamic threshold • Face detection based thumbnails cropping • Applying existing techniques as an example of detecting semantic information in images Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Saliency Based Thumbnail Cropping • Saliency • Visual attention model (color, intensity, etc) • Itti and Koch (1998, 1999) • Does not need prior knowledge on images • Assumption • Saliency More informativeness Original Image Computed Saliency Map Cropped Image Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Face Detection Based Thumbnail Cropping • When semantic information of images can be detected, more efficient cropping is possible • Face Detection: Schneiderman and Kanade (2000) Thumbnails generated with three techniques Original Image Face Detection Face Detection Based Cropping Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Saliency Based Cropping • Brute force algorithm • Find a minimum size rectangle that contains a certain portion (threshold) of total saliency • Static threshold algorithm • Require exhaustive search Find a minimum rectangle R satisfying: Sum of saliency values inside R Threshold λ≥ Sum of all saliency values in image Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Using a static threshold Increasing cropping bounds until it contains X% of total saliency Keep adding the next most salient point outside the current cropping bounds Greedy Algorithm Points with Weak Saliency Current Bounds New Bounds The most salient point outside the current bounds Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Dynamic Saliency Threshold • Scattered saliency • Need to cut out little • Gathered saliency • Larger cutting is possible • The most effective threshold varies from images to images Gathered saliency Scattered saliency Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Find a point of diminishing returns Adding small amounts of saliency requires a large increase of the cropping bounds Binary search for maximum gradient point Cropping With Dynamic Threshold Maximum gradient point Cropping Area Sum of Saliency Values inside Area Area-Saliency Sum Graph Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
0.8 Static And Dynamic Threshold : Maximum Gradient Point Cutting out too little Cutting out too much 0.8 Result using Dynamic Threshold Area-Saliency Sum Graph Original Image Static threshold 80% Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Works well with Corbis images Professionally prepared and already cropped Immune to already cropped images Not perfect but generates reasonable results Landscape, scenery Artistic images Robust Algorithm Thumbnail Cropping with Dynamic Threshold Original Image Some core parts has been cut out Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
User Study Design • Participant • Twenty students recruited on campus • Task • Recognition Task • Visual Search Task • Image Set • Animal Set: Common objects • Corbis Set: Professionally prepared photos • Face Set: Well known figures e.g. Entertainers • Thumbnail Technique • No cropping • Saliency based cropping • Face detection based cropping Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Recognition Task • To measure the effect of thumbnail techniques on object recognition • Target images were shown for two seconds • Participants were asked to click what they saw. • Measured recognition accuracy • # of right answers / # of total tasks Face Set Animal Set Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Recognition Task Hypothesis Big enough to be recognized in both cases Thumbnails are too small anyway Meaningful 100% Recognition Rate Thumbnail Technique A Thumbnail Technique B Difference ? Thumbnail Size Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Recognition Task Result • All curves are different from each others. (p < 0.01) Animal Set Face Set Face Detection Based Cropped Saliency Based Cropped No Cropping Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Visual Search Task • Find an image that matches a given task description • Verbal description (except faces) • PhotoMesa interface • Measured browsing completion time Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
No cropping vs. Saliency Based Cropping F(1, 190) = 3.82 p = 0.05 Three Thumbnail Techniques on Face Set F(2, 87) = 4.56 p = 0.013 Visual Search Task Result Browsing Time Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Future Work • Analysis on broad sets of large image corpus • Currently tested on about one thousand images • Performance Issues • Thumbnail cropping is implemented in Matlab Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Conclusion • Saliency Based Thumbnail Cropping • General technique that does not make assumption about images • Does not need humans’ intervention • Dynamic threshold can prevent cutting out too much or too little • Face Detection Based Thumbnail Cropping • Shows how semantic information can be used to enhance the thumbnail cropping Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Fixed Threshold Brute Force, 90% Greedy, 80% Too big threshold (80% threshold) Cutting out too less Result Brute Force Binary Dynamic Brute Force Gradient Dynamic Greedy Binary Dynamic Greedy Gradient Dynamic Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Brute Force, 90% Greedy, 80% Too small threshold (80% threshold) Cutting out too much Result Brute Force Binary Dynamic Brute Force Gradient Dynamic Greedy Binary Dynamic Greedy Gradient Dynamic Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Brute Force, 90% Greedy, 80% Corbis image: Professionally prepared and already cropped. Saliency values are widely scattered. Result Brute Force Binary Dynamic Brute Force Gradient Dynamic Greedy Binary Dynamic Greedy Gradient Dynamic Automatic Thumbnail Cropping and Its Effectiveness UIST 2003
Greedy, 80% Brute Force, 90% The algorithm is not perfect but generates reasonable results. Result Brute Force Binary Dynamic Brute Force Gradient Dynamic Greedy Binary Dynamic Greedy Gradient Dynamic Automatic Thumbnail Cropping and Its Effectiveness UIST 2003