380 likes | 1.52k Views
Sky Finder: Attribute-based Sky Image Search Litian Tao Lu Yuan Jian Sun SIGGRAPH 2009 Hong Kong University of Science and Technology Beihang University Microsoft Research Asia Difficult to capture a beautiful sky Dynamic rang of scene > dynamic rang of camera
E N D
Sky Finder: Attribute-based Sky Image Search Litian Tao Lu Yuan Jian Sun SIGGRAPH 2009 Hong Kong University of Science and Technology Beihang University Microsoft Research Asia
Difficult to capture a beautiful sky • Dynamic rang of scene > dynamic rang of camera • Good timing is very important
Need to search a sky image • Sky replacement • Background images for composition • 2D design, film production, image editing
Frustrating using search engines Bing Image Search Google Image Search
Our System • Search desired sky images from a sky dataset 500,000 sky images from Flickr
Related Work – Image as Query • Scene Completion [Hays and Efros 2007] • Face Swapping [Bitouk et al. 2008] • Photo Clip Art [Lalonde et al. 2008]
Related Work – Text as Query • Semantic Photo Synthesis [Johnson et al. 2006]
Challenges • Image-query based approahces • Finding a good image example is also a search problem. • Difficult for interactive search in a large dataset • Text -query based (Semantic Photo Syn/Image Search E) • Only can find images including sky • Without further control
Our solution - Attribute based search • Semantic attributes • Category • Layout • Horizon height • Sun existence/position • Richness • A large sky database
Data Collection • Images from Flickr. com • High quality photos • Human-labeled tags sky cloudy sunset sunrise storm colourskies southfloridasky beautyofsky … sky&clouds 5 keywords 95 user groups 1.3 million images
Attributes – Category • 3 Categories • blue-sky • cloudy-sky • sunset • 2000 training images blue-sky cloudy-sky sunset uncertain
Bag-of-Word Representation Feature: SIFT + Color Step1: Codebook Generation Feature Space Partition Feature Quantization Randomized Forests [Moosmann et al. 2006] 2:Representation bag of codewords
Attributes – Category cloudy-sky SVM blue-sky SVM sunset SVM bag of blue-sky words blue-sky score bag of cloudy-sky words cloudy score bag of sunset words sunset score
Attributes • Category • Layout • Horizon height • Sun existence/position • Richness Defined on sky region segmentation
Sky Region Segmentation Test Training Three Sky/non-sky pixel classifiers blue-sky cloudy-sky Graph-cut segmentation sunset
Attributes – Layout & Horizon full-sky object-in-sky landscape normal-sky others A>95% 95%>A>70% A<70% : Bounding Box –cover 95% sky pixels # sky pixels A : in the bounding box # pixels : Horizon height
Attributes – Sun luminance channel magenta channel in CMYK sunset image Bright Region Extraction Shape filter sun mask
Attributes – Richness Images Sky region Edge map Richness
Attribute Accuracy • Test dataset • 6, 000 images • Performance Precision = Recall = # trueclassifiedblue-sky images # true classifiedblue-sky images # classifiedblue-sky images # total blue-sky images
Experimental Evaluation • Test dataset • 6, 000 images • Performance Precision = in blue-sky Recall = in blue-sky # true detectedsky pixels # truedetectedsky pixels Both sky and non-sky region are gloomy # detectedsky pixels # total sky pixels
Experimental Evaluation • Test dataset • 6, 000 images • Performance Precision = in sunset Recall = in sunset # true detectedsuns # truedetectedsuns # detectedsuns # total suns Sun is largely occluded by clouds
Color based re-ranking • Sky color representation • Color signature: • Similarity • Earth Mover’s Distance (EMD) (a) sunset + landscape + horizon + sun position after color-based re-ranking (b)
User Interface Category SVM scores (3D)
User Interface Horizon and sun canvas Category Triangle blue-sky Layout Richness landscape cloudy-sky sunset normal-sky full-sky after PCA object-in-sky others
Path Search • How to find such a sky image? Our solution: sky graph + smoothed path ?
Graph Construction • Building a graph is difficult • Pairwise distance computation is expensive • Semantic metric is required • Sparse graph using attributes • 1: use categoryand richness attributes -> 2000 candidates • 2: re-rank candidates by color ->Top 200 neighbours • 3: use color similarity (EMD) as edge weight
Path Search max-transition-cost = 2 • Finding a path • Shortest path • Our path 2 B C 2 2 A D 5 max-transition-cost = 5
Sky Replacement • Need to change the foreground color • Sometimes not visually plausible • Our method: category-specific color transfer Apply learned transfer from(s0,s1) to foreground original image retrieved image blue-sky a sky_o sky b ? non-sky_o non-sky cloudy-sky
Sky Replacement • Direct cut-and-paste? • Sometimes not visually plausible • Need to change the foreground color • How to compute the color transfer variables? Apply b on foreground original image retrieved image a sunset sky sky b ? non-sky non-sky
Summary • Semantic level search • Semantic attributes • Intuitive user interface • Path finding • Very efficient and scalable • Image search -> attributes -> text search
On-line Demo • http://jiansun5/SkyFinderEntry/
Sun in blue-sky category • Ratio of images containing sun in three categories • blue-sky: 0.9% • cloudy-sky: 10.0% • sunset: 25.6%.
Other attributes • Common attributes in our half a million dataset • Category, layout, horizon height, sun, richness • Other potential attributes: • Lightening • Moon • Polar light • … • We will add these attributes as the database size grows.