1 / 38

Sky Finder: Attribute-based Sky Image Search

Sky Finder: Attribute-based Sky Image Search Litian Tao Lu Yuan Jian Sun SIGGRAPH 2009 Hong Kong University of Science and Technology Beihang University Microsoft Research Asia Difficult to capture a beautiful sky Dynamic rang of scene > dynamic rang of camera

Audrey
Download Presentation

Sky Finder: Attribute-based Sky Image Search

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Sky Finder: Attribute-based Sky Image Search Litian Tao Lu Yuan Jian Sun SIGGRAPH 2009 Hong Kong University of Science and Technology Beihang University Microsoft Research Asia

  2. Difficult to capture a beautiful sky • Dynamic rang of scene > dynamic rang of camera • Good timing is very important

  3. Need to search a sky image • Sky replacement • Background images for composition • 2D design, film production, image editing

  4. Frustrating using search engines Bing Image Search Google Image Search

  5. Our System • Search desired sky images from a sky dataset 500,000 sky images from Flickr

  6. Related Work – Image as Query • Scene Completion [Hays and Efros 2007] • Face Swapping [Bitouk et al. 2008] • Photo Clip Art [Lalonde et al. 2008]

  7. Related Work – Text as Query • Semantic Photo Synthesis [Johnson et al. 2006]

  8. Challenges • Image-query based approahces • Finding a good image example is also a search problem. • Difficult for interactive search in a large dataset • Text -query based (Semantic Photo Syn/Image Search E) • Only can find images including sky • Without further control

  9. Our solution - Attribute based search • Semantic attributes • Category • Layout • Horizon height • Sun existence/position • Richness • A large sky database

  10. Data Collection • Images from Flickr. com • High quality photos • Human-labeled tags sky cloudy sunset sunrise storm colourskies southfloridasky beautyofsky … sky&clouds 5 keywords 95 user groups 1.3 million images

  11. Attributes – Category • 3 Categories • blue-sky • cloudy-sky • sunset • 2000 training images blue-sky cloudy-sky sunset uncertain

  12. Bag-of-Word Representation Feature: SIFT + Color Step1: Codebook Generation Feature Space Partition Feature Quantization Randomized Forests [Moosmann et al. 2006] 2:Representation bag of codewords

  13. Attributes – Category cloudy-sky SVM blue-sky SVM sunset SVM bag of blue-sky words blue-sky score bag of cloudy-sky words cloudy score bag of sunset words sunset score

  14. Attributes • Category • Layout • Horizon height • Sun existence/position • Richness Defined on sky region segmentation

  15. Sky Region Segmentation Test Training Three Sky/non-sky pixel classifiers blue-sky cloudy-sky Graph-cut segmentation sunset

  16. Attributes – Layout & Horizon full-sky object-in-sky landscape normal-sky others A>95% 95%>A>70% A<70% : Bounding Box –cover 95% sky pixels # sky pixels A : in the bounding box # pixels : Horizon height

  17. Attributes – Sun luminance channel magenta channel in CMYK sunset image Bright Region Extraction Shape filter sun mask

  18. Attributes – Richness Images Sky region Edge map Richness

  19. Attribute Accuracy • Test dataset • 6, 000 images • Performance Precision = Recall = # trueclassifiedblue-sky images # true classifiedblue-sky images # classifiedblue-sky images # total blue-sky images

  20. Experimental Evaluation • Test dataset • 6, 000 images • Performance Precision = in blue-sky Recall = in blue-sky # true detectedsky pixels # truedetectedsky pixels Both sky and non-sky region are gloomy # detectedsky pixels # total sky pixels

  21. Experimental Evaluation • Test dataset • 6, 000 images • Performance Precision = in sunset Recall = in sunset # true detectedsuns # truedetectedsuns # detectedsuns # total suns Sun is largely occluded by clouds

  22. Color based re-ranking • Sky color representation • Color signature: • Similarity • Earth Mover’s Distance (EMD) (a) sunset + landscape + horizon + sun position after color-based re-ranking (b)

  23. User Interface Category SVM scores (3D)

  24. User Interface Horizon and sun canvas Category Triangle blue-sky Layout Richness landscape cloudy-sky sunset normal-sky full-sky after PCA object-in-sky others

  25. Demo

  26. Path Search • How to find such a sky image? Our solution: sky graph + smoothed path ?

  27. Graph Construction • Building a graph is difficult • Pairwise distance computation is expensive • Semantic metric is required • Sparse graph using attributes • 1: use categoryand richness attributes -> 2000 candidates • 2: re-rank candidates by color ->Top 200 neighbours • 3: use color similarity (EMD) as edge weight

  28. Path Search max-transition-cost = 2 • Finding a path • Shortest path • Our path 2 B C 2 2 A D 5 max-transition-cost = 5

  29. Sky Replacement • Need to change the foreground color • Sometimes not visually plausible • Our method: category-specific color transfer Apply learned transfer from(s0,s1) to foreground original image retrieved image blue-sky a sky_o sky b ? non-sky_o non-sky cloudy-sky

  30. Sky Replacement • Direct cut-and-paste? • Sometimes not visually plausible • Need to change the foreground color • How to compute the color transfer variables? Apply b on foreground original image retrieved image a sunset sky sky b ? non-sky non-sky

  31. Demo

  32. Summary • Semantic level search • Semantic attributes • Intuitive user interface • Path finding • Very efficient and scalable • Image search -> attributes -> text search

  33. On-line Demo • http://jiansun5/SkyFinderEntry/

  34. Questions?

  35. Sun in blue-sky category • Ratio of images containing sun in three categories • blue-sky: 0.9% • cloudy-sky: 10.0% • sunset: 25.6%.

  36. Other attributes • Common attributes in our half a million dataset • Category, layout, horizon height, sun, richness • Other potential attributes: • Lightening • Moon • Polar light • … • We will add these attributes as the database size grows.

More Related