1 / 31

Learning To Tag

Lei Wu , Linjun Yang, Nenghai Yu, Xian- Sheng Hua University of Science and Technology of China Microsoft Research Asia. Learning To Tag. Motivation. Motivation. Image Search. Query. Annotations. Images. Search Result. Text Based: Annotation by surrounding text. Content Based:

perdy
Download Presentation

Learning To Tag

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lei Wu, Linjun Yang, Nenghai Yu, Xian-ShengHua University of Science and Technology of China Microsoft Research Asia Learning To Tag

  2. Motivation

  3. Motivation

  4. Image Search Query Annotations Images Search Result Text Based: Annotation by surrounding text Content Based: Annotation by the content of images Social Based: Annotation by users in a social network Automatic Automatic Manual

  5. Why Need Tag Recommendation? • Three Issues: Tags are • Ambiguous • Incomplete • Noisy Recommendation: Sea Ocean Water Sky Cloud Island Coast Sea Ocean

  6. Why Need Tag Recommendation? • Three Issues: Tags are • Ambiguous • Incomplete • Noisy Red Apple, Fruit Apple • Possible Tags: • Apple • Fruit • Red • Corporation • Logo • Products Apple Corp. Logo Apple Ambiguous Incomplete

  7. Problems with Social Tagging Social Tagging Different keywords Ambiguous keywords Ignore facts Typing error

  8. Tag Distribution on Flickr Tags are noisy

  9. Tag Recommendation • What is Tag Recommendation • Given one or more initial tags for an image • Provide a list of possible tags automatically (ordered by relevance scores) • Advantages of Tag Recommendation • Enable fast tagging • Enable high-quality tagging • Higher Relevance/Accuracy • Wider Coverage • Less noises

  10. Current Methods • Recommendation based on collective knowledge [www2008] • Drawback: • Cannot deal with synonym • i.e. “soccer” == “football” • Cannot deal with polysemy • i.e. “apple” fruit != “apple” logo • Cannot deal with meronymy • i.e., car vs. wheel • Target image is not taken into account (image independent) • Different images with the same initial tags will get the same recommended tag list

  11. Learning To Tag • Correlation Measure Tag Visual Textual Visual Language Model Tag Co-occurrence Flickr Distance /Tag Content Correlation Image Conditioned Similarity (ICS)

  12. Learning to Tag • Three Features to Tag Correlations • Tag Concurrence (TC) • The same as previous work • Tag Content Correlation (TCC) • To solve the first three issues (synonym, polysemy & meronymy) • Image-Conditioned Tag Correlation (ITC) • To solve the fourth and second issue (image independent, & polysemy)

  13. Learning To Tag ITC TCC TC

  14. Learning To Tag • Correlation Measure Tag Visual Textual Visual Language Model Tag Co-occurrence Flickr Distance /Tag Content Correlation Image Conditioned Similarity (ICS)

  15. Similarity in Visual Domain • Visual Language Modeling (Lei et al. MIR07)

  16. Learning To Tag • Correlation Measure Tag Visual Textual Visual Language Model Tag Co-occurrence Flickr Distance /Tag Content Correlation Image Conditioned Similarity (ICS)

  17. Tag Content Correlation Based on “Flickr Distance” or “PicNet Distance” – ACM MM 09 Best Paper Candidate Can handle concurrency, synonym, polysemy & meronymy Concept A: Airplane Concept B: Airport Tag search in Flickr LT-VLM Concept Model A Concept Model B Jensen-Shannon Divergence Flickr Distance (A, B)

  18. Similarity in Visual Domain • Tag Content Correlation (TCC) • Inverse of Flickr Distance

  19. Learning To Tag • Correlation Measure Tag Visual Textual Visual Language Model Tag Co-occurrence Flickr Distance /Tag Content Correlation Image Conditioned Similarity (ICS)

  20. Image Conditioned Similarity(ITC) related Apple Pear Test Image

  21. Image-Conditioned Tag Correlation Based on Visual Language Model (VLM) Visual Words …… VLM of Tag 1 VLM of Tag 2 VLM of Tag N …… Projection 1 Projection 2 Projection N New Features for Tags (Image Dependant)

  22. Image Conditioned Similarity(ITC) ITC

  23. Learning To Tag • Our Approach ITC TCC TC

  24. Recommendation • Ranking Features • Loss Function

  25. Algorithm • Final Ranking

  26. Results Tag Concurrence Only Tag Concurrence & Tag Content Tag Concurrence & Tag Content & Image-Cond. Correlation

  27. Multi-domain relevance V.S.Tag Cooccurrence • Evaluation

  28. Rankboost V.S. linear combination

  29. Parameter influence

  30. Summary • Tagging may be the final solution for visual understanding • For both search and advertising • Learning2Tag provides an efficient tagging scheme • Which ensures high-quality tagging results • It can also work as a “tag completion” approach • Automatically expands tags for multimedia content • It is beyond the use for tagging • It can also work for general image annotation combined with active learning.

  31. Thanks. Any questions?For further discussion, please send email to leiwu@live.com or visithttp://wuleibig.googlepages.com

More Related