1 / 21

Descriptive Semantic Image Retrieval

Descriptive Semantic Image Retrieval. David Norton Derral Heath. Motivation. Retrieve an image based on a descriptive query: “Find me an image that is red, dark, scary, and beautiful”. Content-Based Image Retrieval. Retrieve an image strictly from image features color texture shape

meira
Download Presentation

Descriptive Semantic Image Retrieval

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Descriptive Semantic Image Retrieval David Norton Derral Heath

  2. Motivation • Retrieve an image based on a descriptive query: • “Find me an image that is red, dark, scary, and beautiful”

  3. Content-Based Image Retrieval • Retrieve an image strictly from image features • color • texture • shape • General semantic based image retrieval is hard • “Find me a picture of a piranha”

  4. Emotional Semantic Image Retrieval • Query images matching emotional words or word-pairs • “Find me a happy picture” • Usually adjectives • Usually a small orthogonal subset of terms • Query via single words (or pairs)

  5. Descriptive Semantic Image Retrieval • Open to all descriptive words • Query via any number of words • “Find me an image that is red, dark, scary, and beautiful”

  6. Three Components • Extraction of image features • Semantic representation of image • Mapping between visuals and language

  7. Extraction of Image Features • Color (12) • Average RGB values • Color count • Texture (50) • Entropy • Shape (10) • Eccentricity

  8. Extraction of Image Features

  9. Extraction of Image Features

  10. Semantic Representation of Image • How do we obtain a description? • What is a descriptive word? • What are the features?

  11. User Input Interface

  12. Bright has 11 Senses • 1. (17) bright -- (emitting or reflecting light readily or in large amounts; "the sun was bright and hot"; "a bright sunlit room") • 2. (6) bright, brilliant, vivid -- (having striking color; "bright dress"; "brilliant tapestries"; "a bird with vivid plumage") • 3. (5) bright, smart -- (characterized by quickness and ease in learning; "some children are brighter in one subject than another"; "smart children talk earlier than the average") • 4. (3) bright -- (having lots of light either natural or artificial; "the room was bright and airy"; "a stage bright with spotlights") • 5. (1) bright, burnished, lustrous, shining, shiny -- (made smooth and bright by or as if by rubbing; reflecting a sheen or glow; "bright silver candlesticks"; "a burnished brass knocker"; "she brushed her hair until it fell in lustrous auburn waves"; "rows of shining glasses"; "shiny black patents") • 6. (1) bright -- (splendid; "the bright stars of stage and screen"; "a bright moment in history"; "the bright pageantry of court") • 7. undimmed, bright -- (not made dim or less bright; "undimmed headlights"; "surprisingly the curtain started to rise while the houselights were still undimmed") • 8. bright, brilliant -- (clear and sharp and ringing; "the bright sound of the trumpet section"; "the brilliant sound of the trumpets") • 9. bright -- (characterized by happiness or gladness; "bright faces"; "all the world seems bright and gay") • 10. bright, shining, shiny, sunshiny, sunny -- (abounding with sunlight; "a bright sunny day"; "one shining morning"- John Muir; "when it is warm and shiny") • 11. bright, promising -- (full or promise; "had a bright future in publishing"; "the scandal threatened an abrupt end to a promising political career")

  13. Narrowing Down the Feature Space • Interface: • Adjectives from WordNet • Restrict characters • Reduce available senses

  14. Narrowing Down the Feature Space • Post Processing: • Use Synsets • Frequent synsets • Fit ORM ontology lexicons

  15. Image ORM Ontology

  16. Mapping between visuals and language • Series of Neural Networks • Bayes Net • Fuzzy Logic

  17. Evaluation • Let machine label images • Let humans label images • Let different humans evaluate machine and human labels • Compare evaluations

  18. Related Work • Aesthetic Visual Quality Assessment of Paintings (2009) • Congcong Li and Tsuhan Chen • Labeled impressionistic style landscape paintings as ‘high’ or ‘low’ quality using machine learning. • Algorithmic Inferencing of Aesthetics and Emotion in Natural Images: An Exposition (October 2008) • Ritendra Datta, Jia Li, and James Z. Wang • Overview of research involving predicting the quality class, score, and emotional label of photographs.

  19. Related Work • A Survey on Emotional Semantic Image Retrieval (2008) • Weining Wang and Qianhua He • Surveys ongoing Emotional Semantic Image Retrieval research. • Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction (October 2006) • Wang Wei-ning, Yu Ying-lin, and Jiang Sheng-ming • Labeled paintings with 12 emotional word pairs. Psychological research involved in choosing word pairs.

  20. Further Motivation • Augment the study of human perception and cognition. • Establish a linguistic-visual foundation for an artificially creative artist.

  21. Questions?

More Related