Image Subject Searching: What We Know and Where We Need to Go

1. Image Subject Searching: What We Know and Where We Need to Go Rachael Bradley April 2007 Acknowlegement: This presentation builds upon research conducted with CLiMB (Computational Linguistics for Metadata Building), supported by the Mellon Foundation

2. Purpose • Images are used in design, journalism, education, medicine, entertainment and many other areas. • Increasing numbers of images are available in digital format and can be searched online. • This presentation focuses on image subject retrieval in order to generate evaluation criteria and future research needs in image retrieval.

3. Outline • Introduction • Content and Meaning • User Studies • Key Characteristics • Current Technology • Evaluation • Future Research • So What?

4. Introduction • Introduction • Context • Example (1) • Content and Meaning • User Studies • Key Characteristics • Current Technology • Evaluation • Future Research • So What?

5. Context Four types of image attributes* • Biographical • Birth (Examples: Creator, Date, Title) • Travels (Examples: Who has owned it, Cost ) • Exemplified (Examples: Painting, .jpg, Sculpture) • Relationship (Examples: sketches-final painting, image-critiques) • Subject • Of/About (Examples: Of a lion/About pride) This presentation focuses on subject attributes • Additional attributes may be as important or more important to the end user when searching for an image

6. Artemisia Gentileschi Judith Slaying Holofernes, c1620 Florence, Uffizi Example (1) Artemisia Gentileschi Judith Slaying Holofernes, 1612-13 Naples, Museo di Capodimonte

7. Content and Meaning • Introduction • Content and Meaning • Levels of Content • Image Analysis • Establishing Meaning • Types of Meaning • Example (2) • Data Sources for Establishing Creator’s Intended Meaning • Language of Images • Symbols • Example (3) • Data Sources for Establishing Audience Interpretation • Evolution of Audience Interpretation • User Studies • Key Characteristics • Current Technology • Evaluation • Future Research • So What?

8. Levels of Content • Pre-iconography • Generic description of objects and events in an image • Knowledge gained by everyday experience is all that is needed • Iconography • Specific information, conventional matter • Requires familiarity with a specific culture • Iconology • Intrinsic meaning or content • Requires synthesis of information (Pre-iconography + Iconography + Knowledge of culture, artist, etc) • Each level includes time, space, activities/events, and/or objects

9. Panofsky-Shatford Matrix1 • Each level can be divided into factual or expressional • Simplified into Specific Of, Generic Of, and About 1. Shatford, 1986; Armitage and Enser, 1997

10. Image Analysis • Descriptive Analysis • Recognition and description of visual elements in a work of art • Shapes, forms, lines and colors • Formal Analysis • Recognizing visual relationships between shapes, forms, lines and colors • Images have coherent structure held together and ordered by the use of similar shapes, forms and colors • Internal Analysis • Focus on works inherent aspects (iconographic, narrative, symbolic) • External Analysis • Analysis of work within a larger context (historical, ideological, political, psychological, etc)

11. Establishing Meaning 1. Traditions of Representation • Known to the artist and to the actual or intended beholders • Recorded in symbolic dictionaries or recognized through repeated use in art 2. Pictoral Context and Location • Visual design in context of the rest of the picture • Location of the artwork in relation to other art or the building itself 3. Social and political background • Historical knowledge of events contemporary to the painting 4. Situation of the artist • Training, interests, emotional conflicts, attitudes, beliefs, economical and psychological relations to the patron and to the beholders 5. Intentions • Intentions of the particular artist • Intentions of most artists in a particular period 6. Responses of the beholders • Response of particular persons in particular situations • Response of normal people in normal situations • Conscious vs unconscious response

12. Types of Meaning • Creator’s intended meaning • Traditions of representation • Pictoral context and location • Social and political background • Situation of the artist • Intentions • Audience Interpretation • Responses of the beholders • Pictoral context and location • Social and political background

13. Example (2) Caravaggio, 1599 For Discussion: 1. Traditions of Representation 2. Pictoral Context and Location 3. Social and political background 4. Situation of the artist 5. Intentions 6. Response of the beholder Donotello, 1460

14. Text sources Associated metadata Primary sources Diaries Announcements News articles Contracts Religious works/ fictional texts Symbolic dictionaries Histories Image Sources Original Image Related Images Preliminary drawings Other works by creator Images the creator was aware of Architectural drawings Data Sources for Establishing Creator’s Intended Meaning

15. Language of Images Images can never mimic reality • Limited physical media do not allow for exact representation of reality • Images are information encoded by the creator and decoded by the viewer • “To say a drawing is a correct view...means that those who understand the notation will derive no false information from the drawing (90).” • “...the correct portrait, like the useful map, is an end product on a long road through schema and correction. It is not a faithful record of a visual experience but the faithful construction of a relational model (181).” Style defines the visual possibilities • “Styles, like languages, differ in the sequence of articulation and in the number of questions they allow the artist to ask (90).” Image Internal Meaning (Style - Artist Variations) + Symbolism + Relationships

16. Symbols • Symbols • Visual Elements • Contents: Time, Space, Activities/Events, and/or Objects • The symbol makes an informed viewer will think of what it symbolizes • The viewer can specify what it symbolizes • The symbol does not depict what it symbolizes • Natural Symbols • A natural connection exists between the symbol and what it symbolizes • Conventional Symbols • A tradition exists connecting the symbol and what it symbolizes • Identifying Symbols • Care in representation • Central/conspicuous position • Someone points to the motif • Presence is out of place

17. Example (3) Judith I Klimt, 1901

18. Text sources Primary sources Critiques Diaries Announcements News articles Histories Accession records Changes in use Price Image Sources Original Image Personal response Related Images Derived images Later works by artist Data Sources for Establishing Audience Interpretation

19. Evolution of Audience Interpretation Interpretation changes over time* • Creation • Artist Birth to Death • Quotation • Subsequent artists emulate images, style and technique • Interpretation • Frame: Classify, organize and interpret life experiences (Artist Anecdotes) • Artist Anecdote: Story of artist’s life and work that • Recontextualization • Work enters broader cultural/commercial context • Appropriation, Commercialization, Commodification • Consumption • Currency exchanged for some form of artist experience Interpretation is based on individual and cultural factors

20. User Studies • Introduction • Content and Meaning • User Studies • Image Study Methodology • Visual Elements • Pre-Iconographic and Iconographic Terms • Variation in Search Terms • Image Constructs • Iconological Terms • Image Selection • User Confidence • Query Modification • Browsing • Additional Findings • Key Characteristics • Current Technology • Evaluation • Future Research • So What?

21. Image Study Methodology • Analyzing email requests to a reference service • Requests created independently from a retrieval system • Provides some contextual information • Analyzing query logs from image search engines • Interface dependent • Large samples • No contextual information available • Possible bias because only queries with pre-identified image terms are selected • Self administered questionnaires describing searches • Contextual information available • Testing risk • User studies involving questionnaires, interviews and/or observations • Provide rich information on entire search process • Focuses on specific groups, possibly transferable but not generalizable • Keister found that requests varied by user types

22. Visual Elements Descriptive Analysis • Color, Line, Shape, Style, Focal Point In user studies, use of visual elements for search has been limited. *Used to distinguish between color and black and white photos

23. Pre-Iconographic and Iconographic Terms • Pre-Inconographic: NonUnique, Noun, Generic • Iconographic: Unique, Proper Noun, Specific • Refiners, used in many studies, confuse these analyses The level of content description in search terms is highly variable, likely due to task and collection differences

24. Variation in Search Terms Observations • "It is not so much that a picture is worth a thousand words, for many fewer words can describe a still picture for most retrieval purposes. The issue has more to do with the fact that those words vary from one person to another (p.17).1” • “No attempts to technically reduce such a notion to thesaurus or subject headings could ever encompass the richness of human induction when exposed to an image. If a picture is worth a thousand words to one viewer, it is worth a million words to 1,000 viewers. No individual or small group of individuals, no matter how professional or rule intensive the approach, could ever capture a full panoply of impressions invoked by an image (p.7).2” Results • In a study of 33,149 queries on Excite search engine from 9855 users, most terms only occurred once. The most frequently used terms occurred less than 10% of the time.3 • In a study of image professional’s use of a commercial image provider over one month, the top term (woman and women) occurred 7% of the time.4 In all levels of content, vocabulary varies greatly

25. Image Constructs • Similar to Risatti’s Formal Analysis • Introduced by Keister from analysis of reference requests at NLM • In an “Image Construct Query” the terms are used as a visual construction rather than simply isolated terms. • Examples : Man sitting in the chair with a box on his head People racing in wheelchairs Surgeons standing Results • Image constructs make up 1/3-1/2 of image requests1 • Visual constructs made up 83% requests2 Many searchers describe the object relationships within an image

26. Iconological Terms • Only one study has specifically examined use of Iconological Terms • 1,749 requests from 7 different image archives1 • Who: Mean 1.9 (Standard Deviation 3.8) • What: Mean 1.2 (Standard Deviation 2.3) • Where: Mean 0 (Standard Deviation 0) • When: Mean 0 (Standard Deviation 0) In the only study to examine iconology, use of iconological terms was limited.

27. Emotional Response Five studies reference search by emotional response Use of emotional response terms in search has been limited but has been used more than other iconological terms

28. Image Selection Study of 38 faculty and graduate students of American History1 • Topicality most important factor in making relevance judgments. • Most users did not feel comfortable making a relevance judgment based on image alone. • To make a final judgment users used both image and text. User study of journalism related requests2 • Selection based on (in order): 1) Topicality, often based on caption 2) Technical and biographical criteria 3) Impression to be conveyed (Difficult to convey in words) User study of journalism related image queries3 • “Topicality was a necessary but insufficient criterion for relevance...Final selection criteria could also be preferential or reactive; selections were based on personal impressions of images being ‘more interesting’', ‘funny’, ‘different’, ‘most dramatic'. (p. 107)” • “Searchers tended to alternate between viewing the textual description and the actual image during the selection process (p. 106).” Iconological factors become increasingly important during selection. Associated text is necessary during selection.

29. User Confidence • “Image needs were often fuzzy and could not be fully explicated. Most often it was however possible to name a critical object that should appear in the image. The search was then based on querying for this object (105).1” • No consistent rational manner for asking for pictures(9).2 • “The difficulty users often have in translating their image needs into verbal or written expressions is exemplified by the patron who states, ‘I can’t tell you what I want, but I’ll know it when I see it! (46).3’” • “The selection of search keys for general search topics was considered difficult. Journalists presumed that the archive contained photos relating to topics of interest, but they just had not discovered the right way to retrieve them (275)4.” Users find it difficult to express image information needs in words

30. Query Modification Study of 33,149 queries on Excite search engine from 9855 users1 • 40% of queries were first time queries and 60% were modified Study of image professional’s use of a commercial image provider • From 420 image search sessions, the mean number of queries per search 2.1 • 48% of queries were modified • 14% added one or more terms • 5.6% eliminated one or more terms • 28% changed one or more terms Approximately half of all queries are modified.

31. Browsing Study of 64 university student’s online image queries1 • Browsing was the primary strategy in satisfying 20% of information needs (199) User study of journalism related requests2 • Browsing was the main search strategy • "General search topics easily led to multiple queries and heavy browsing. Specific needs led more likely to just one or two queries and browsing sessions (274).“ • Trial and error method rather than carefully constructed queries Study of 1852 journalism related image queries3 • Browsing was the main search strategy after the initial query and especially important in abstract image needs and collaborative retrieval (105). Study of image professional’s use of a commercial image provider4 • Browsing took place in 90% of sessions in Sample 1 and 81% of sessions in Sample 2. • An average of 93 thumbnails were browsed per session in Sample 1 and 129 in Sample 2 (1354). Browsing is important during search and selection.

32. Additional Findings Query by Example Study of 404 queries to Google Answers’ Visual Arts1 • 10% provided examples (Cunningham, Bainbridge and Masoodian, 48) Study of 1 Month of search logs from a commercial image provider2 • “Other changes to queries included using terms that appear in image captions as additional terms ...these represent a change in search strategy to a “query by example” (QBE) form of search, but using text associated with the image rather than the image itself.” (Jorgensen and Jorgensen, 1355) Query for All Existing Material User study of journalism related image queries3 • Requests for all existing material on a certain topic accounted for nearly tenth of all image requests. This type of request has yet to receive any attention, even though it might affect retrieval measures such as recall (109) Query by example and specifying all existing material may be important to some users.

33. Key Characteristics • Introduction • Content and Meaning  • User Studies • Key Characteristics • Access Characteristics • Search and Selection Characteristics • Current Technology • Evaluation • Future Research • So What?

34. Access Characteristics Increasingly complex and variable access points • Visual Elements • Rarely used • Pre-iconography and Iconography • Often used • Use likely varies based on collection and tasks • Level of description can vary by individual, collection and task • Terminology can vary by individual and collection • Relationships between items is often important • Iconology • Rarely used • Level of description can vary by individual, collection and task • Interpretations vary widely by individual • Terminology also varies Although use of Visual Elements and Iconology has been rarely observed to date, this may be a result of testing limitations.

35. Search and Selection Characteristics • Users lack confidence in expressing their image needs • Users often modify queries based on results • Browsing is a key strategy in image search and selection • Iconology becomes increasingly important during selection • Both the image and associated text are important during selection

36. Current Technology • Introduction • Content and Meaning  • User Studies • Key Characteristics • Current Technology • Concept-Based Retrieval • Content-Based Retrieval • Social Tagging • Evaluation • Future Research • So What?

37. Concept-Based Retrieval Text to Text retrieval Text Associated with Images • Metadata • Ontologies and Classification Schemes • Keyword search on associated texts Challenges • Term agreement • Subjectivity • Level of agreement

38. Content-Based Retrieval Image to image retrieval • Color • Possible users: medical diagnosis, fashion and interior design, art history, journalism and advertising • Overall color or color by location • Texture • Coarseness, contrast and directionality • Shape • Boundaries or regions • Face Recognition • Difficulties disambiguating foreground and background? Query by Example • Input an example image or better yet set of images (typically selected) • Model the desired color, texture or shape (selected or created) Challenges • 3-Dimensions • Boundary delineation (foreground and background) • Variations in angles

39. Social Tagging • Allows the general public as well as professional community to apply text descriptions to images • Steve.museum • “At The Metropolitan Museum of Art, early studies indicate a significant variation between the existing collections documentation – recording artist, date, medium, dimensions, and iconography – and the words that are supplied by naïve viewers, describing the visual elements of an image and what it ‘literally’ depicts.1” Challenges • Vocabulary quality • Interface design

40. Evaluation • Introduction • Content and Meaning  • User Studies • Key Characteristics • Current Technology • Evaluation • Evaluation Criteria • Future Research • So What?

41. Evaluation Criteria Does the image retrieval system support: • searching by visual elements (likely using content based retrieval methods)? • query expansion methods for pre-iconographical and iconographical terms? • keyword searching of associated text for pre-iconographical, iconographical and iconological terms? • social tagging: a large number of users can apply iconological and other terms to images? • browsing both images and associated text? • query modification? • query by example?

42. Future Research • Introduction • Content and Meaning • User Studies • Key Characteristics • Current Technology • Evaluation • Future Research • Task Types • Type of Image Need • Additional Technology • So What?

43. Task Types • Attentional: Maintain or draw attention • Retentional: Assist with recall • Explicative: Explain visually what would be cumbersome to explain verbally • Descriptive: Show what an object looks like • Expressive: Make an impact on a reader • Constructional: Explain how various components fit together • Functional: Enable the viewer to follow a process or organization • Logico-mathematical: Diagram mathematical concepts • Algorithmic: To show possibilities • Data-display: Allow quick comparison and easy access to data • The majority of studies have focused on descriptive tasks. Research Question 1: How do search and selection strategies change across task types?

44. Type of Image Need • Image Needs • A preliminary experimental study indicated that keyword searching increased and browsing decreased with the specificity of the image need1. • Studies of journalism image search indicate that selection strategies differ for single and multiple images2. Research Question 2: How do search and selection strategies change with image needs?

45. Additional Technology • Style recognition has not been addressed • Social tagging has not been fully exploited as a mechanism for broadening iconological terminology • Available technologies have not been combined to create an overall image search experience Research Question 3: Is style recognition technologically feasible? Research Question 4: How can social tagging be used to improve the search experience? Research Question 5: How can concept-based retrieval, content-based retrieval, and social tagging retrieval be combined successfully?

46. So What? • Introduction • Topicality vs Contents  • Image Subject Search • Image Relevance • Key Characteristics • Current Technology • Evaluation Methods • Future Research Studies • So What?

47. Take Home Message • Research is currently being conducted in both content-based and context-based image retrieval but they are not coordinated • Variations in terminology and categorization across theory, user studies, and technology studies make it difficult to build on previous knowledge • Combining theory from various disciplines and empirical knowledge in image retrieval will provide the best chance of creating a successful search and selection experience.