Image Tagging

1. Image Tagging Attaching textual meta-information or semantic linkages to images By Perry Rajnovic

2. What is a Digital Image? • Digital Images are usually defined as an organized display of pixels, often called a bitmap. • Each pixel is a numeric representation of the color intensities of that point.

3. What is a Digital Image? • Each pixel may be explicitly defined or be the result rendered by a vector or graphics package functionality. • These representations include no inherent textual elements or semantic description. • Due to the above, images are not easily machine-readable.

4. Why machine-readability? • Most searches are done via textual queries, thus there must be a mechanism to link applicable keywords or phrases to images. • For blind persons, being able to convey information about the image in another medium would be good for accessibility.

5. Image Contents • The contents of an image can be a full description written in prose (i.e. the adage “A picture is worth 1000 words”), or might simply have a few keywords describing spatial, temporal, or emotional aspects. • In many cases, accurately identifying the content of images requires human intervention.

6. Identifying Image Contents • Many good pattern recognition algorithms exist, however few are able to interpret the patterns extracted. • Artificial Intelligence algorithms can learn recognized patterns, but such a system’s flexibility is limited by its predefined knowledgebase.

7. Identifying Text in Images • CAPTCHAs (or Completely Automated Public Turing test to tell Computers and Humans Apart) are images which contain a distorted rendering of some text.

8. Identifying Text in Images • Their goal is to provide an easy task for humans to do, but that is extremely hard for computer programs to perform equally. • For this task, OCR is generally not sufficient enough to extract the text. • This is a good example of why machine-readable information should be available.

9. As an example of what might be provided to tag an image, to the right is a list of words and phrases to describe this slide’s header. Navy Blue Squares Fade-out Horizontal Bar Minimalist Decorative Example Tag Contents

10. User Applications • Many applications take advantage of image tagging, below are a few examples. • Apple iPhoto • Google Picasa • Adobe Photoshop Elements • Generally these programs use tagging for organization and user-defined searching.

11. Web Applications • Several Web-based applications are now including tagging for images, as well as other non-image based features. • Google ImageLabeler • Flickr.com • Facebook.com • 23hq.com • Fotki.com

12. Google Images • Luis von Ahn developed the “ESP Game” which could be used to tag images. • He presented a Google tech talk about the game as a form of human computation. • Google later licensed the technology to create a similar web application called the Google Images ImageLabeler.

13. Google ImageLabeler • The ImageLabeler game allows to random users to generate tags that accurately describe images. • The tags should be accurate due to game constraints, and gain specificity after several rounds. • The computed tags can improve searches.

14. Flickr • Flickr is a “Web 2.0” photo hosting and sharing site. • Users are encouraged to upload photos, then to name, describe, tag, annotate, geotag, comment on, and group their photos in collaborative ways.

15. Flickr - Tags • Tags are words or phrases meant to act as keywords. • They are searchable within the site, and can show popular topics. • They improve search relevance.

16. Flickr - Geotagging • Geotagging is a term for adding geospatial metadata to images such as the latitude, longitude and other directional indications of where a photo was taken at.

17. Wikipedia defines them as: Extra information associated with a particular point in a document or other piece of information. The US DoD defines them as: A marking placed on imagery or drawings for explanatory purposes or to indicate items or areas of special importance. What are annotations?

18. Annotating Images • The use of annotations with images can provide several useful functions. Below are some examples: • Point out a specific piece of content. • Explain some icon or graphic. • Summarize the meaning of some region. • Provide additional information via text.

19. Flickr - Notes • Flickr provides a feature called Notes. It uses a Flash-based implementation of an annotation system. • You can dynamically size a rectangular region over a portion of the image, then attach a snippet of text to describe it.

20. FotoNotes • FotoNotes is a data format for annotating images. • Allows you to embed the metadata directly into the image files for portability. • Flickr’s Notes feature is inspired by this standard and accompanying visualization implementation.

21. FotoNotes - More • It was developed by Greg Elin. • The homepage provides links to groups working with the standard. • Additionally, an implementation which works in most browsers is provided as-is for customization.

22. Facebook • Facebook.com has a tagging feature that is integrated with “My Photos”. • It allows you to add a textual descriptor (tag or person’s name) to a specific point in the image. • This allows the module to describe who or what are included in a specific album.

23. Facebook – Tag Display • When the images are viewed, placing the mouse over a tag displays a fixed sized square indicating where the tag (person) is located within the image. • This enables users to identify objects by visual inspection or by matching the list of contained objects with their tag displays.

24. Facebook – Links • Another capability incorporates the site’s concept of friends. If the person you tag is identified as your friend on the site, their name will link to their profile. • The site will also count this image in the “photos of” feature on their profile, allowing inclusion of photos added by other users.

25. Other Image Metadata: MPEG • The MPEG-7 standard is a “Multimedia Content Description Interface” • “MPEG-7 is not aimed at any one application in particular; rather, the elements that MPEG-7 standardises shall support as broad a range of applications as possible.”

26. Other Image Metadata: Adobe • Adobe Systems created a new MetaData framework for images called XMP (Extensible Metadata Platform). • It is publicly documented, based on W3C standards, built on XML, and is designed to eliminate growing incompatibility for metadata storage.

27. Other Image Metadata: IPTC • The International Press Telecommunications Council created standards for the interchange of news data over a decade ago. • These standards still persist in their IIM standard, as well as being usable in the newer XMP framework.

28. Improving Clustering Search Interfaces Joint Term Project By Perry Rajnovic and Mark Zalar

29. Term Project • For my term project, I will be working with Mark Zalar to develop a new search engine interface • It will draw inspiration from all of the top search engines today, along with the enhancements now possible using emerging technologies.

30. Project Goal • The Goal of the project is to implement a search site that provides a highly usable interface for query refinement. • Our backend will use clustering mechanisms to allow for easy refocusing of search topics • Our frontend will use AJAX for flexibility.

31. Frontend Design • The frontend will be designed with a technology known as Asynchronous JavaScript and XML (AJAX). • This technology allows the site designer to run unseen requests to the server and parse XML-based results in the scripting language for interactivity.

32. Frontend Theory • Most clustering based search solutions available today use minimally interactivity. • Our theory is that making the ability to harness the power of clustering dynamically as you refine your search will improve results as well as time necessary to finish a search.

33. AJAX Functionality • Our site will use AJAX to dynamically reconfigure the clustering menu. This allows a quicker browsing of clusters to identify the optimal range of pages to search within. • The menu will also use a novel interface that shows sibling and parent clusters.

34. AJAX Functionality 2 • The results will be displayed to the user with some animation. • This will help to alert users when changes are made to the order or set of results. • Another advantage of this is that users will be more aware of the difference between clusters as they browse them.

35. Search Target • This search engine could target both websites and images. • Valid keywords improve content knowledge. • Clustering would be highly useful in finding an image with a desired scene or set of objects.

36. Example: search “creature” • The engine might identify a general cluster of “animal” or “being”. • Animal might have more results, so the medium level clusters are shown for that.

37. Example: search “creature” • User wants a general discussion of mammals. Selects that cluster. • The results change to focus on those related to mammals as a group and in specific.

38. Results Display • To provide animation, a similar technology to that found in “TiddlyWiki” will be used. • This interface allows topics to be added and removed dynamically with animation. • Additionally, extra links can be attached to each topic for more functionality (open in new window, similar items, etc.)

39. Future Enhancements • Our implementation will provide a basic mockup of the interface and refinement techniques made available. • Several enhancements could be made to this interface that would improve its usability or functionality.

40. Enhancements in Search • Taking advantage of a meta-search would allow the clustering algorithm to have a higher volume of data with which to generate data topologies to be explored. • Using adaptive search (by userID or global optimization) would improve clustering by choosing ones more often used.

41. Enhancements in Interface • Because the site will be AJAX based, a large amount of flexibility is possible with respect to changes in the interface. • The browser window is similar to a canvas, with all of the site’s underlying Document Object Model available for addition, modification or deletion.