1 / 29

Advances in Multimedia Information Retrieval Models

Explore advanced topics in multimedia information retrieval, including data modeling, query processing, and relevance assessment. Learn about the challenges and techniques used in multimedia search applications.

lorrij
Download Presentation

Advances in Multimedia Information Retrieval Models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Special Topics in Computer ScienceAdvanced Topics in Information RetrievalLecture 5 (book chapter 11): Multimedia IR:Models and Languages Alexander Gelbukh www.Gelbukh.com

  2. Previous Chapter: Conclusions • Inverted files seem to be the best option • Other structures are good for specific cases • Genetic databases • Sequential searching is an integral part of manyindexing-based search techniques • Many methods to improve sequential searching • Compression can be integrated with search

  3. Previous Chapter: Research topics • Perhaps, new details in integration of compression and search • “Linguistic” indexing: allowing linguistic variations • Search in plural or only singular • Search with or without synonyms

  4. Motivation • Applications: • office, • CAD, • medical, • Internet • Example: • Artists sings a melody and sees all the songs with similar melody

  5. What’s different • Different from text IR: • Structure of data is more complex. Efficiency is an issue • Using of metadata • Characteristics of multimedia data • Operations to be performed • Aspects: • Data modeling: Extract and maintain the features of objects • Data retrieval: based not only on description but on content

  6. Retrieval process • Query specification • fuzzy predicates: similar to • content predicates: images containing an apple • data type predicates: video, ... • Query processing and optimization • Parsed, compiled, optimized for order of execution • Problem: many data types, different processing for each • Answer • Relevance: similarity to query • Iteration • Bad quality, so need to refine

  7. Modeling

  8. Data modeling • To model is to simplify, in order to make manageable. “We will represent an image as...” • From the user’s point of view • From the system’s point of view (technically) • A problem: very large storage size. Modeling needed • Objects are represented as feature vectors • Images / Video: shape. House, car, ... • Sound: style. Music: Merry, sad, ... • Features are defined directly or by comparison • Degree of certainty is stored

  9. Multimedia support in commercial DBMSs (1999) • Variable length data. • Non-standard • Different and usually very limited sets of operations • SQL3: • provides user-extensible data types • Object-oriented • Implemented partially in many systems • Example: data blades of Informix • Content-based functions on text and images • E.g.: date = 1997 AND contains (car)

  10. Spatial data types • Informix: 2D, 3D data blades • Boxes, vectors, ... • Operations: intersect, contains, center, ... • Text: containWords, .... • Supports query images by content

  11. Example: MULTOS • Multimedia document server • Documents are described by: • logical structure: title, into, chapter, ... • layout structure: pages, frames, ... • conceptual structure: allows content-based queries • Docs similar in conceptual structures are grouped into conceptual types • Example: Generic_Letter

  12. Example of conceptual structure...

  13. ...continued

  14. Image data in MULTOS • Analysis • low level: detect objects and positions • high level: image interpretation • Result of analysis: • description of objects found and their classes • certainty values • Indices are used for fast access to this info • Object index. Includes pointers to objects and certainty values • Cluster index, with fuzzy clusters of similar images

  15. Internet • How Google does it? • No image processing. Textual context! • File names, nearby words • Distance from image to words • “give me images with flower in the file name or near the image”

  16. Languages

  17. Query languages • As a query, either a description of the object or an example object is submitted • “show me images similar to this one” • in what respects similar?! • Exact match is inadequate. Additional means are needed • Content is not a single feature

  18. What defines query language • Interface. How to enter the query • Types of conditions to specify • Handling of uncertainty, proximity, weights

  19. Interface • Browsing and navigation • Search: description or query by example • Query by example: • specify what features are important. Give me all houses with similar shape but different colors • Libraries of examples can be provided

  20. Conditions... • Attribute predicates • structured content – the predefined types extracted beforehand • Exact match. E.g.: size, type (video, audio, ...) • Structural predicates • structure: title, sections, ... • metadata are used. Find objects containing an image and a video clip • Semantic predicates • unrestricted content. • Find all red houses: red = ?, house = ? Fuzzy

  21. ... conditions • Predicates • Spatial: contain, intersect, is contained in, is adjacent to ... • Temporal: Find audio where first politics and then economy is discussed • Spatial and temporal predicates can be combined: Find clips where the logo disappears and then a graph appears at the same place • A predicate can be applied to a part of document • As path expressions in OO databases

  22. Uncertainty, proximity, weights • Similarity function • The user can assign importance weights to individual predicates in a complex query • This gives ranking, as in text IR • The same models can be used, e.g., probabilistic model

  23. Examples of query languages: SQL3 • Functions and stored procedures: user-defined data manipulation • Active database support: database reacts on the events, not only commands. This enforces integrity constraints • Good news: rather standard • Bad news: no ranking supported! • Effort to integrate SQL3 with IR techniques.SQL MM Full Text and other similar languages

  24. ... examples: MULTOS • One of design goals: easy navigation • Paths are supported • Identification of components by type, not by position • All images in the document, not the image in 3rd chapter • Types of predicates: • on data attributes, on textual components, on images (image type, objects contained, ...) • Example:

  25. MULTOS example

  26. Another example of MULTOS

  27. Research topics • How similarity function can be defined? • What features of images (video, sound) there are? • How to better specify the importance of individualfeatures? (Give me similar houses: similar = size?color? strructure? Architectural style?) • How to determine the objects in an image? • Integration with DBMSs and SQL for fast access and rich semantics • Integration with XML • Ranking: by similarity, taking into account history, profile

  28. Conclusions • Basically, images are handled as text described them • Namely, feature vectors (or feature hierarchies) • Context can be used when available to determine features • Also, queries by example are common • From the point of view of DBMS, integration with IRand multimedia-specific techniques is needed • Object-oriented technology is adequate

  29. Thank you! Till ??, 6 pm

More Related