1 / 43

Multimedia Information Retrieval

Multimedia Information Retrieval. It is of the highest importance, in the art of detection, to be able to recognize out of a number of facts which are incidental and which are vital ... -Sherlock Holmes Presented by Dipti Vaidya. Overview of the Presentation.

mshank
Download Presentation

Multimedia Information Retrieval

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Multimedia Information Retrieval It is of the highest importance, in the art of detection, to be able to recognize out of a number of facts which are incidental and which are vital ...-Sherlock Holmes Presented by Dipti Vaidya

  2. Overview of the Presentation • Overview of MultiMedia Information Retrieval • Research Issues & Approaches • Case Study of 2 of these Approaches: - Content Based Approximate Picture Retrieval - Knowledge Based Approach for Retrieving Images by Content • What next ?

  3. Multimedia Information Retrieval • Types of Associated Information • Content-independent metadata (CIM) • Format,author’s name,date • Content-dependent metadata (CDepM) • Low level features concerned with perceptual facts eg:color,texture,shape,spatial relationship • Content-descriptive metadata (CDesM) • High level content semantics eg: good weather

  4. New Generation MMIR • Retrieval not only by concepts but also by perception of visual content • Objective measurements of visual contents and appropriate similarity models • Automatically extract features from raw data by image processing, speech recognition,pattern recognition and other computer vision techniques

  5. MMIR

  6. Image Retrieval • By perceptual features • for each image in database,a set of features are computed • to query the image database, through visual examples authored by user or extracted by image samples • select features and choose a similarity measure • Compute similarity degree, ranking and relevance feedback

  7. Image retrieval

  8. Image retrieval

  9. Research Issues • Extraction of perceptual feature (CDepM) • Color,Texture,shape:Because of perception subjectivity,there doesn’t exist a single best presentation for a given feature. • Segmentation Comparison of existing methods:

  10. Comparision of Existing techniques • Histogram Matching : big in size,complicating database creation,sensitive to brightness creation,lacks spatial information,difficult to create partial match • Texture Mapping : not applicable to non textured regions,hard to figure out what model to use for a given application • Region based mapping: results can go ugly when segmentation is not done properly

  11. Research Issues • To make CBIR truly scalable to large size image collections, efficient multidimensional indexing techniques needs to be explored • Retrieval Speed • Multi-Dimensional Indexing techniques: - Bucketing Algorithm - K-d tree - K-D-B tree - R- tree and it’s variants R+ tree and R*- tree Problems: These techniques cluster data based on minimizing the number of disk access per data retrieval, but do not consider semantic difference of image of image features; thus no global conceptual view of image clustering can be provides eg: LARGE, NEARBY tumor ????

  12. MMIR Systems • QBIC • Visual Seek and WebSEEK • MARS • Photobook

  13. Content-Based Approximate Picture Retrieval -A Prasad Sistla, Clement Yu at UIC

  14. Problem Formulation • The objective is to find the pictures stored in the system that are very similar to the user’s description • A similarity measure is used to find the degree by which two descriptions match

  15. Solution Approach • There is some metadata associated with each picture in our database which describes the content of each picture • Metadata contains information about the objects in the picture,their properties and the relationships among them • A knowledge base containing a thesaurus and certain rules is used for some deductions

  16. Representation of Pictures A picture can be represented by an E-R graph as follows: • Entities: objects identified in a picture • Relationships: associations between the identified objects • Attributes:properties (color,size etc) that qualify of characterize objects

  17. Example Example: Bridge Over Right of RIVER TREE blue tall Left Of HILL green short

  18. Relationship Classification • Action • Mutual eg: handshake • Directional eg: moving • Spatial • Mutual eg: adjacent • Directional eg: above • Subject Entity • Object Entity

  19. User Interface Queries are entered by the user via the interactive interface • Identify all the objects • Identify various attributes of the object eg:color, size • Choose the appropriate relationship from the list provided • Visual language also provided to specify part of a query

  20. Mapping Queries to Formal Language Expressions Using UniSQL/X Example: “ Find the picture(s) of president Clinton” SELECT P.pid FROM Picture P WHERE P.Entities[X] And X.Cname =‘president’ and X.Pname=‘Clinton’ Where Cname = Common Name and Pname = Proper Name Plan to write 2 more query examples…..

  21. RETRIEVAL • Two pictures are similar if both the corresponding entities and the corresponding relationships are similar Similarity of 2 Entities: • The name of the entities are the same, synonyms or in an IS-A relationship hierarchy • The attribute values of the 2 entities do not conflict

  22. Similarity of 2 Entites Let e be an user specifies entity and E be an entity stored in the system e = {n,a1,a2,…ak} E = {N,A1,A2,…,AK} Where n and N are the name of the entities e and E respectively and ai and Ai are the ith attribute values of the corresponding entities The similarity of entities can now be defined as: k • Sim (e,E) = ½ (Sim (n,N) + 1/kSim (ai,Ai)) • i=1

  23. Similarity of 2 Entity names • The 2 names are same => Sim(n,N) = 1 • The 2 names are synonyms => Sim(n,N) >0 3. The 2 names are antonyms => Sim(n,N)= -inf 4. One of the names is NULL or both are NULL=> Sim(n,N) = 0

  24. Similarity of 2 Attributes We say that an attribute is neighborly if there is a closeness predicate that determines for any 2 values of the attribute whether the 2 values are close or not. For Eg: The “age” attribute is neighborly it can take one of the values- “very young”, “young”,”middle age”,”old” and “very old” Two values are considered “close” if they occur next to each other in the list

  25. Similarity of 2 neighborly attributes: • Sim (a,A) = w if a = A • Sim (a,A) = cw if a or A are close • Sim (a,A) = -inf if a and A are not close Here, w is determined using the inverse document frequency method & c is a positive constant less than 1

  26. Similarity of 2 Relationships Let r be the relationship of some entities which are specified in the user’s description and R be another relationship of some entities which are given in a picture stored in the system r = {n,(e1,t1),(e2,t2),…(em,tm)} R= {N,(E1,T1),(E2,T2)…..(Em,Tm) T = type (O-Object, S-Subject) m m • Sim (r,R) = 1/3 (Sim (n,N) + 1/m Sim(ei,Ei) +1/m Sim (ti,Ti)) i = 1 i =1 Sim(ti,Ti) = 1 if ti = Ti = 0 if one is type N and other is not = - inf if one is type S and other is type O

  27. A sample Retrieval blue moving 1 • E-R Diag (Query) Lu,mu,ru Black,white medium River Mountain moving Ll,ml,rl Sail Boat 6 still brown large In by 2 4 Tree 5 Sailor Fisherman rl ¦ still green black small still white still small

  28. A Sample Retrieval 2 E-R Diag (DB-pic1) moving blue Lm,mm,rm Rm,ru 3 Right of 1 Over Bridge River Tree small green brown mm stationary still Above Left Of brown blue 4 5 sky Mountain Lm,mm stationary Lu,um,ru stationary

  29. A Sample Retrieval stationary 25 3 moving blue moving E-R Diag (DB-pic2) black tree Behind in river Sail boat 2 1 green Ll,ml,rl Lm,mm,rm behind 7 In On person beach Ll,ml,rl black Person stationary 5 Lu,mu,ru ll 3d-irregular Mountain still white 6 brown Very large

  30. Research Issues • Not very feasible for large databases • Coding of the descriptions of the pictures is a labor intensive task • Handling type matches eg:clouds “in” the sky or “cloudy” sky • Deduction of Spatial Relationships

  31. A Knowledge_Based Approach for Retrieving Images by Content Chih-Cheng Hsu, Wesley Chu -UCLA

  32. Overview • A Knowledge-based spatial image model (KSIM) which supports queries with semantic and similar-to predicates • Interested objects in the images are represented by contours segmented from images • Image content are these object contours using domain specific knowledge • These image features are classified using MDISC and represented by TAH for knowledge-based query processing

  33. KSIM • A three-layered model is used to integrate the image representations and image features together with image content interpretation knowledge - The Representation Layer (RL) - The Semantic Layer (SL) - The knowledge Layer (KL)

  34. Image Representation • Raw images are stored in RL • Image objects are represented by contours which can be segmented manually or semi-automatically in the RL Difficulty: automated segmentation of these objects is still not achieved which leads to the deployment of this technique.

  35. Semantic Layer • The shape model and spatial relationship model in the SL are used to extract image features from the contours Example: Object feature - Conceptual terms Tumor.size small,medium,large Tumor.roundness circular,non-circular Lateral_ventricle. Symmetric, upper_protrusion_pressed_right L_R_Symmetry upper_protrusion_pressed_left

  36. Segmentation Layer • Spatial relationship table: Spatial RelationshipRep.featuresdefined semantic term SR(t,b) (xc,yc,ia) slightly occupied, extremely occupied SR(t,l) (oc,dc,xc,yc) nearby,far_away …. ….. …..

  37. Mapping Queries to Formal Language Expression using CoBase Example Query: “find large tumor nearby the lateral ventricle” Select patientWithImage(patient: t.patient,image:t.image) From Tumors t, Lateral_ventricle l Where t NEARBY l and t.size IS ‘large’

  38. Query Interpretation via TAH • The concept in the TAH node is represented as the value range of the features • The TAH nodes can be labels with the conceptual term (eg. large, small) to represent specific knowledge • There is a TAH directory that stores such information as object names,set of features, spatial relationships ,user type,purpose of TAH,etc • Based on this information, the system selects and retrieves the appropriate TAH for processing the query

  39. TAH • Example:

  40. Query Processing • The query analysis and feature selection phase • The knowledge based content matching phase • Query relaxation phase

  41. Flow diag of query processing • Query Query Processing Satisfactory answers Post Processing Relaxation manager TAHs, user model Query modification

  42. A Sample Retrieval Example • User: brain surgeon • mandatory matched objects:Lesion and Brain • optional matched objects: lateral ventricle and frontal lobe • relaxation order: SR(l,lv) and SR(l,f) are more important than SR(l,b) in order.. 1 SR(l,f) Frontal Lobe SR(l,b) Lesion brain 2 SR(l,lv) Lateral Ventricle 1

  43. Research Issues • Use XMLMetadata for describing images and the use of this data for automatic generation of the web pages.

More Related