Multimedia Databases

1. Multimedia Databases Prepared by Chengcui Zhang Lab: KDDM www.cis.uab.edu/kddm Email: zhang@cis.uab.edu www.cis.uab.edu/zhang 2010 Spring

2. Trends in Internet, Mobile Phones, Mobile Internet • Smart phones! • 40 million of these mobile phone users in Europe are mobile multimedia users. • The total Western European mobile market is worth 120 billion ECU per year in 2010. • The mobile multimedia segment of this Western European market are worth 30 billion ECU in 2010.

3. Introduction • Multimedia system: • A variety of information sources (text, voice, image, video, audio, animation, etc.) • Characteristics: • All the different media are brought together into one single unit, all controlled by a computer • Requirements: • Management and delivery of extremely large bodies of data at a very high rate • Real-time constraints … • Challenges: • Synchronization… • Semantic heterogeneity

5. Problems of Relational Database Model • Conventional data modeling techniques lack the ability to manage the composition of multimedia objects in a heterogeneous multimedia database environment. • Relational database system is only good to manage textual and numerical data. • Retrieving data is often based on simple comparisons of text or numerical values. • Relational data model has limited capabilities in modeling the structural and behavioral properties of real-world objects. • Relational data model has difficulty to model time-dependent multimedia data (video or audio). • BLOBs (Binary Large Objects) are incapable of interactively accessing various portions of objects since a BLOB is treated as a single entity in its entirety.

6. Problems of Object-Oriented Model • It provides a better facility for managing the multimedia data. • Good features: • Inheritance • Information hiding • Can include image data • Composite object (an object consisting of other objects) provides the capability to handle the structural complexity of the data

7. Problems of Object-Oriented Model (cont.) • Lack of facilities for the management of spatio-temporal relations. • Still, the O-O DBMS is not designed to support multimedia information management. • Multimedia extension is needed to handle the mismatch between multimedia data and conventional O-O database management systems.

8. Important Characteristics of Multimedia Objects (MO) • MO are complex and therefore less than completely captured in an MDBMS. • MO are audiovisual in nature and are amenable to multiple interpretations. • MO are content sensitive. • Queries looking for MO are likely to use subjective descriptions that are often fuzzy in their interpretation. • MO may be included in fuzzy classes. • …

10. Requirements for Modeling Multimedia Data • Specify incomplete information • Extend the definition of some individual documents beyond the definitions of its type • Integrate data from various databases and handle them uniformly • Describe structural information • Distinguish between internal modeling and external presentation of objects

11. Requirements for Modeling Multimedia Data (cont.) • Share data among multiple documents • Create and control versions • Include appropriate operations • Handle document access control

12. New trend: SMELL! http://staff.science.uva.nl/~gevers/master2007/PDF/lecture1_small_2007.pdf

13. Multimedia Database Applications • Education: CAI (Computer Assisted Instruction) • Internet search (e.g., Google image/video search) • Medical Imaging • Surveillance Systems • Biometrics databases • Video-on-demand • Game …

14. Application: Image search engines – Goggle! http://www.google.com/mobile/goggles/#landmark

15. Application: Fingerprint Matching and retrieval

16. Application: real-time skin detection for human recognition • Are HP computer webcams really racist? • http://blogs.consumerreports.org/electronics/2009/12/racist-hp-webcam-video-blog-consumer-reports-response.html

18. Application: real-time object recognition and tracking

19. Application: Surveillance http://www.nydailynews.com/ny_local/2010/01/08/2010-01-08_new_jersey_man_arrested_over_security_breach_at_newark_liberty_airport.html

20. Content-Based Image Retrieval • An picture is worth a thousand words!

21. Text-based Retrieval

22. Content-Based Image Retrieval • Content-Based Image Retrieval (CBIR) • Image databases can be huge, containing hundreds of thousands or millions of images. • In most cases they are only indexed by keywords that have to be decided upon and entered into the database system by a human categorizer. • However, image can be retrieved according to their content, where content might refer to color distributions, texture, region shapes, or object classification.

23. Image Database Examples • IBM: Query by Image Content (QBIC) • Retrieves images based on visual content, including such properties as color percentage, color layout, and texture. • Virage, Inc. • Virage search engine can retrieve images based on color composition, texture, and structure. • Google Image search. • National Library of Medicine provides a database of x-rays, CT scans, MRI images, and color cross-sections, taken at very small intervals along the bodies of male and female cadaver. • The NASA collects huge databases of images from its satellites and makes them available for public acquisition. (for free )

24. State-of-the-Art in MDBMS • First wave – query by text • In a second wave, commercial systems were proposed which handle multimedia content by providing complex object types for various kinds of media. • Broadly used commercial MMDBMSs are extensible Object-Relational DBMS (ORDBMSs). • Oracle 10g, IBM DB2, and IBM Informix.

25. DB2 Image Extender • DB2 Image Extender defines the distinct data type DB2IMAGE with associated user-defined functions for storing and manipulating image files • (http://www-306.ibm.com/software/data/db2/extenders/ ). • The DB2 Image Extender provides similarity search functionality based on the QBIC technology • (http://wwwqbic.almaden.ibm.com/ )

26. Query By Example (QBE) • The image DB user should be able to: • show the system a sample image, or • Paint one interactively on the screen, or • Just sketch the outline of an object. • The system should then be able to return similar images or images containing similar objects.

27. IBM-QBIC • The Hermitage Web site was voted the best in Russia. It uses the QBIC engine for searching archives of world-famous art. • http://www.hermitagemuseum.org/fcgi-bin/db2www/qbicSearch.mac/qbic?selLang=English • Color percentage • Color layout

28. A sample query • SELECT CONTENTS(image), QBScoreFROMStr(`averageColor= <255,0,0>’, image) AS SCORE FROM signs ORDER BY SCORE

29. Photobook System Figure 1. The texture retrieval of PhotoBook system (http://web.media.mit.edu/~tpminka/photobook/).

30. Search Using Sketch • Sketch entry • Results of search

31. ImageScape System Figure 2.5 The interface of ImgeScape visual query system (http://skynet.liacs.nl/imagescape/).

32. Relevance Feedback in CBIR • Motivation: • Human perception of image similarity is subjective, semantic, and task-dependent. • The CBIR based on the similarities of pure visual features are not necessarily perceptually and semantically meaningful. • Each type of visual feature tends to capture only one aspect of image property and it is usually hard for a user to specify clearly how different aspects are combined … • Relevance Feedback is introduced to address these problems. • It is possible to establish the link between high-level concepts and low-level features.

33. Relevance Feedback RF (cont.) • RF is a supervised active learning technique used to improve the effectiveness of information systems. • Main idea: use positive and negative examples from the user to improve system performance.

34. Query Image Initial Query Results User Relevance Feedback Query Results After User Feedback Initial query results Collect user’s feedback Real-time learning Refine query results

35. Training System Interface

36. Object-based Image Retrieval • Object-based CBIR: • Motivation • The basic unit of user interests usually is individual objects. • Images are segmented into homogeneous regions, and the image features are extracted for each region. • Image similarity is then measured in term of region similarity.

37. Spatial Indexing

38. Object-Based Image Retrieval with Relevance Feedback • Techniques used: • Image segmentation • Neural network • Multiple instance learning • …