MULTIMEDIA DATABASE SYSTEMS

1. MULTIMEDIA DATABASE SYSTEMS

2. What is MMDB • A multimedia database is a database that hosts one or more primary media file types such as .txt (documents), .jpg (images), .swf (videos), .mp3 (audio), etc. And loosely fall into three main categories: • Static media (time-independent, i.e. images and handwriting) • Dynamic media (time-dependent, i.e. video and sound bytes) • Dimensional media (i.e. 3D games or computer-aided drafting programs- CAD)

3. Some Practical Applications • There are numerous different types of multimedia databases, including: • The Authentication Multimedia Database (also known as a Verification Multimedia Database, i.e. retina scanning), is a 1:1 data comparison • A newly-emerging type of multimedia database, is the Biometrics Multimedia Database; which specializes in automatic human verification based on the algorithms of their behavioral or physiological profile.

4. Questions • How can multimedia database systems be formally defined so that they are independent of any specific application domain? • Can indexing structures for multimedia database systems be defined uniform, domain independent manner? • Is it possible to uniformly define both query languages and access methods based on these indexing structures? • What constitutes a multimedia presentation and can this be formally defined independent of any specific application domain?

5. Overview • Introduction • MMDBS Design Example • A system for the representation of multimedia frames independent of the underlying application is presented. • Indexing structure • Query language

6. Multimedia Database Management Systems • The main task of DBMS is to abstract data retrieval and management from storage details. • Besides the need for large storage capacity, real time requirements are also demanded of MMDBS. Other properties provided by MMDBMS are: • Persistence of data: DBMS should be able to manipulate data even after the changes of the surrounding programs. • Device independent interface • Format independent interface

7. Multimedia Database Management Systems (Properties provided by MMDBMS) • Consistent view of data • In multi-user environments consistent view of data during data processing • Security of data and integrity protection • Transaction concept • Query and retrieval of data • Query languages • Descriptive search methods • “the picture of a woman with a red scarf.”

8. Multimedia Database Management Systems (Properties provided by MMDBMS) • Corresponding storage media • Integrated and/or secondary storage • Device independen interface • Interface to MMDBMS should be device independent • Format independent interface • Allows new technologies to be used without changing the current application • View specific and simultaneous data access • Consistent access to shared data • Management of large amounts of data • Real time data transfer • Long transactions (retrieval of a movie)

9. Multimedia Database Management Systems (Properties provided by MMDBMS) • Relational consistency of data management • The attribute relation: different descriptions on presentation of the same object (image, spectrum, explanatory text..) • The component relation: includes all the parts belonging to a data object (book consisting of chapters, sections, subsections…) • The substitution relation: defines different kinds of presentation of the same information (table, graph, animation,..) • Synchronization relation: temporal relations between data units

10. Data Structures

11. Operations on Data Storage and retrieval operations for all data types (raw, registered, descriptive) are needed. Operations are: • Input (insert/record) data will be written to db. Raw and registering data are always needed. • The output (play) operation : reads the raw data according to registering data. For decoding a JPEG coded image the Huffman table can be transmitted to the decoder in advance. • Modification : usually considers the raw data. Modification attributes are stored in registering data. If modification is on data format than registering data should also be modified. Modification may also mean medium change ex. from text to speech. • Delete: consistency. If raw data are deleted all related registering and descriptive data should be deleted. • Comparison: Queries are based on search which depends on comparison of • raw data. • pattern may be stored in registered data • content oriented descriptive data

12. MULTIMEDIA DATABASE SYSTEMS Towards a Theory of Multimedia Database Systems Sherry Marcus and V.S Subrahmanian

13. Concepts • Media instance(video instance) consists of body of information (set of video clips) represented using some storage mechanism (a quad-tree, R-tree, etc.) in some storage medium (disk) together with some functions or relations ( who appears in the video) expressing various aspects, features, and/or properties of this media instance. • Media instances can be used to represent wide variety of data including documents, photographs, object oriented databases and logic programs to name a few.

14. Concepts • Multimedia System – a set of media instances. • Media event – the global state of different media at a fixed point in time • A media presentation – is a sequence of media events. A media presentation shows how the states of different media instances change over time.

15. Basic Ideas This framework is totally independent of physical storage and conceptual representation of media. • Suppose M is a medium and this medium has several “states” representing different bodies of knowledge expressed in that medium – associated with this data is a set of “features” – these capture the salient aspects or objects of importance in that data. • Logical relations and/or properties occurring within a state are encoded as a logic program. • Since a given medium can have a multiplicity of states the presence of a corpus for state transition functions is assumed

17. Media Instances • Media instances can be used to represent wide variety of data stored in different kinds of media (raster, bitmap data for video) • Additional information on how to access and manipulate information is also contained in media instances.

18. Media Instances • Media instance is a 7-tuple Mi = (ST, fe, , R, F, Var1, Var2) • ST: set of objects called states • fe: set of objects called features •  : is a map that tells which features are possessed by a given state • R: is a set of inter-state relations on the set S • F: is a set of feature-state relations • Var1: set of objects called state variables ranging over states • Var2 : set of objects called feature variables ranging over features.

19. The Clinton Example (video domain) ST: set of all possible bitmaps fe: {bush, clinton, nixon, gore, reno, kissenger} (s1): {bush, clinton, nixon} (s2):{clinton, reno} R: delete.nixon(S, S’) F: left(clinton, reno, s2)

20. Audio Example • Add audio information to the two states in the video example • fe = {clinton, gore, bush, nixon, reno, kissenger, who, unesco, world_bank} • a1: Clinton speaking about the WHO • a2: Audio tape with Clinton and Gore discussing UNESCO • a3: Bush and Clinton arguing on numerous subjects • ST = {v1, v2, a1, a2, a3} • (a1) = {clinton, who} • (a2) = {clinton, gore, unesco} • (a3) = {clinton, bush} • R = {after(a1, a2), after(a2, a3)} • F = {topic(who, a1), topic(unesco, a2), speaker(1, clinton, a2), speaker(2, gore, a2), speaker(3, clinton, a2)} /* speaker(i, person, a): ith speaker is person in frame a */

21. Document Example • Suppose d1, d2, d3 are documents reflecting information about policies adopted by various organizations. • d1: position statement of WHO about Clinton • d2: Statement made by Clinton about WHO • d3: Statement made by Clinton about UNESCO • ST = {v1, v2, a1, a2, a3, d1, d2, d3} • (d1) = {who, clinton } • (d2) = {who,clinton} • (d3) = {unesco, clinton} Assume contents feature-state relation : contents(author, topic, state) • F = {cotents(who, clinton, d1), contents(clinton, who, d2), contents(clinton, unesco, d3) } • R is left empty for this example

22. Indexing Structures and Query Language for Multimedia Systems • Consider a multimedia system M = {M1,… , Mn} that a user wishes to retrieve information from. Frame-based Query Language • Constant symbols • Each feature is a constant symbol • Each state is a constant symbol • Each integer is a constant symbol • Function symbols • flist is a binary function symbol in the query language • Variable symbols: Infinite number of logical variables are assumed • Predicate symbols • A binary predicate symbol frame type • A binary predicate symbol, , • Predicate symbol for R (of arity j) • Predicate symbol for F (of arity j)

23. Indexing Structures and Query Language for Multimedia Systems A query is an existentially closed conjunction of atoms ()(A1 & … &An) Example from video domain of Clinton Assume we have the following feature state relations: running_mate(X, Y, S): X’s running mate is Y in S appointed(X, Y, P, S): X appointed Y to position P in state S with(X, Y, S): X is with Y in S In the first two relations S does not matter Clinton’s running mate is Gore Clinton appointed Reno as Attorney General 3rd relation is state specific with(clinton, bush, 1) with(clinton, nixon, 1) with(clinton, reno, 2) Background inference rules are allowed with(Y, X, S)  with(X, Y, S)

24. Indexing Structures and Query Language for Multimedia Systems (X, P, S) appointed(clinton, X, P, S) & with(clinton, X, S) & frametype(video) (X, Y, S, S1, S2) presidet(X, S1) & president(Y, S2) & (X clinton) &(Y clinton)&(X Y )& with(clinton, X, S) & with(clinton, Y, S) & frametype(S, video) Is there any one who is a Clinton Appointee who appears in a picture/video frame with Clinton Is there any picture in which three presidents of the USA (one of whom is Clinton) appear together/

25. Indexing Structures and Query Language for Multimedia Systems (S) (clinton  flist(S) & horse  flist(S) & on(clinton, horse) & frametype(S, video) Is there a picture of Clinton on horse?

26. Indexing Structures and Query Language for Multimedia Systems In general if we are given a media instance Mi = (Sti, fei, i, Ri, Fi, Var1i, Var2i) Then we will store information about the state relations as a logic program

27. Indexing Structures and Query Language for Multimedia Systems There are two kind of facts State independent facts vice_president(clinton, gore, S) State dependent facts left(clinton, reno, s2) Inference rules right(X, Y, S)  left(Y, X, S)

28. The Frame Data Structure Frame data structure is set up to access multimedia information efficiently Suppose we have n media instances Mi = (Sti, fei, i, Ri, Fi, Var1i, Var2i) for 1<=i <=n

29. The Frame Data Structure OBJECT-TABLE is used to store information about which states contain a given feature. For each feature there is an entry with a pointer to the list of states containing that feature • Specifies where the information pertaining to the object is kept • it can be organized as a standard hash-table

30. type=“object” type=“node2” link2 next objname frameptr

31. The Frame Data Structure Data Structures for frame- based Feature Storage

32. The Frame Data Structure Data Structures for frame- based Feature Storage

33. The Frame Data Structure Frame – With each state s, we associate a pointer which points to a list of nodes each of which points to a feature in the object table. We assume that state-frame relationships are stored as logic programs.

34. The Frame Data Structure • These definitions specify frames independent of the medium (e.g. audio, video, latex file, quad tree, bit map, etc. ) used to store the specific data involved. • The internal structure is specified using the data structure framerep.

35. Indexing for a single medium(Clinton video example) This example shows how a collection of objects can be represented as a frame. We have 2 video frames and 4 objects (clinton, bush, nixon, reno) • v1 and v2 are variables of type frame v1.rep = 100 /* the disk address at which v1.starts at */ v2.rep = 590 v1.flist = pointer to a list of three nodes of type node1 (feature list) (features in v1: clinton, nixon, bush) The list of video frames in which bush appears as a feature is pointed by the pointer P1.link2 = ((v1.flist ).objid).link2

36. bush P1 clinton P2 nixon P3 reno P4 object table

37. type=“object” type=“node2” link2 next objname frameptr P4.objname = Reno P4.link2 points to the start address of v2

38. Advantages of the presented indexing scheme • Queries based on both”object” as well as “video” frame can be easily handled • Data structures are independent of data structures used for physical storage • Precise mechanism for storing a picture (bit-map, quad-tree, etc.) does not effect the conceptual design. • Nature of the medium is irrelevant to the data structure

39. Example Let (X, Y) with (X, Y) denote the query • “Given a value for X find all people Y who appear in common video frame with X. • If X=Clinton • From the object table set objname=clinton • Let N be the node with its objname field set to clinton (object structure) • N.link2 points to a list of nodes N’ such that N’.frameptr points to a frame with Clinton in it.

40. Example • For each node N’ in the list pointed to by N.link2, do the following: • Traverse the list pointed to by (N’.frameptr).flist. • Print out the value (N’.frameptr).flist).objname for every node • Repeat this process for each node in the list pointed to by N.link2.