Event-Based Fusion of Distributed Multimedia Data Sources

1. Event-Based Fusion of Distributed Multimedia Data Sources Vincent Oria Department of Computer Science New Jersey Institute of Technology Newark, NJ 07102

2. Outline • Classical Data Integration Problem • Multimedia Data • An Architectural approach to Multimedia Data Integration • Event-Based Integration of Data Sources • Conclusion

3. Classical Data Integration* * Borrowed from M. Lenzerini

4. Classical Data Integration Issues • How to construct the global schema? • (Automatic) source wrapping • How to discover mappings between the sources and the global schema? • Limitations in the mechanisms for accessing the sources • Data extraction, cleaning and reconciliation • How to process updates expressed on the global schema, and updates expressed on the sources? • The modeling problem: How to model the mappings between the sources and the global schema? • The querying problem: How to answer queries expressed on the global schema? • Query optimization

5. Multimedia Data • Multimedia data management is more than physical server design Logical data modeling is important • Multimedia data management is more than similarity search • “Show me all the images that are similar to this one [in terms of color, texture, shape].” Querying is much more complicated • Give me all the news items on Baghdad over the last 2 weeks

6. Multimedia Data … • Multimedia data is heterogeneous in both format and in access primitives and this has to be accommodated You cannot store all the data in a single DBMS; the system has to be open • Query-based access to multimedia data is important as well as browsing and some transactional access Some DBMS-like interface and control over multimedia data should be provided

7. <article> ..... </article> Multimedia Database Processing MM Data Pre- processor <!ELEMENT ..> ..... <!ATTLIST...> Meta-Data Recognized components Additional Information QueryInterface MM Data MM Data Instance MM Data Instance Users Multimedia DBMS Multimedia Data Preprocessing System Database Processing

8. <article> ..... </article> Document Database Architecture <!ELEMENT ..> ..... <!ATTLIST...> DTD/ XML Manager Schema Parser DTD or XML Schema files DTD/ XML Schema Type Generator Query Interface <!ELEMENT ..> ..... <!ATTLIST...> Document content Document Parser DTD/ XML Schema XML or SGML Document Instance Documents Parse Tree Types Users Document DBMS Instance Generator Objects Document Processing System Database Processing

9. Image Analysis and Pattern Recognition <article> ..... </article> Image Database Architecture Semantic Objects Syntactic Objects Image Content Description Meta-Data Query Interface Image Annotation Image Users Image Image DBMS Image Processing System Database Processing

10. Video Analysis and Pattern Recognition <article> ..... </article> Video Database Architecture Key Frames Video Content Description Meta-Data Query Interface Video Video Annotation Video Users Video DBMS Video Processing System Database Processing

11. Multimedia Data Integration: An Architectural Perspective • Simple Client-Server • Integrated Server • Database Server • Middleware and Mediation

12. Video/ Audio Meta- data Text Images Simple Client-Server Client Image Server Database Server Text Server CM Server • Heavy-duty client • Synchronization, user interface, QoS, … • Client has to access each server • Scalability problems • client code has to be updated when new servers come on-line

13. Video/ Audio Meta- data Text Images “Integrated” Server Client DBMS Functions Object Storage Server Image Server CM Server • Heavy-duty server • DBMS should be able to handle multiple storage systems • Real-time constraints on CM

14. Meta- data Text Video/ Audio Images Database Server Client Database Server Image Server Text Server CM Server • Lighter client • Client has to access only one server • Scalability problems • server may become a bottleneck - distribute and interoperate

15. Client Text Video/ Audio Images Document Server Structured Document DBMS Image DBMS CM DBMS • Document-centric view • Multimedia objects are parts of documents • Might be suitable for, e.g., e-commerce catalogs

16. Middleware Mediator Mediator Mediator Mediator Mediator Text DBMS Text DBMS Video DBMS Image DBMS Image DBMS Interoperable System Client Client Wrapper Wrapper Wrapper Wrapper Wrapper

17. Event-Based Multimedia Data Integration • An event aims at modeling any happening • Facts, context • An event has 3 components • Time • Space (location) • Objects

18. Event1 Event3 Image Image Video Video Text Image Events: Temporal Dimension • Time Line and Temporal relationships Event2 Time Line Image Video

19. Events: Spatial Dimension • GIS (Location and Spatial Relationships) Event2 Event1 Event3 Directional and Topological relationships

20. Events: Object Dimension • Which real world objects are involved in the event? • Object Recognition • Classical Data Integration

21. Event: Spatio-Temporal Dimension • Moving Objects and their Trajectories • Raw representation: The trajectory T of a moving object is defined as a sequence of vectors T=[t1, …, tn] Each ri show the successive positions of the moving object over a period of time. • Movement sequence: The trajectory of a moving object is represented by a sequence of (movement direction, distance ratio) pairs. This representation is not affected by rotation, shifting or scaling. M=[m1, …, mn-1] Each mi is a pair of (movement direction, distance ratio).

22. Event Model • Events model interpretation context • Example: KIMCOE 2006 is an event • Participants are objects • Location: Hilton Garden Inn, Suffolk, Virginia • Date/Time: October 24 - 27, 2006 • Has sub-events like sessions or visit of Lockheed Martin's Center For Innovation • Event Properties • Discrete or continuous • Local or distributed • Simple or composite • Descriptors: Data (classical and multimedia)

23. Event Querying Time Objects: RDBM, XML Space: GIS

24. Event Querying Time Objects: RDBM, XML Space: GIS

25. Event Querying Time Objects: RDBM, XML Space: GIS

26. Event Operators • Temporal Operator • Spatial Operators • Spatio-Temporal Operator • Aggregation

27. Aggregation and Concept Hierarchy • Dimensions are hierarchical by nature: total orders or partial orders • Example: Location(continent  country  province  city) • Time(yearquarter(month,week)day) Industry Country Year Category Region Quarter Product City Month Week Office Day

28. Aggregation and Concept Hierarchy: Operators • roll-up (increase the level of abstraction) • drill-down (decrease the level of abstraction) • slice and dice (selection and projection) • pivot (re-orient the multi-dimensional view) • drill-through (links to the raw data)

29. Aggregation and Concept Hierarchy: Roll-up • Use of aggregation to summarize at different levels of a dimension hierarchy • Ex: if we are given total sales per city we can aggregate on the market to obtain sales per state Time (Quarters) Q1 Time (Quarters) Q2 Q3 Q4 Market (city, state) Newark S. Orange Drama Q1 Q2 Q3 Q4 N. York Market (States,, USA) New Jersey Comedy Category New York Drama Dayton Horror Arizona Comedy Ohio Category Sci. Fi.. Horror Sci. Fi.. Roll-up on Market

30. Aggregation and Concept Hierarchy: Drill-down • Inverse of roll-up • Given a total sales by state, we can ask for more detailed presentation by drilling down on market Q1 Time (Quarters) Q2 Q3 Q4 Market (city, state) Newark S. Orange Drama Q1 Q2 Q3 Q4 N. York Market (States,, USA) New Jersey Comedy Category New York Drama Dayton Horror Arizona Comedy Ohio Category Sci. Fi.. Horror Sci. Fi.. Drill-down on Market

31. Months Slice on January Products Cities Newark Products Electronics Dice on Electronics and Newark January January Aggregation and Concept Hierarchy: Dice and Slice

32. Conclusion • Event model: A data Integration model • This is a work in progress: We need to fully define the event model • We want to build on existing Technology (RDBMS, XML, GIS,..)