Temporal Aspects of Intelligent Multimedia Retrieval and Presentation

1. Temporal Aspects of Intelligent Multimedia Retrieval and Presentation Zhisheng Huang, Jacco van Ossenbruggen, and Lynda Hardman CWI The Netherlands

2. Outline • Motivation and Background • Temporal approaches for intelligent multimedia retrieval and presentation • Temporal reasoning in the Semantic Web • Research Plan • Past work (by Zhisheng) • Demo

3. Time and Multimedia Documents and Presentation • Time is one of the most crucial aspects ofmultimedia documents and presentation. • Audio, video, animation clips are presented with time goes, • Image archivies are usually displayed with timelines, • Text information describes events, time instants, intervals/periods, • Presentation involves a series of actions which occur in time.

4. Temporal Aspects and Their Description • Time instant: year, date, time clock • Time intervals and events: golden age, Monet’s born, • Temporal Modifiers: after, before, in, during, • Temporal operators: always in the past, sometimes in the past, never in the past, in the future, now, at present • Action operators: doing, acting, …

5. Semantics of Time Three Main Approaches: • The Quantitative Time Approach: time is a collection of time instants and intervals • The Temporal Logic Approach: time fluent is considered as state transitions • The Dynamic Logic Approach: time fluent is considered as results of taking actions

6. Temporal Retrieval vs. Temporal Reasoning • Temporal retrieval: find entities which satisfy a temporal specification. • Temporal reasoning: check if a temporal property holds. • Temporal retrieval can be achieved by converting answers with “yes/no” into answers with entities.

7. Quantitative Time • Time interval: Show all work which involve Monet in years 1850-1890; • Time instant: Show all Dutch impressionist work in 1926; • Time instant with time modifier: Show all Dutch impressionist work after 1926; • Event: Show all Dutch impressionist work in year Monet died; • Event with time modifier: Show all Dutch impressionist work after Monet died;

8. Temporal Logics • Linear time vs. branching time • Temporal operators: • always-in-the-past, • sometimes-in-the-past, • never-in-the-past, • at-present, • always-in-the-future, • Until • Since

9. Time in Temporal Logics • Temporal retrieval: show the first Dutch impressionist work (in this art archive). ( Defined in terms of never-in-the-past operator and the at-present operator) • Temporal reasoning: Is it never the case that a Cubist work occurred earlier than the first impressionist work ?

10. Dynamic Logics • State vs. Action • A state corresponds with a time instant • An action is a set of state pairs which correspond with the starting state and the ending state of the action. • Dynamic operators: • [a]p: after doing action a, the property p always holds; • <a>p: after doing action a, the property p may holds;

11. Dynamic Actions • Dynamic action operators: • sequential actions, • parallel actions, • action choice, • state checking • conditional action

12. Time in Dynamic Logics • Temporal Presentation: a sequential execution of actions • Temporal retrieval: a search action with temporal specification • Temporal reasoning: model checking with dynamic actions

13. Temporal Reasoning in the Semantic Web • Temporal reasoning with RDF/RDFS data • Temporal Ontologies • Time in OWL

14. Research Plan • Compare the three temporal approaches • Investigate their expressibility on intelligent multimedia retrieval and presentation • Quantitative time reasoning for image retrieval and presentation • Temporal retrieval and reasoning with the support of the Semantic Web technology.

15. Some Past Work (by Zhisheng) • Temporal reasoning for multi-version ontologies (2005) • PAMELA: an intelligent presentation embodied agent (2004) • Facial expressions for embodied agents in STEP (2004) • XSTEP: a markup Language for embodied agents (2003) • STEP: a scripting language for embodied agents (2002) • WASP: Web Agent Support Program (2001)

16. Pamela: Demo