Video Information Retrieval

1. Video Information Retrieval Mark Ruzomberka IST 497 11/07/02

2. Joke

3. Outline • What is Video Information Retrieval (VIR) ? • Reasons VIR is necessary • Theoretical • Where we are today • Examples • Problems • Future Work • Conclusion

4. What is Video Information Retrieval (VIR) ? • Recognition technologies • Image • Voice • Text transcripts • Document retrieval technologies • Topic segmentation • Topic matching • Text summarization • Presentation Technologies • Combine Recognition and retrieval technologies • Result is an integrated application

5. VIR-Need, or Why do I care? • Consider the task of trying to find a five minute video clip of interest in a library of 1000 hour long tapes. • Consider the “go to the part where” problem

6. What do people want from IR D-Lib Magazine’s asks: “What do People want from Information Retrieval?” # 8 Multimedia

7. Specificly, Reasons for Video IR • Reading is slow compared to your potential for understanding information • Humans think in pictures not words • Reading is particularly slow on a computer screen • Example: Daydreaming while some one is talking • Reading a page in a book and not remembering what it was about

8. VIR makes for quicker human understanding. • Palm/Grafitti 25 • Hand Writing 35-40 • Typing 50-70 • Speaking 135-175 • Reading 200 • Listening 400 - 500 • Thinking 500+ Video IR allows for faster access to information

9. Theoretical: • Think of the “Jetsons mail system” • You “talk” to the computer, • Computer intelligently “talks” back to you

10. Where we are today • Two of Video Information Retrieval System are currently available: • Type One- keyword/text based • Type Two- Content based

11. Type One- keyword/text based DVR- basic expansion of image IR, not as interesting

12. Type Two- Content based MSR Video Skimmer Video Mail Informedia

13. Example: Video Mail • University of Cambridge • 1994-1996 • AT&T • 1999 • 2000-project ended

14. Video Mail: Medusa network • Medusa multimedia environment at Olivetti Research Ltd. In Cambridge • It takes a modular approach unlike that of a pc or workstation • Unified by a common interface to ATM network • Devices plug directly into network and include: • Cameras • Audio devices • Networked frame buffers • Processor farms • Disk drives

15. Video Mail: Medusa Network • “The network is the computer” metaphor is used • Solves storage and network speed problems • Complicates expense problem

16. How it works-Overview

17. The Integrated Application • “narrow” by sender,date, time

18. Video Mail: Video Browser • Content is now being viewed • Keywords are flagged

19. Video Mail: Video Browser • In the latest version “thumb-nailed” pictures of key frames replace color coded line of the search keyword

20. Informedia The Informedia Digital Video Library Project automatically combines speech, image and natural language understanding to create a full-content searchable digital video library.

21. Informedia

22. Informedia: human factor issues • Interaction • Motivation • Effective usage modes • Commercial compression • VHS quality playback. • Terabyte (1,000 gigabytes) of storage • 1000 hours of video.

23. Problems • Human understanding • Spoken document retrieval • Poor video browsers • Expensive • Slow access to data • Large amounts of data

24. Microsoft Research (MSR) Video Skimmer

25. Enhanced Browser Controls: Time Compression Pause Removal Textual Indices: TOC, Notes Visual Indices Shot Boundary Frames Timeline Markers Jump Control (Back/Next) Microsoft Research (MSR) Video Skimmer

26. Problem: Poor Content Based Video Browsers • Current VCR model allows for poor navigation • “go the the part where they say” problem

27. Problem: Expensive • Hard drive space expensive • Video adds to problem • High bandwidth needs are also expensive http://www.littletechshoppe.com/ns1625/winchest.html

28. Problem: Slow Access to Data • Broadband still not available everywhere • Availability doesn’t mean acceptance • Especially after dot com crash 2000

29. Problem: Large Amounts of Data • Current Systems use MPEG2 • Newer compression technologies • MPEG 4-DIVX -DVD Quality • Video consumes orders of magnitude more storage than text • MPEG 7 is on horizon

30. Future Work ? • Sky the limit ? • Sci-Fi the limit ? • Hard Drive Space, Bandwidth are current limitations.

31. Conclusion • Not yet ready for prime time • Storage and Network Costs decreasing • Success is in day to day usage • Slowly Becoming Mainstream E.x.Tivo • Problems of “real world tests” • Idiot proof • ATM and Medusa aren’t mainstream

32. Papers • Video Mail Retrieval Using Voice: Report on Keyword.. - Jones, Foote, Jones.. (1994) • What do people want from Information Retrieval?. Croft, Bruce W. D-Lib Magazine. (1995) • Video Skimming for Quick Browsing based on Audio and Image.. - Smith, Kanade (1995) • The VISION digital video library (context) - Gauch, Li et al. – (1997) • Informedia: News-on-Demand Multimedia Information.. - Hauptmann, Witbrock (1997) • M.G. Christel and D.J. Martin, "Information Visualization within a Digital Video Library", J. Intelligent Info. Systems 11(3), (1998), pp. 235-257 • Browsing Digital Video. Li, Gupta, Sanocki et. Al.

33. Questions?

34. Joke? • "There are 10 types of people in the world... • those who understand binary and those who don't."