Efficient Video Browsing

1. Efficient Video Browsing Using Multiple Synchronized Views Heymo Kou

2. Question • What is the two main technologies applied for efficient video browsing? (one for audio, one for visual content)

3. Table of contents • Background • Current technology • Advanced technology • Summary • Reference

4. Growth of digital contents data

5. Digital video market growth

6. Digital video becomes ubiquitous • From your • Smart phones • Notebooks • Webcams • Digital camera and camcorders • Security and monitoring cameras • With advanced streaming technology • Fast Internet access • MPEG-4 format

7. Current technology for finding a video data • Search through categories • Similar to Internet shopping mall • We search for big categories • Then smaller categories • …and so on… • User should choose which to browse • Should check whether the selected data matches what user was finding • Time consuming! • Manual categorizing and annotation • One by one?

8. Problem with current video search and browsing technologies • Too complicated • Lack of efficient algorithm • Time consuming • Multimedia calculation ∝ exponential • Inaccuracy • Video data is increasing exponentially • Cataloging manual has a somewhat limit point • Manually cataloging is done by human hand that mistakes can be happened

9. Technologies for advancedimage and video retrieval • MPEG-7 Standards • Speech indexing • Shot Boundary Detection • Time Scale Modification of Audio Signals • Storyboards, Moving Storyboards and Animation • Adaptive Accelerating Fast Playback • Streaming Synchronized Views

10. MPEG-7 standard • Standardized by ISO/IEC • International Standard Organization • International Electrotechnical Commission • Not a video encoding format • XML to store metadata • Attached to timecode in multimedia • By this tag • Able to index and search efficiently • Yet, improvement is needed

11. Speech indexing • Search through speech transcripts • Finds familiar metaphor of free text search • Automatic speech recognition (ASR) • Indexed transcript → semantic information • Main advantage : Representation • Speech is built of words

12. Definitions • Frame • Key frame • Shot • Group of frameswhich represents similar frames Start key frame end key frame animation

13. Definitions • Context • Meaningful information within multimedia data • 3 levels of video browsing • Browsing a large collection of videos • Browsing a ranked list of videos • Browsing a single video to find relevant segments

14. Shot Boundary Detection • Shot Boundary Detection(SBD) algorithm • Completely automatic • Key frames are selected and extracted • Saved as JPEG files • High Accuracy and Efficiency • Still, fault detection problem is unsolved

15. SBD algorithm

16. Similar to scene selection of dvd

17. Time Scale Modification ofAudio Signals • Audio browsing is as important asvideo browsing • Except images, most digital contents are audible • Faster audio browsing is necessary • Speeding up of audio signal by • By deleting small audio segments • Especially, human speech signals are quasi-periodic

18. Improvement of TSM Time-Domain Harmonic Scaling(TDHS) technique Time-Domain, Pitch Synchronous Overlap Add Time Scale Modification (TSM) algorithm Waveform Synchronous Overlap(WSOLA)

19. Synchronous Overlap-Add SOLA

20. Storyboards, Moving Storyboards and Animation • Storyboard • a set of one or more pages, each consists of a two dimensional array of key-frames, sorted in chronological order. • Animation • a quick slide show, where each of the key-frames is shown for a fixed short period (e.g., 0.6 seconds) • Moving Storyboard (MSB) • the animated key frames, fully synchronized with the original audio track. Each key-frame is shown for the entire duration of the associated shot.

21. Adaptive Accelerating Fast Playback • Very fast video playback (without audio) • Ordinary fast forward depends only on speed • There is a chance to miss important scene • Accelerates until new scene is met • Requires less computation load

22. Image for adaptive fast playback

23. Real-use of adaptive fast playback • Example in surveillance camera

24. Streaming Synchronized Views • Server preprocesses media • Keep same media, but different speed encoded • When user selects other speed • 1. pause current media • 2. open file with same content with selected speed • 3. seek to the corresponding position • 4. play the selected view • Needs no extra computational load • However, requires more storage: Tradeoff

25. Browsing Multiple Videos: MovieDNA • Can browse multiple videos at once • Split frames every given time • (i.e 10 seconds) • Strong information scent is visible • With aggregation of occurrences

26. Image of movieDNA

27. Summary ofmain properties

28. Conclusion • Streaming synchronized views and movieDNA • Less computation, multiple videos at once • Active accelerating fast playback • Most useful at analyzing surveillance videos • SBD & TSM • Efficient for implementing above technologies • Then, what is current limitation?

29. Q & A Any questions?

30. Answer • What is the two main technologies applied for efficient video browsing? (one for audio, one for visual content) • Answer : The two main technologies are Shot Boundary Detection(SBD) for visual content and Time Scale Modification(TSM) for audio signals

31. References (1/2) • Shot Boundary Detection • http://muvis.cs.tut.fi/sbd.html • Key frame • http://en.wikipedia.org/wiki/Key_frame • Synchronous Overlap-Add • http://www.surina.net/article/time-and-pitch-scaling.html • Digital Video Market Growth • http://articles.businessinsider.com/2011-12-13/research/30508929_1_fios-cable-providers-video-streaming-service/2 • Amount of Digital data • http://www.emc.com/collateral/analyst-reports/diverse-exploding-digital-universe.pdf

32. References (2/2) • Streaming synchronized view • http://www.visus.uni-stuttgart.de/forschung/visualisierung-und-visual-analytics/visuelle-analyse-videostroeme/adaptive-fast-forward-for-video-surveillance.html • MovieDNA • http://homepage.mac.com/juggle5/WORK/publications/HICSS34.pdf