Video summarization by video structure analysis and graph optimization

1. Video summarization by video structure analysis and graph optimization M. Phil 2nd Term Presentation Lu Shi Dec 5, 2003

2. Outline • Motivation • Video structure • Video skim length distribution • Spatial-temporal graph modeling • Optimization based video shot selection • Experimental results

3. Motivation • Huge volume of video data are distributed over the Web • Browsing and management in the huge video database are time consuming • Help the user to quickly grasp the content of a video • Two kinds of applications: • Video skimming (dynamic) • Video static summary (static)

4. Goals • Conciseness • Content coverage • Spatial and temporal • Coherency • Not too jumpy

5. Flowchart

6. Video structure • Video narrates a story just like an article does • Video (story) • Video scenes (paragraph) • Video shot groups • Video shots (sentence) • Video frames

7. Video structure • Graphical example

8. Video structure • Can be built up in a bottom-up manner • Video shot detection • Video shot grouping • Video scene formation

9. Video structure • Video shot detection • Video slice image [1] • Column - pairwise distance • Filtering and thresholding • ……

10. Video structure • Video shot grouping • Window-sweeping algorithm [2] • Spatial similarity • Temporal distance • Intersected video shot groups form loop scenes

11. Video structure • Summarize each video scene respectively • Loop scenes and progressive scenes • Loop scenes depict an event happened at a place • Progressive scenes: “transition” between events or dynamic events

12. Video structure • Scene importance: length and complexity • Content entropy for loop scenes • Measure the complexity for a loop scene

13. Video structure • Determine each video scene’s target skim length • Determine each progressive scenes’ skim length • If , discard it, else • Determine each loop scenes’ skim length • If ,discard it • Redistribute to remaining scenes

14. Graph modeling • Spatial-temporal dissimilarity function • Linear with visual dissimilarity • Exponential with temporal distance

15. Graph modeling • The spatial temporal relation graph • Each vertex corresponds to a video shot • Each edge corresponds to the dissimilarity function between shots • Directional and complete

16. Skim generation • The goal of video summarization • Conciseness: given the target skim length • Content coverage • The spatial temporal dissimilarity function • The spatial temporal relation graph • A path corresponds to a series of video shots • Vertex weight summation • Path length is the summation of the dissimilarity between consecutive shot pairs

17. Skim generation • Objectives: • Search for a path in the graph such that: • Maximize the path length (dissimilarity summation) • Vertex weight summation should be close to but not exceed it • The objective function

18. Skim generation • Global optimal solution • Let denote the paths begin with , whose vertex weight summation is upper bounded by • The optimal path is denoted by • The target is

19. Skim generation • Optimal substructure • Dynamic programming • Effective way to compute the global optimal solution • Trace back to find the optimal path • Time complexity , space complexity

20. Experiments • Key frames of selected video shots

21. Experiments • There is no ground truth so that it is hard to objectively evaluate a video skim • Subjective experiment • Parameters:

22. Conclusion • Video structure analysis • Scene boundaries, sub-skim length determination • Graph scene modeling • Optimization based sub skim generation • Generate a video skim

23. Reference • [1] C. W. Ngo, Analysis of spatial temporal slices for video content representation, Ph. D thesis, HKUST, Aug.2000 • [2] Y. Rui, T.S. Huang, and S. Mehrotra, Constructing table-of content for videos, ACM Multimedia Systems Journal, Special Issue Multimedia Systems on Video Libraries, vol. 7, no.5, pp. 359~368, Sept 1999.

24. Q & A Thank you!!