ADVISE: Advanced Digital Video Information Segmentation Engine

1. ADVISE: Advanced Digital Video Information Segmentation Engine Presented byNg Chung Wing

2. Outline • Introduction • Overview of ADVISE • System architecture and services provided • Technologies in ADVISE • Construction of Video Table-of-Contents (V-ToC) • Video Summarization • Video Matching • Conclusion

3. Introduction • Videos is getting more popular in education, entertainment and information sharing • Evident growth of video contents on the Internet • 57.2% of Internet users watched video chips and 7.3% edited video clips on their personal computers (Survey by PC Data, 2000) • Important task  retrieve an interested video! • Two problems in video retrieval: • Not enough information to describe the video contents • Difficult to search for videos with similar contents Introduction

4. Introduction • We propose the “ADVISE” system to solve the above problems • ADVISE • Advanced Digital Video Information Segmentation Engine • Web-based video browsing and retrieval system • Provides a set of services: • For description of videos: • Video table-of-contents (VToC) • Video summarization • For searching similar videos: • Video feature similarity matching Introduction

5. Contributions • We propose the framework of a video browsing and retrieval system called ADVISE • We build the image-based video description called Video Table-of-Contents (V-ToC) in ADVISE • We develop the Video SummarizationAlgorithm for generating video summaries in ADVISE • We propose two Video Tree Matching Algorithms, which measure the feature similarity between videos, in ADVISE Introduction

6. Overview of ADVISE - Outline • Introduction • Overview of ADVISE • Objective of ADVISE • System Architecture • Services provided by ADVISE • Technologies in ADVISE • Conclusion

7. Objectives of ADVISE • To provide an efficient way to describe the video contents • To save the time for browsing the whole video to know the contents • To search videos with similarity in certain video features • To provide services through the Internet Overview of ADVISE

8. (III) Video streaming server  Setup the Real System Server for delivering video contents System Architecture of ADVISE • ADVISE consists of 3 modules • (I) Video preprocessing module • (II) Web-based retrieval module • (III) Video streaming server  Major contributions  Process source videos  Provide services to users of ADVISE  Setup the Real System Server for delivering video contents Overview of ADVISE

9. System Architecture of ADVISE II I III Overview of ADVISE

10. (II) Web-based Video Retrieval Module • User interface for accessing services of ADVISE • This module reside on a web server • There are 3 services provided by ADVISE 1 • Service 1: V-ToC Presentation • Service 2: Generation of SMIL Video Summary • Service 3: Querying Similar Videos 2 3 Overview of ADVISE

11. Module (II) ~Service 1: V-ToC Presentation • Image-based description for the video content • Use the V-ToC structure resulting from Module (I) • Used XML with XSL to generate a flexible web-based presentation Each V-ToC show us the contents organization in a video Overview of ADVISE

12. Module (II) ~ Service 2: Generation of SMIL Video Summary • Generate a video summary according to user’s preference • Used SMIL to deliver the customized video summary to the user User’s input Resulting SMIL Video Summary Overview of ADVISE

13. Module (II) ~Service 3: Querying Similar Videos • Show similar videos in descending order of the similarity score • Results of video matching in Module (I) • User can select matching different video features • Color histogram feature • Shot style feature List of videos with different similarity scores with the query video Overview of ADVISE

14. Technologies in ADVISE - Outline • Introduction • Overview of ADVISE • Technologies in ADVISE • For service 1: Construction of Video Table-of-Contents (V-ToC) • For service 2: Video Summarization • For service 3: Video Matching • Conclusion

15. Construction of Video Table-of-Contents (V-ToC) • Uses • Image-based video description which show the organization of video contents  Video Table-of-Contents (V-ToC) • Provide the hierarchy for structural matching of video • Video structure used in ADVISE • Hierarchical tree structure with 4 levels • Storage and Presentation • Use XML and XSL Technologies in ADVISE - Construction of V-ToC

16. Video Structure in ADVISE • Decompose a video into 5 levels: • Video Frames • Video Shots • Video Groups • Video Scenes • Whole Video Hierarchical Representation of a Video Technologies in ADVISE - Construction of V-ToC

17. Video Structure in ADVISE • Example: 2 Shots: 1 Scene 1 3 Video Group 1 4 Shot 1 Scene 2 Group 2 Shots 2,4,6 Group 3 Shots 3,5,7 Technologies in ADVISE - Construction of V-ToC

18. Video Structure in ADVISE • Structure videos from the bottom level • 5 steps in video structuring • i. Color Histograms Extraction • ii. Video Shot Boundaries Detection • iii. Video Groups Formation • iv. Video Scenes Formation Technologies in ADVISE - Construction of V-ToC

19. Storage and Presentation • Resulting Presentation of V-ToC using XML and XSL <?xml version="1.0"?> <!DOCTYPE advise SYSTEM "./toc.dtd"> <advise> <video length ="25" src="rstp://localhost/video1.rm"> <scene id="1"> <group id="1"> <shot id="1"> <keyframe img="./sh_1.jpg"/> <time value="0"/> </shot> <shot id="2"> <keyframe img="./sh_2.jpg"/> <time value="11"/> </shot> </group> </scene> </video> </advise> Technologies in ADVISE - Construction of V-ToC

20. Technologies in ADVISE - Outline • Introduction • Overview of ADVISE • Technologies in ADVISE • For service 1: Construction of Video Table-of-Contents (V-ToC) • For service 2: Video Summarization • For service 3: Video Matching • Conclusion

21. Video Summarization • User may still not be able to know the exact video contents with V-ToC • Video summary can provide all types of information in the video • Objectives: • Select the major contents • Shorten the duration for browsing • Difficulties • No standard method to pick the important contents from video • Importance of contents depends on user’s need • In ADVISE: • We accept user’s input for generating video summary such that the result can be the best suitable for the user Technologies in ADVISE - Video Summarization

22. Inputs for Video Summarization Algorithm • Video features used: • Human faces • Male and female voices • Volume level • Caption text • User’s inputs for customization of the video summary • Weights of different video features • Time constraint for video summary • Clustering control constant Technologies in ADVISE - Video Summarization

23. Step (i) Step (ii) Step (iii) Video Summarization Algorithm • 4 steps to summarize a video • i. Combining extracted video segments • ii. Scoring the extracted video segments • iii. Selecting extracted video segments • iv. Refining the selection result • Example: Step (iv) Technologies in ADVISE - Video Summarization

24. Video Summary in SMIL • SMIL presentation are delivered to user of ADVISE • Can be generated instantly • Can be browsed by the user on the Internet using a stream-based protocol • Resulting SMIL video summary Technologies in ADVISE - Video Summarization

25. Technologies in ADVISE - Outline • Introduction • Overview of ADVISE • Technologies in ADVISE • For service 1: Construction of Video Table-of-Contents (V-ToC) • For service 2: Video Summarization • For service 3: Video Matching • Conclusion

26. Video Matching • Video Matching • Match the extracted video features • Color, motion, shape, etc. • Sequential matching • Non related to video structure • VToC is a tree structure • Can apply tree matching algorithm • Matching related to video structure • In ADVISE, we propose two tree matching algorithms • (1) Non-ordered tree matching algorithm • (2) Ordered tree matching algorithm (Consider temporal ordering) Technologies in ADVISE - Video Matching

27. Input Features for Video Matching • Two video features used • Color histograms feature • Take the first frame of a video shot as the key frame to compare in order to reduce the computational complexity. • Compare the visual similarity. • Shot style feature • Compose of camera motion and length of a video shot. • Select the first camera motion in a video shot as the representative. • Compare the similarity in video pace. Technologies in ADVISE - Video Matching

28. Video A Video B Group a1 Group a2 Group b1 Group b2 (1) Non-ordered Tree Matching Algorithm • Not constrained by temporal ordering • Capture all similar components • Algorithm • Extract features at the bottom level • Propagate the similarity score up to the root level Technologies in ADVISE - Video Matching

29. Video A Video B Group a1 Group a2 Group b1 Group b2 (2) Ordered Tree Matching Algorithm • Constrained by temporal ordering • Temporal ordering can affect the video contents • Reduce the problem to match components in order • Capture only ordered similar components Technologies in ADVISE - Video Matching

30. (2) Ordered Tree Matching Algorithm • Algorithm: • Recursive dynamic programming • Hierarchical matching • From the tree root (video level) • Until shot level, extract the video feature similarity • Reduced the complexity compare with approach (1) Technologies in ADVISE - Video Matching

31. Conclusion • The ADVISE system, which enhanced video browsing and retrieval system on the Internet, is proposed. • The generation and presentation of the image-based video description are developed. • The automation of video summarization into SMIL format is provided. • Two video tree matching algorithms for measuring the similarity between videos are proposed. Conclusion

32. Questions & Answers