A fuzzy video content representation for video summarization and content-based retrieval

1. A fuzzy video content representation for video summarization and content-based retrieval Anastasios D. Doulamis, Nikolaos D. Doulamis, Stefanos D. Kollias 2000 • Presented by Mohammed S. Al-Logmani

2. Agenda • Introduction • Motivation/ Problem Statement • Video Sequence Analysis • Fuzzy Visual Content Representation • Video Summarization • Content-Based Retrieval • Experimental Results • Future Work • Conclusion

3. Introduction • The increase amount of digital image & video data requires new technologies for efficient searching, indexing, content-based retrieving & managing multimedia databases. • Drawbacks with keyword annotations: • Large amount of effort for developing them. • Cannot efficiently characterize the rich visual content using only text

4. Introduction Cont. • Content-based algorithms • QBIC • VisualSeek • Virage • Cannot easily applied to video DBs. • Perform queries on every frame is inefficient & time consuming • Videos DBs. are distributed which impose large storage & transmission requirements

5. Introduction Cont. • Content-based sampling algorithms • Extract small but meaningful info. (summarization) • Require a more meaningful representation of visual content than the traditional pixel-based one • Related Work: • A hidden Markov model for color image retrieval • An approach of image retrieval based on user sketches • A hierarchical color clustering method • Construction of a compact image map or image mosaics for video summarization • A pictorial summary of video sequences based on story units

6. Motivation/ Problem Statement • Increase the flexibility of content-based retrieval systems • Provide an interpretation closer to the human perception • Result a more robust description of visual content • possible instabilities of the segmentation are reduced

7. fuzzy representation of visual content • Video summarization • Performed by minimizing a cross correlation criterion among the video frames using a GA. • The correlation is computed using several features extracted using a color/ motion segmentation on a fuzzy feature vector formulation basis. • Content-based indexing & retrieval • The user provides queries (images or sketches) which are analyzed in the same way as video frames in video summarization scheme. • A metric distance or similarity measure is then used to find a set of frames that best match the user's query.

8. Video Sequence Analysis • A color/motion segmentation algorithm is applied for visual content description • Multiresolution Recursive Shortest Spanning Tree (M-RSST) • recursively applies the RSST to images of increasing resolution. (a truncated image pyramid is created) • Produces same results as RSST with less time. • Eliminates regions of small segments

9. Video Sequence Analysis cont. • Factors affect the segmentation efficiency • The initial image resolution level • selected to be 3 (downsampling by 8x8 pixels) • The selection of threshold used for terminating the algorithm • Euclidean distance of the color or motion intensities between two neighboring segments • Terminate the segmentation if no segments are merged from one step to another.

10. Video Sequence Analysis cont.

11. Fuzzy visual content representation • The size & location cannot be used directly • segments # is not constant for each video frame • To overcome this problem, pre-determined classes of color/motion properties • To avoid the possibility of classifying two similar segments to different classes, a degree of membership is allocated to each class • Resulting in a fuzzy classification formulation • Create a fuzzy multidimensional histogram

12. Fuzzy visual content representation Cont. • Example: property (s) is used for each segment. • s takes values in [0,1] • It is classified into Q classes using Q membership functions • degree of membership of s in the nth class

13. Fuzzy visual content representation Cont. • Assume a video frame consists of K segments • First, evaluate the degree of membership of feature si = 1,2, … K, of the ith segment • Then, find the degree of membership of K in the nth class through the fuzzy histogram

14. Video summarization

15. Video summarization Cont. • Extraction of key-frames • Key-frames are extracted by minimizing a cross-correlation criterion, so that the selected frames are not similar to each other. • The generic approach (GA) • Similarities to the traveling salesman problem (TSP). • Initially, a population of m chromosomes is created. • Evaluate the performance of all chromosomes in population P(n) using a correlation measure. • Evaluate the chromosomes quality using fitness functions. • Select appropriate parent so that a fitter chromosome gives a higher number of offspring • The GA terminates when the best chromosome fitness remains constant for a large number of generations

16. Video summarization Cont. • Examined about170 shot, # Kf=6 , Q=3

17. Content-based retrieval • Apply the previous scheme to discard all the redundant temporal video information • The user can submit: • Images (query by example) • Sketches (query by sketch) • Analyze the query using M-RSST • Extract and classify the segments • Apply a distance similarity measure

18. Experimental results

19. Experimental results Cont.

20. Experimental results Cont.

21. Future Work • Increase the system accuracy by developing a fuzzy adaptive mechanism for estimating the distance weights.

22. Conclusion • Presented a fuzzy video content representation • Efficient for: • Video summarization • Content-based image indexing & retrieval • Experimental results indicate that this approach outperforms the other methods for both accuracy and computational efficiency