RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES

1. RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES Mingming Lu, Qiyu Zhang, Wei-Hung Cheng, Cheng-Chang Lu Department of Computer Science Kent State University

2. Data Mining and Knowledge Management • Processing multimedia objects • Defining and extracting features • Feature dimension reduction • Multimedia data retrieval • Knowledge representation and management Kent State University

3. Current Tasks • Off-line data training • Segment images – batch mode • Find region of interest (ROI) • Interface with feature extraction and analysis • Feature domain processing Kent State University

4. Current Tasks (cont.) • Users Interfaces • Reading user-input images • Segmentation • Find ROI • Feature extraction of ROI • Compare with trained data in repository • Return data (images) satisfying certain criteria Kent State University

5. Data Training Image Domain Feature Domain Interface Segmentation Finding ROI Feature Extract Dimension Reduction Image & Feature Data Repository Sending Images for Processing Store Feature Data back Kent State University

6. Image Domain Procesisng • Segmentation – Color VQ, Texture based image segmentation • Find ROI • ROI occupies large area • ROI locates near the image center • ROI contains homogenous texture Kent State University

7. Color-Texture SegmentationApplications • Identify Regions of Interest (ROI) in a scene • Image classification • Image annotation • Object based image and video coding Kent State University

8. Color-Texture SegmentationCurrent Limitations • Many existing techniques work well on homogeneous color regions, while natural scenes are rich in color and texture. • Many texture segmentation algorithms require the estimation of texture model parameters, which is a difficult problem and often requires a good homogeneous region for robust estimation. Kent State University

9. Color-Texture SegmentationAdvantage of Color VQ and Texture based segmentation • Does not attempt to estimate a specific model for a texture region. • Tests for the homogeneity of a given color-texture pattern, which is computationally more feasible than estimation of model parameters. Kent State University

10. Color-Texture SegmentationTwo-Step Process • Color Quantization • Performed in the color space without consideration of spatial distribution of colors. • Label each pixel with a quantized color to form a class-map. • Spatial Segmentation • Performed on the class-map Kent State University

11. Color-Texture SegmentationColor Quantization • Use Peer Group Filtering • As a result, coarse quantization can be obtained while preserving the color information in the original images. • Usually 10-20 colors are needed in the images of natural scenes. Kent State University

12. Color-Texture SegmentationCriteria for Good Segmentation Kent State University

13. Color-Texture Segmentation-A Criterion for Good Segmentation • When the color classes are more separated from each other, J is getting larger. • If all color classes are uniformly distributed over the entire image, J tends to be small. Kent State University

14. Color-Texture SegmentationA Criterion for Good Segmentation • Now let us recalculate J over each segmented region instead of the entire class-map and define the average by • A segmentation which can minimize J is considered a good segmentation. Kent State University

15. Color-Texture Segmentation-Spatial Segmentation • Seed Determination • Seed Growing • Region Merge Kent State University

16. Color-Texture Segmentation-Spatial Segmentation Kent State University

17. ROI Determination • Find ROI – Mechanism • Pixel closer to the center contributes more weight to the region it belongs to. • Region with more pixels tends to get higher weight Kent State University

18. Results of Image Domain Processing • Results of Color Quantization • Results of Finding ROI Kent State University

19. Results of Image Domain Processing V1 = 500, V2 = 1, V3 = 0.5 V1 = 500, V2 = 1, V3 = 0.5 Auto Kent State University

20. Results of Image Domain Processing V1 = 500, V2 = 1, V3 = 0.5 Auto Kent State University

21. Kent State University

22. Interface with Feature Domain • Find the rectangle circumscribing the ROI • Store its coordinate information into to a temporary file for feature domain’s use. Kent State University

23. Feature Domain(Overview) • Two Stages: • Feature Extraction • Dimension Reduction (DR) Feature Domain Interface Image Domain Feature Extract Dimension Reduction Image & Feature Data Repository Store Feature Data back Kent State University

24. Implementations • Acquire ROI information from the image domain • Extract features based on Gabor Filter and color histogram on HSV space • Integrate two feature spaces • Reduce the high feature dimensions to a very low number Kent State University

25. Implementations (cont.) • Calculate the similarity measurement between the query object and the objects in the image repository • Search the similar images in the repository based on similarity index • Output the corresponding retrieval images • Knowledge extraction Kent State University

26. Feature Extraction Algorithm • Gabor Filter Feature • One of the most important wavelets with multi-scale and multi-resolution • Mainly reflect texture information • Color histogram on HSV space • Provide color features Kent State University

27. Gabor Filter Concept • A complete but non-orthogonal basis wavelet set • A significant aspect: localized frequency description – composed of space information Kent State University

28. Gabor(cont.) • A two dimensional Gabor function g(x, y) and its Fourier transform G(u, v) can be written as: Kent State University

29. Gabor(cont.) • Let g(x, y) be the mother Gabor wavelet, then this self-similar filter dictionary can be obtained by appropriate dilations and rotations of g(x, y) through the generating function Kent State University

30. Color Histogram in HSV Space • HSV color space includes • Hue (H) • Saturation (S) • Value (V or Lightness) • Only consider Hue and saturation information, since the lightness of pictures is very sensitive to the surrounding conditions. Kent State University

31. HSV space Figure Kent State University

32. HSV space bands • Design bands in the HSV space • 8 hue bands • 4 saturation bands, • Total 32 sub-spaces • Compute color histogram feature in each sub-space to form 32 feature dimensions eventually Kent State University

33. Feature Integration • Normalize both Gabor filter and HSV color histogram features • Set a weight factor to balance two feature spaces. Usually Gabor filter features will have the bigger weight value. Kent State University

34. DR Algorithm • Disadvantages in the high dimension space • The computational complexity arise sharply • The database indexing becomes difficult • Principal Component Analysis (PCA) • PCA seeks to reduce the dimension of the data by finding a few orthogonal linear combinations (Principal Component “PC”) Kent State University

35. DR implementation • Original feature dimensions • Gabor filter features: 6*5*2 = 60 • HSV color histogram features: 4*8 = 32 • Total dimensions: 92 • Feature dimensions after DR • 10 ~15 dimensions Kent State University

36. Simulation Results in the Feature Domain • We randomly select 11 query pictures as the test samples in this report. • At each query time, at most 14 retrieval pictures are retrieved. • The minimum square error method is served as the similarity measurement. • The value in the tables as below means the positive pictures out of the 14 retrieval pictures. Kent State University

37. Query pic# 1 2 3 4 5 6 7 8 9 10 11 Gabor 6 7 7 4 12 1 1 2 4 3 2 HSV 8 2 9 1 2 3 1 1 4 2 3 Integrated 10 5 11 4 12 3 3 2 5 2 4 Performance between different feature extraction techniques • the integration of Gabor Filter and HSV color Histogram gains the better performance. • See pictures in detail. Click here Kent State University

38. Query pic# 1 2 3 4 5 6 7 8 9 10 11 Integrated 10 5 11 4 12 3 3 2 5 2 4 DR 9 6 5 5 12 2 1 1 4 3 2 Performance between with and without DR applied • The performance after DR applied slightly degrades on average in comparison to the results before DR takes on stage • See pictures in detail. Click here Kent State University

39. More Simulations • Performance between different weight used • Performance between different dimensions retained after DR Kent State University

40. Final Integration Results • Simulation results when both the image domain and the feature domain are used • See the detail pictures, Click here Kent State University

41. Integration • UAV media capture and analysis • WWW based media analysis • Vehicle based media capture and analysis Kent State University

42. Future ResearchExtended to video objects • Object based video coding • Non-object based video coding • Video indexing • Knowledge extraction and management Kent State University

43. Future ResearchData Fusion • Multimodality medical imaging • CT – Structural information • PET – Functional information • Fusion • Knowledge management Kent State University