Color-Attributes-Related Image Retrieval

1. Color-Attributes-Related Image Retrieval Student: Kylie Gorman Mentor: Yang Zhang

2. Problem and Solution • Content based image retrieval is a common problem in computer vision • Object-related image retrieval is a popular area related to this issue • Attributed-related image retrieval is a possible solution • Enable a person to retrieve an image based on attributes of an object • Some people have tried to use color as a starting point, but this is still a very novel concept

3. Related Papers • Van De Weijer, Joost, et al. "Learning color names for real-world applications."Image Processing, IEEE Transactions on 18.7 (2009): 1512-1523. • Keen,Noah. “Color Moments”(2005). • Shahbaz Khan, F., et al. "Color attributes for object detection." Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012.

4. Background • Color moments: measures that differentiate images based on color • 3 moments per box based on mean, standard deviation, and skewness • Higher difference of color moments means less similar than two images with lower DOM value • Learning color names from real-world images is more useful than chip-based color mapping • Using color attributes in conjunction with object detection provides more accurate results

5. Beginning Code • Separates an HSV image into boxes and calculates the color moments of each box • One moment for mean, standard deviation, and skewness • My contribution: • Function for HSV images should also take in corresponding binary images • Break up binary image into corresponding blocks of HSV image • Locate parts of image that contain desired object • Concatenate these boxes into a single matrix to isolate object and only take the boxes from HSV image that contain the object

6. Further Contribution • Use Google images and calculate feature matrix of HSV image • Concatenate all of the matrices calculated and use PCA (Principal Component Analysis) function in MATLAB • PCA can be used for image compression • Store result and multiply by each individual feature matrix

8. Current Contribution • Mixture Model: a probabilistic model for representing the presence of subpopulations within an overall population • Gaussian Mixture Model: a mixture of K multivariate Gaussian distributions. • GMM clusters the existing points • [means, covariances, priors] = vl_gmm(data, numClusters);

9. Fisher Vector • Since all of the data consists of real-world images, all of the images have different number of vectors • Fisher vector will allows us to quantize the vectors • Similar to Bag of Words: will compare local descriptors to a dictionary obtained with GMM • Different than Bag of Words: instead of storing visual word occurrences, stores difference between dictionary elements and pooled local features • Compute Fisher Vector encoding with VLFEAT MATLAB interface • encoding = vl_fisher(datatoBeEncoded, means, covariances, priors);

10. Current Progress and Future Plans • Read related papers and understand background concepts • Validate current code with small data set of real-world images • Add more features to the system to improve its performance • Apply code to larger data set • Collecting Dataset: There are not any existing color image datasets on the Internet. Use automatic image collecting tool to create our own color object dataset. • Also include object detection • Possible Bonus: Implement novel ideas about general attribute image retrieval system. Determine if it is effective or not.