Object Recognition a Machine Translation

1. Object Recognition a Machine Translation Learning a Lexicon for a Fixed Image Vocabulary Miriam Miklofsky

2. Lexicons • A vocabulary of terms used in a subject • A specialized list of terms • Devices that predict one representation given another representation

3. Dataset • Aligned bitext • Annotated images • Images with regions • Unknown which region of image goes with which word from text

4. EM

5. Clustering • K means clustering • Vector quantize the image region representation • Kullback-Leibler divergence • Relative entropy • Measure of difference of two probability distributions over the same event space

6. Evaluation • Auto annotate images • Quantize regions • Use lexicon to determine word • Annotate image with word

7. Results - Annotation • Base results • 80 words of 371 word vocabulary could be predicted • Retraining • Similar results but some words with higher recall and precision

8. Results(cont.) • Null probability • Recall decreases • Precision increases • Clustering of like words • Recall values of clusters higher than for single words

9. Results -Correspondence • Base results • Some good words up to 70% correct prediction • Null prediction • Predict good words with greater probability • Word clustering • Prediction rate generally increases

10. Evaluation • Human evaluation • Images viewed by hand • Somewhat subjective

12. EM (cont.)

13. KL Divergence