CLEF 2007 Medical Image Annotation Task Budapest, September 19-21 2007

1. CLEF 2007Medical Image Annotation TaskBudapest, September 19-21 2007 An SVM-based Cue Integration Approach Tatiana Tommasi, Francesco Orabona, Barbara Caputo IDIAP Research Institute, Centre Du Parc, Av. Des Pres-Beudin 20, martigny, Switzerland

2. Overview • Problem Statement • Features • Classifier • Results • Conclusions

3. Problem Statement Automatic Image Annotation task’s GOAL: classify a test set of 1000 medical images, having a training set of 11000 medical images. IRMA db: Radiographic images divided into 116 classes according to the IRMA code IRMA code consists of four independent axes: modality - body region - body orientation - biological system Score: errors annotation depends on the level of the hierarchy at which the error is made - a greater penalty is applied for incorrect classification than for a less specific classification in the hierarchy

4. Local Features – SIFT Scale Invariant Feature Transform : local feature descriptor invariant to changes in - illumination - image noise - rotation - scaling - minor changes in viewing direction No keypoint orientation SIFT extracted at only one octave SIFT points = local maxima of the scale space Really the most informative for a classification task ?? • Dense random sampling of the SIFT points better than interest point detectors • Radiographs : low contrast images

5. Vocabulary of Visual Words - SIFT • 30 SIFT points extracted from each of the 12000 images • K-means algorithm with K=500 • Define a vocabulary of 500 words 1500 points Feature Vector of 2000 elements

6. Global Features – Raw Pixels • Images resized to 32x32 pixels • gray value of each pixel normalized to have sum 1 ….. Feature Vector of 1024 elements

7. Support Vector Machine Training data: (x1,y1) ,…,(xm,ym) xiN ,yi {-1, +1} Optimal separating hyperplane: that with maximum distance to the closest points in the training set (·x +b = 0) f(x) = sign(i=1…m i yi·xi + b) the xi with non zero i are SUPPORT VECTORS Non linear SVM: x (x)K(x,y)= (x) ·(y) instead of (· x) Chi-square kernel: K(x,y)= exp{-² (x,y)} ² = i { (||xi-yi||) ² / ||xi+yi|| }

8. Multi-Class SVM one-vs-all - for c classes employs c classifiers. e.g. 3 classes: margin(x) 1 vs 2,3 margin(x) 2 vs 1,3 margin(x) 3 vs 1,2 x  class max(margin) one-vs-one - for c classes employs c(c-1)/2 classifiers. e.g. 3 classes: (x) 1 vs 2  class 2 (x) 1 vs 3  class 3 (x) 2 vs 3  class 3 x  class 3

9. Discriminative Accumulation Scheme - DAS Main idea: information from different cues can be summed together M object classes, each with Nj training images {Iij} i=1,…, Nj j=1,…M For each image we extract a set of P different cue Tp = Tp(Iij), p = 1,…,P So for an object j we have P new training sets {Tp(Iij)} i=1…Nj I’ = test image, M  2,  cue the distance from the separating hyperplane is Dj(p) = i=1…mjpijpyijKp(Tp(Ii j),Tp(I’))+bjp Having all the distances for all the j objects and p cues, the image I’ is classified through j*=argmax j=1…M {p=1…P apDj(p) }ap +

10. Discriminative Accumulation Scheme - DAS Example with two cues: class1 : 2 images class2 : 3 images class3 : 2 images

11. Multi Cue Kernel - MCK Main idea: a new kernel which combines different features extracted from images through a positively weighted linear combination of kernels each of them dealing with only one feature • KMC({Tp (Ii)}p,{Tp(I’)}p) =p=1…P apKp(Tp(Ii),Tp(I’)) • It is possible to • optimize the weighting factors ap and all the kernel parameters together; • works both with one-vs-all and one-vs-one SVM extension to the multiclass problem

12. Experiments • Single feature Evaluation • - 5 random and disjoint train/test splits of 10000/1000 images are extracted • best parameters that giving the lowest average score on the 5 splits • experiments with one-vs-one and one-vs-all SVM multiclass extension SIFT features outperform the raw pixel ones

13. Experiments Cue Integration DAS - distances from the separating hyperplanes associated with the best results of the previous step - cross validation used only to search the best weights for cue integration MCK - cross validation applied to look for the best kernel parameters and the best feature’s weights at the same time In both cases weights varied form 0 to 1

14. Results When the label predicted by the raw pixel is wrong the true label is far from the top of the decision ranking

15. Results The best feature weight for SIFT results higher than that for raw pixels for all the integration methods The number of support vectors for the best MCK run is higher than that used by the correspondent single cue SIFT but lower than that used by PIXEL and DAS.

16. Results First, second and third column contain examples of images misclassified by one of the two cues but correctly classified by DAS and MCK The fourth column shows an example of an image misclassified by both cues and by DAS but correctly classified by MCK

17. Conclusions and Future Work Cue integration pays off Cross Validation pays off We would like to … use various types of local and global descriptors, to select the best features for the task; add shape descriptors in our fusion schemes, which should result in a better performance; exploit the natural hierarchical structure of the data.

18. Thank you