MIRACLE M ultilingual I nformation R etriev A l for the CLE F campaign

1. MIRACLEMultilingual Information RetrievAl for the CLEF campaign DAEDALUS – Data, Decisions and Language, S.A. www.daedalus.es Universidad Carlos III de Madrid (UC3M) www.uc3m.es Universidad Politécnica de Madrid (UPM) www.upm.es Partially funded by IST-2001-32174 (OmniPaper) and CAM 07T/0055/2003 projects

2. ImageCLEF 2003 Participation in: • Monolingual task: • English -> English: 5 different runs • Bilingual tasks: • Spanish->English: 6 runs • German->English: 6 runs • French->English: 4 runs • Italian->English: 4 runs • TOTAL: 25 runs

3. System Architecture • IR engine: XAPIAN (based on Probabilistic IR model) • Filtering components: text and word extraction, topic extraction, word count, statistic calculations • Linguistic components: tokenizers, stemmers (based on Porter algorithm), German word decompounding module, stopword filters • Translation components: API to FreeTranslation.com (full text) and ERGANE dictionary (word by word) • Semantic components: Synonym expansion for English (WordNet) • Our idea is to couple these components in different ways to evaluate different approaches and compare the influence of each one in the P/R of the IR process for each language

4. IR Process: Index • All the images are indexed in the same XAPIAN collection • For each image, HEADLINE and TEXT fields are used (without tags and IDs)

5. IR Process: Retrieval • Different runs, basically consisting on: • Create the query from the topic • Execute the query in XAPIAN system • Retrieve 1000 best results (ranked list) For each topic, only the TITLE field and the 1st translation variant are used • Evaluation: four relevance sets (2 judges) • Union (any assessor) / Intersection (both assessors) • Strict (relevant only) / Relaxed (also partially relevant) • In our evaluation, we have considered the intersection-strict, which is the most restrictive

6. Monolingual Runs (en->en) • OR: • Word extraction in topic title  stop word filtering  stemming  weighted OR operator with stems • Intended as the baseline run • ORlem: • Word extraction in topic title  stop word filtering  stemming  weighted OR operator with stems and original words • Idea: measure the effect of stemming • ORlemexp: • Word extraction in topic title  stop word filtering  synonym expansion  stemming  weighted OR operator with stems and original words and synonyms • Idea: measure the effect of increasing the recall despite the penalization in precision • Doc: • Index topic title as document  retrieve similar docs • Idea: Confirm that this is a similar approach to vector space model • ORrf: • Query with OR operator with stems  Top 25 docs  250 most important terms  new weighted OR query • Idea: measure the effect of simplest blind relevance feedback

7. P-R curve (en->en) • Best runs have too high precision values (“the set of relevant documents is not complete”) • Relevance feedback is the worst (“noise due to unappropriate parameter values - 250 terms when the mean length of image description is about 50 words”) • Any kind of term expansion reduces precision (“low number of documents, existence of ambiguity”)

8. Average Precision (en->en) • Best run is weighted OR query and Doc (“in Probabilistic IR model, weighted OR is like term weights in Vector Space Model”) • The evaluation with other relevance sets gives a slight increase in overall precision

9. Bilingual Runs (fr,ge,it,sp->en) • TOR1: • Topic title  FreeTranslation  Word extraction  stop word filtering  stemming  weighted OR operator with stems • Similar to monolingual OR, intended as the baseline run • TOR3: • Topic title  FreeTranslation + ERGANE  Word extraction  stop word filtering  stemming  weighted OR operator with stems • Idea: improve translation by combining different sources • Tdoc: • Topic title  FreeTranslation  Index as document  retrieve similar docs • TOR3exp: • Topic title  FreeTranslation + ERGANE  Word extraction  stop word filtering  synonym expansion  stemming  weighted OR operator with stems and original words and synonyms • TOR3full: • The same as TOR3 but also including topic title in original language • Idea: evaluate the effect of text that cannot be or is incorrectly translated • TOR3fullexp: • Combination of TOR3exp and TOR3full

10. P-R curve (fr,ge,it,sp->en)

11. P-R curve (fr,ge,it,sp->en) • Although all results are similar, TOR1 and Tdoc are the best ones in all cases • Using word by word translation with ERGANE has proved to be worse: translation is not adequate or the expansion of the query makes wider queries thus reducing precision • Again, as in monolingual task, any kind of term expansion reduces precision, if not coping with ambiguity • Spanish, German and Italian have similar results, but French is slightly worse: FreeTranslation is worse for French or the French topics are harder to translate • Spanish->English gives our best individual results !!! • Comparing bilingual/monolingual results, a difference of about 10-15% arises (similar to our participation in CLEF tasks this year)

12. Average Precision (fr,ge,it,sp->en)

13. Conclusions and Future Work • As new-comers to CLEF, we have worked hard to build the infrastructure to be able to easily execute different runs • Simplest approaches have proved to be the best if not handling ambiguity caused by term expansion • Next time…: • POS filtering for syntactic disambiguation to handle ambiguity • Evaluate the effect of using stemming (and its quality) or not in high flexible languages like Spanish/French/Italian • More focus on Spanish: better stemmer, better synonym expansion (directly in Spanish) • Evaluate the quality of translation engines with respect to the IR process