1 / 27

Welcome to CLEF 2009

Welcome to CLEF 2009. Carol Peters ISTI-CNR, Pisa, Italy. Happy 10 th Birthday. CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece. CLEF Objectives. Stimulate the development of multilingual IR systems (for European languages !) Create a CLIR/MLIA community

keefe
Download Presentation

Welcome to CLEF 2009

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Welcome to CLEF 2009 Carol Peters ISTI-CNR, Pisa, Italy

  2. Happy 10th Birthday CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  3. CLEF Objectives Stimulate the development of multilingual IR systems (for European languages !) Create a CLIR/MLIA community Construct publicly available test-suites BY • Conducting annual evaluation campaigns • Designing tracks/tasks to meet emerging needs and to stimulate research in the”right” direction Objective: truly multilingual/multimedia systems CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  4. Evolution of CLEF

  5. CLEF Tracks: 2000 - 2009

  6. CLEF 2009 Tracks • Multilingual textual document retrieval (Ad Hoc) • Interactive cross-language retrieval (iCLEF) • Multiple language question answering (QA@CLEF) • Cross-language retrieval in image collections (ImageCLEF) • Multilingual information filtering (INFILE@CLEF) • Cross-language video retrieval (VideoCLEF) • Intellectual property (CLEF-IP) – New this year • Log file analysis (LogCLEF) – New this year • MorphoChallenge (in collaboration with Pascal NoE) Pilot: Grid Experiments (Grid@CLEF) CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  7. CLEF2009 Coordination Adaptive Informatics Research Centre, Helsinki University of Technology, Finland Berlin School of Library and Information Science, Humboldt-Universität zu Berlin Business Information Systems, U. Applied Sciences Western Switzerland, Sierre, Switzerland CEA LIST, France Centre for Evaluation of Human Language & Multimodal Communication Technologies (CELCT), Italy Centruum vor Wiskunde en Informatica, Amsterdam, NL Computer Science Department, U. Basque Country, Spain Computer Vision and Multimedia Lab, U. Geneva, CH Data Base Research Group, U. Tehran, Iran Dept. of Computer Science and Information Systems, U. Limerick, Ireland Dept. of Information Engineering, U. Padua, Italy Dept. of Information Science, U. Hildesheim, Germany Dept. of Information Studies, U. Sheffield, UK Dept. of Medical Informatics and Clinical Epidemiology, Oregon Health and Science U., USA Dept. Medical Informatics, U. Hospitals and University of Geneva, Switzerland Department of Medical Informatics, Aachen University of Technology (RWTH), Germany Evaluations and Language Resources Distribution Agency, Paris, France GERiiCO, Université de Lille, France Idiap Research Institute, Switzerland Information Retrieval Facility (IRF), Austria Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur (LIMSI), Orsay, France U. Nacional de Educación a Distancia, Madrid, Spain Linguateca, Sintef ICT, Norway Linguistic Modelling Lab., Bulgarian Acad Sci Matrixware Information Services, Austria Mediamatics, Delft University of Technology, The Netherlands Mitre Corporation, USA NIST, USA NLE Lab., Universidad Politènica de Valencia, Spain Research Institute for Artificial Intelligence, Romanian Academy, Romania Romanian Institute for Computer Science, Romania Royal Institute of Technology (KTH), Stockholm, Sweden School of Computing, DCU, Ireland Swedish Institute of Computer Science, Sweden TALP Research Center, Universitat Politécnica de Catalunya, Barcelona, Spain. CLEF is coordinated by the Istituto di Scienza e Tecnologie dell'Informazione, Consiglio Nazionale delle Ricerche, Pisa The following Institutions are contributing to the organisation of the different tracks of the CLEF 2008 campaign: CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  8. CLEF Steering Committee Maristella Agosti, U.Padove, Italy Martin Braschler, Zurich, Switzerland Amedeo Cappelli, ISTI-CNR & CELCT, Italy Hsin-Hsi Chen, National Taiwan U., Taipei, Taiwan Khalid Choukri, ELRA/ELDA, Paris, France Paul Clough, University of Sheffield, UK Thomas Deselaers, RWTH Aachen University, Germany Giorgio Di Nunzio, U. Padova, Italy David A. Evans, Clairvoyance Corporation, USA Nicola Ferro, U. Padova, Italy Christian Fluhr, CEA-LIST, Fontenay-aux-Roses, France Norbert Fuhr, University of Duisburg, Germany Frederic C. Gey, U.C. Berkeley, USA Julio Gonzalo, LSI-UNED, Madrid, Spain Donna Harman, NIST, USA Gareth Jones, Dublin City University, Ireland Franciska de Jong, University of Twente, Netherlands Noriko Kando, NII, Tokyo, Japan Jussi Karlgren, SICS, Sweden Michael Kluck, German Institute for International and Security Affairs, Berlin, Germany Natalia Loukachevitch, Moscow State University, Russia Bernardo Magnini, ITC-irst, Trento, Italy Thomas Mandl, U. Hildesheim, Germany Paul McNamee, Johns Hopkins University, USA Henning Müller, University & University Hospitals of Geneva, Switzerland Douglas W. Oard, University of Maryland, USA Anselmo Peňas, LSI-UNED, Madrid, Spain Vivien Petras, Humboldt University, Berlin, Germany Maarten de Rijke, University of Amsterdam, Netherlands Diana Santos, Linguateca, Sintef, Oslo, Norway Jacques Savoy, University of Neuchatel, Switzerland Peter Schäuble, Eurospider Information Technologies, Switzerland Richard Sutcliffe, University of Limerick, Ireland Hans Uszkoreit, German Research Center for Artificial Intelligence (DFKI), Germany Felisa Verdejo, LSI-UNED, Madrid, Spain José Luis Vicedo, University of Alicante, Spain Ellen Voorhees, NIST, USA Christa Womser-Hacker, University of Hildesheim, Germany CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  9. CLEF2009:Track Coordinators Ad Hoc: Abolfazl AleAhmad, Hadi Amiri, Eneko Agirre, Giorgio Di Nunzio, Nicola Ferro, Nicolas Moreau, ArantxaOtegi, Vivien Petras iCLEF: Paul Clough, Julio Gonzalo, Jussi Karlgren QA@CLEF: IñakiAlegria, DavideBuscaldi, Luís Miguel Cabral, Pere R. Comas, CorinaForascu, Pamela Forner, Olivier Galibert, Danilo Giampiccolo, Nicolas Moreau, Djamel Mostefa, Petya Osenova, Anselmo Peñas, Álvaro Rodrigo, Sophie Rosset, Paolo Rosso, Diana Santos, Richard Sutcliff and JordiTurmo ImageCLEF: Brian Bakke, Steven Bedrick, Barbara Caputo, Paul Clough, Peter Dunker, Thomas Deselaers, Thomas Deserno, Ivan Eggel, Mark Oliver Güld, William Hersh, PatricJensfelt, Charles E. Kahn Jr., Jana Kludas, JayashreeKalpathy–Cramer, Henning Müller,Stefanie Nowak, Monica Lestari Paramita, AndrzejPronobis, SaïdRadhouani, Mark Sanderson, Tatiana Tommasi, Theodora Tsikrika and Petra Welter VideoCLEF: Gareth J.F. Jones, Martha Larson and Eamonn Newman INFILE: Romaric Besançon, StéphaneChaudiron, Khalid Choukri, MeriamaLaïb, Djamel Mostefa and IsmaïlTimimi CLEF-IP:FlorinaPiroi, Giovanna Roda, John Tait and Veronika Zenz LogCLEF: Maristella Agosti, Giorgio Di Nunzio, Christine Doran, Inderjeet Mani, Thomas Mandl, Julia Maria Schulz and Alexander Yeh Grid@CLEF: Nicola Ferro and Donna Harman MorphoChallenge:Graeme W. Blackwood, William Byrne MikkoKurimo, Ville T. Turunen and SamiVirpioja CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  10. CLEF 2009: ParticipatingGroups CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  11. CLEF: Trend in Participation CLEF2009: Europe= 81(69); N. America = 18(12); Asia = 16(15), S. America = 1(3), Africa = 1(1) CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  12. CLEF2009:Participants per Track Ad Hoc: 28(26) iCLEF: 6(6) QA@CLEF: 25(29) ImageCLEF: 62 (42) INFILE: 5(1) • VideoCLEF: 7(5) • MorphoChallenge: 9(6) • CLEF-1P: 15 (new) • LogCLEF: 6 (new) • Grid@CLEF: 2 (pilot) CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  13. Participation by Track CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  14. CLEF 2000 – 2009Participation per Track CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  15. CLEF Achievements • Creation of very active multidisciplinary internationalresearch community • Investigation of core issues in MLIA • development of multiple language processing tools • creation of linguistic resources • implementation of appropriate cross-language retrieval models and algorithms for different tasks and languages; • Creation of important reusable test collections and resources in diverse media for a large number of European languages • Significant and quantifiable improvements in the performance of MLIA systems CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  16. Test Collections 2000 News documents in 4 languages GIRT German Social Science database 2009 • A subset of CLEF multilingual corpus of news documents (Ad Hoc WSD-Robust) • Hamshahri Persian newspaper corpus (Ad Hoc Persian task) • TEL Library catalog records plus log files (Ad Hoc TEL task and LogCLEF) • Flickr web-based image database (iCLEF) • ResPubliQA document collection, (QAatCLEF: ResPubliQA) • Transcripts of European parliamentary sessions & French news broadcasts (QAST) • BELGAPICTURE image collection (ImageCLEFPhoto) • Multilingual collections of Wikipedia documents and images (ImageCLEFwiki) • Articles and images from Radiology and Radiography; IRMA collection for medical • image annotation (ImageCLEFmed and medAnnotation) • Dutch and English documentary television programs (VideoCLEF) • Agence France Press (AFPnewswire stories in Arabic, French and English (INFILE) • Patent documents from the European Patent Office (CLEF-IP)

  17. DIRECT@work in CLEF Outline of Talk Why, How, What CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  18. CLEF Publications • 9 CLEF Post-workshopProceedings, Springer LNCS • Nearly1000 Working Notes Reports on-line • Google Scholar> 15,000 publicationswith CLEF – cross-language information retrieval CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  19. CLEF 2008 Proceedings EvaluatingSystemsforMultilingual and Multimodal Information Access 9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, RevisedSelectedPapersSeries: Lecture Notes in Computer Science , Vol. 5706 Peters, C.; Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, Th.; Peñas, A.; Petras, V (Eds.) Many, manythanksto Danilo Giampiccolo, CELCT CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  20. CLEF & TrebleCLEF CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece CLEF 2008 & 2009 is an activity of the TrebleCLEF Coordination Action under the Seventh Framework Programme of the European Commission. TrebleCLEF organises a set of dissemination activities in the multilingual information access field.

  21. Treble-CLEF The CLEF research results have led to development of a new generation of multilingual retrieval system prototypes BUT lack of technology transfer Treble-CLEF extends the CLEF activity by: continuing to promote MLIA R&D via evaluation campaigns; providing a consistent training activity: tutorials, workshops, summer school; producing best practice guidelines for system implementation; providing resources to encourage the multilingual system development. www.trebleclef.eu CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  22. Approach • Evaluation • test collections and laboratory evaluation • user evaluation • log analysis • Best Practices & Guidelines • system-oriented aspects of MLIA applications • collaborative user studies • user-oriented aspects of MLIA interfaces • Dissemination and Training • tutorials • workshops • summer school CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  23. Treble-CLEF Events Workshop on Novel Methodologies for Evaluation in Information Retrieval, ECIR’08, Glasgow, Scotland Workshop on Best Practices for the Development of Multilingual Information Access Systems, Segovia, Spain, June 08 Workshop on Best Practices for System Developers: Bringing Multilingual Information Access to Operational Systems, Winterthur, Switzerland, October 2008 Workshop on Best Practices in Query Log Analysis, London, May 2009 Summer School on Multilingual information Access, Pisa, June 2009 MLIA Technology Day – Dissemination of Results of Best Practices Workshops, 8 December 2009 CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  24. CLEF 2010 Can CLEF continue withoutexternalfundingfor the centralcoordination? ?????? Participate in the discussion on Fridayafternoon CLEF 2009 Workshop 30 September – 2 October, Corfu, Greece

  25. Before I Forget Posters: Put up between 8.00 & 13.30 Thursdaymorning Pool Room 2: ImageCLEFPosters Pool Room 1: AllotherPosters & Demos USB Stick ECDL ConferenceProceedings All Workshop Proceedings Social Dinner - Tonight Hotel restaurant – 8 pm (Badge)

  26. Come to CLEF – and see Europe! CLEF 2003 Trondheim CLEF 2008 Aarhus CLEF 2004 Bath CLEF 2001 Darmstadt CLEF 2005 Vienna CLEF 2007 Budapest CLEF 2000 Lisbon CLEF 2002 Rome CLEF 2006 Alicante CLEF2009 Corfu CLEF2010 ?????

  27. CLEF 2009 Thank you for your attention and ENJOY THE Workshop ! CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008

More Related