1 / 28

What’s happening in iCLEF? (the iCLEF Flickr Challenge)

What’s happening in iCLEF? (the iCLEF Flickr Challenge). Julio Gonzalo (UNED), Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED), Víctor Peinado (UNED). A piece of good news for CLEF!. People have started to hyperlink MT versions of web pages!. The original URL.

kadeem-emerson
Download Presentation

What’s happening in iCLEF? (the iCLEF Flickr Challenge)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What’s happening in iCLEF?(the iCLEF Flickr Challenge) Julio Gonzalo (UNED), Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED), Víctor Peinado (UNED)

  2. A piece of good news for CLEF! People have started to hyperlink MT versions of web pages!

  3. The original URL

  4. The world is in danger

  5. The world needs Multilingual Information Access research/solutions …(that do care about users)

  6. Retrospective: goals for iCLEF 2006 • Find realistic scenarios where multilingual search arises naturally  multilingual web communities growing around (mostly) non-textual items (ebay, Flickr) Explore Web 2.0 “new text” challenges: folksonomies, blogs, wikis, etc. • Reduce the cost of entry for new participants.  Provide a default multilingual search interface to Flickr. • Improve the cost-effectiveness of the traditional experiment design.  Let participants work on their own experimental designs.

  7. Flickr

  8. Flickr advantages • Naturally multilingual • Web 2.0 challenges (folksonomies) • Replicability (unlike general web image search facilities)

  9. iCLEF 2006 Methodology • Participants must propose their own evaluation measures and experiment design. Topics • Ad hoc: find as many photographs of (different) european parliaments as possible. • Creative: find five illustrations for this article about saffron in Italy. • Visual: What is the name of the beach where this crab is lying on?

  10. Experiments • UNED focused on user’s behaviour • How users deal with native/passive/unknown languages? • Do they actually use CLIR facilities when available? • SICS focused on user’s perceptions • Satisfaction (all tasks) • Completeness (creative,ad-hoc) • Quality (creative) • Sheffield also measured search effectiveness • How many facets were retrieved (creative, ad-hoc) • Was the image found? (visual)

  11. Example result: choice of target languages along time.

  12. Achievements • Found a natural CLIR task • Flickr works for CLIR testing • New perspectives for experiment design. • Moved from bilingual to multilingual search problems.  Stick to Flickr in 2007

  13. Why participation was low? • Kept on recruiting users and forcing them to search whatever we asked for. • Experiment design still too costly. • User populations still too small. • Cost of entry still too high • Building a Flickr interface / adapting the provided interface was not trivial. • Experiment design too open for newcomers. • Schedule collapses with standard tracks. • A problem for CLEF groups potentially interested.

  14. How to improve? • Kept on recruiting users and forcing them to search whatever we asked for. • Experiment design still too costly. • User populations still too small. • Cost of entry still too high • Building a Flickr interface / adapting the provided interface was not trivial. • Experiment design too open for newcomers. • Schedule collapses with standard tracks. • A problem for CLEF groups potentially interested. • Move to Flickr/Web users • Make search task a game • Provide experiment design • Participants ≈ Searchers  Adjust iCLEF calendar

  15. iCLEF 2007 (i) Target search task 海 (sea, wave) Find this image • Clear goal for the user • Clearly defined measures of success (appropriate for a “Hall of Fame” experience) • Invokes different searching strategies

  16. iCLEF 2007 (ii) Evaluation aspects • User’s behaviour (Observational/Log Analysis) • Search strategies with native/passive/unknown languages • User’s perceptions • Usefulness • Difficulty • Clarity, etc. • Search effectiveness • Success rate and its correlation with user profiles, search strategies, etc.

  17. iCLEF 2007 (iii) CL search interface Single web interface for all experiments. Available for: • Groups willing to recruit users and conduct their own experiments. • Any web user wishing to enter the iCLEF Hall of Fame (by finding more images than anyone else!). • Groups willing to experience CL searching from the user’s perspective: who is the best CLEF team searching cross-language? Full experiments Log analysis

  18. iCLEF 2007/2008 schedule • End of October: Feedback from you! on • the multilingual search interface • experiment design • evaluation measures • Mid-December: Guidelines + interface goes public • Groups start their own experiments. • People can start competing in the search challenge. • Organization starts collecting logs from Flickr users • Mid-April: • Flickr/Web user Logs released to interested groups • Thematic session in Chorus Conference on Multimedia Search and Access (hosted by Yahoo!). • Paper submission: aligned with CLEF 2008

  19. Start testing the interface now! http://bender.lsi.uned.es:8080/iCLEF2007 Feedback to julio@lsi.uned.es

More Related