1 / 37

WSCD09 Workshop on Web Search Click Data 2009

WSCD09 Workshop on Web Search Click Data 2009. Survey and evaluation of query intent detection methods David J. Brenes, Daniel Gayo Avello , Kilian Pérez-González . phenoxyphenyl entertainment tonight jewish christmas mortgage calculator eoc phi famington , new mexico

dolan
Download Presentation

WSCD09 Workshop on Web Search Click Data 2009

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WSCD09 Workshop on Web Search Click Data 2009 Survey and evaluation of queryintentdetectionmethods David J. Brenes, Daniel Gayo Avello, Kilian Pérez-González

  2. phenoxyphenyl entertainmenttonight jewish christmas mortgagecalculator eoc phi famington, new mexico "oneidaindiannation" "oneida, ny" and "employment" simcity 4 cheats benihana of novi total body kc ferencmolnar cucumbershampoomoltonbrown wholikesgrinding in wow? solitudeutahski nicknilsson training in orlando homunculitattoo "robertlouisstevenson"+quotes wwiimilitaryinstallationssouth carolina bogen 3016 sch 1830v samsung www.brookdaleliving.com air temperature gauge oldcalvarychurch steelers playoff game

  3. a query log (prettyraw, huh?) onequery(stillraw)

  4. apple

  5. apple

  6. readingtheuser’smind. i.e.inferringqueryintent.

  7. Intentisjustone of thequeriesdimensions.

  8. st. thomas bed and breakfast

  9. intent (transactional?) st. thomas bed and breakfast

  10. intent (transactional?) st. thomas bed and breakfast topic (shopping/travel/lodging?)

  11. intent (transactional?) st. thomas bed and breakfast topic (shopping/travel/lodging?) geographicallocation (VirginIslands?)

  12. Intentisjustone of thequeriesdimensions. querytopic geographicallocation commercial/product/jobseeking ... human vs ‘robot’ queries

  13. KDD Cup 2005LogCLEF 2009 WSCD’09

  14. querytopicclassification KDD Cup 2005LogCLEF 2009 WSCD’09 log analysis and geographic query identification user goalshumans vs robotstopic analysis of queriesetc.

  15. However… Topic and Intent aretheclassicalones and we are mainlyinterestedin queryintentdetection.

  16. Ourproposalforthe WSCD’09

  17. “To replicate most of the techniques to perform automatic query intent detection in addition to study the feasibility of a pooling strategy à-la-TREC to evaluate the different techniques.” “We hope to anticipate some of the benefits and difficulties from a hypothetical bakeoff on broad query classification.”

  18. Queryintent Informational. The intent is to acquire some information assumed to be present on one or more web pages. Navigational. The immediate intent is to reach a particular site. Transactional. The intent is to perform some web-mediated activity. Andrei Z. Broder

  19. Queryintent Informational. The intent is to acquire some information assumed to be present on one or more web pages. Navigational. The immediate intent is to reach a particular site. Transactional. The intent is to perform some web-mediated activity. Andrei Z. Broder difficulttodistinguish...

  20. Queryintent NavigationalTransactional Informational symptoms separation anxiety dogs should snow dogs wear booties what is addison's diseases for dogs yellowpages the huns yellow pages.com

  21. Queryintent NavigationalTransactional Informational symptoms separation anxiety dogs should snow dogs wear booties what is addison's diseases for dogs yellowpages the huns yellow pages.com findthe ‘right’ website findthe ‘right’ answer tellingapartinformationalfromnavigational/transactionalqueriesmatters.

  22. Queryintentdetectionmethodsreplicated Leeet al. 2005, click-through data + documentcollection, ad hocthresholds Liuet al. 2006, click-through data, ad hocevidences Jansenet al.2008, rule-based Brenes and Gayo 2008, click-through data, ad hocevidences thesewerejusttheevaluatedmethods. thoserequiring training and/orexternalcorporawereleftforfutureresearch.

  23. Queryintentdetectionmethodsreplicated • Leeet al. 2005 • Clickdistribution (fromclickthrough-data) and anchor-link distribution (fromdocumentcollection) • Flat distributions informationalqueries • Skeweddistributions  navigational/transactional • Additionally, averagenumber of clicks per query

  24. Queryintentdetectionmethodsreplicated • Liuet al. 2006 • nClicksSatisfied (nCS)number of sessionswheretheuserclicked, at most, nresults (n=2) • top nResultsSatisfied (nRS)number of sessionswheretheuserclicked, at most, the top nresults (n=5) • Additionally, clickdistribution(Lee et al. 2005)

  25. Queryintentdetectionmethodsreplicated • Jansenet al. 2008 • Navigationalqueries: • Containnames of companies, businesses, organizationsorpeople • Containdomainsuffixes • Theyhavelessthanthreeterms • Most of the data obtainedfromFreebasehttp://www.freebase.com

  26. Queryintentdetectionmethodsreplicated • Brenes and Gayo 2008 • Weight of themost popular result (cPopular) • Number of distinctvisitedresults (cDistinct) • Navigationalqueriestendtoappearissolated (cSession)

  27. Proposedevaluationmethod Precision, recall and F-measure 6,624 queries (1‰) weremanuallytagged 10 judges plus theauthors Eachjudgetagged1,000 queries (aprox.) Thirdjudgepolishedinconsistencies

  28. Whataboutthepoolingstrategy? Unfeasible :( 81% actual navigationalqueriesappear in the pool Most of thequeriesflagged as navigational80% of thequeries (5 million) tobemanuallyevaluated

  29. Butthesamplewaspretty “small”… Sure. Biggersamples! Dolores Labs Blog: a mustread!

  30. And what abouttheresults?

  31. Bestachievers • Liuet al. nCS(0.378), nRS(0.398) • Brenes and Gayo cPopular(0.374) • Jansenet al.heuristic rules (0.360) • Promisingfeatures • nRS • usingdomains and websitenames • Monte Carlo simulation (?)

  32. Bestachievers • Liuet al. nCS(0.378), nRS(0.398) • Brenes and Gayo cPopular(0.374) • Jansenet al.heuristic rules (0.360) • Promisingfeatures • nRS • usingdomains and websitenames • Monte Carlo simulation (?) usersbehaviour externalknowledge fillingthe gap

  33. Conclusions and Future work Feasible evaluation methodology Pooling evaluation is unfeasible Larger tagged samples are needed Further refinement to improve performance

  34. Uhh… Questions?

More Related