1 / 37

Jaime Teevan MIT, CSAIL

Amazing. The. Re:Search Engine. Jaime Teevan MIT, CSAIL. “Pick a card, any card.”. Abracadabra!. Case 1 Case 2 Case 3 Case 4 Case 5 Case 6. Your Card is GONE!. People Forget a Lot. Change Blindness. http://www.usd.edu/psyc301/ChangeBlindness.htm. Change Blindness.

pshipley
Download Presentation

Jaime Teevan MIT, CSAIL

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Amazing The Re:Search Engine Jaime Teevan MIT, CSAIL

  2. “Pick a card, any card.”

  3. Abracadabra! Case 1Case 2Case 3Case 4Case 5Case 6

  4. Your Card is GONE!

  5. People Forget a Lot

  6. Change Blindness http://www.usd.edu/psyc301/ChangeBlindness.htm

  7. Change Blindness http://www.usd.edu/psyc301/ChangeBlindness.htm

  8. Re:Search Engine ?

  9. Merge Old and New Results Old Merged New

  10. We still need magic!

  11. Overview • Memorability study • Recognition study • Assumptions • Implementation issues • Evaluation issues • Choose your own adventure

  12. Memorability Study • Participants issued self-selected query • After an hour, asked to fill out a survey • 129 people remembered something

  13. Data Analysis Probability of being remembered • Anything? # of words? # of fields? • Features • Result features: clicked, not clicked, last clicked, rank, dwell time, frequency of visit, etc. • Query features: query type, query length, # of search in session, elapsed time, etc. • Remembered rank v. real rank • Map remembered rank to real rank

  14. “Memorability”

  15. Remembered Results Ranked High

  16. Recognition Study • Same set-up as Memorability Study • Follow-up survey: Results the same? • Case 1: Old results • Case 2: New results • Case 3: Clicked to top • Case 4: Intelligent merging • 92 people have completed both steps 16% 74% 65% 17%

  17. Assumptions • Re-search v. search • Memorable v. relevant • Results change v. stay the same • Hide change v. show change • Forget v. remember as forgettable • Merge v. identify old or new Why? How to test? What if I’m wrong?

  18. Implementation Issues • Page of cached result may disappear • Multiple result pages • Identifying repeat queries • User identified • Search sessions are not repeat queries • Exact query may be forgotten

  19. Evaluation Issues • Various goals to test • Does a merged list look like the original? • Does merging make re-finding easier? • Is search improved overall? • Lab study • How to set up re-finding task? • Timing differences significant enough? • Longitudinal study – What to measure? • What are good baselines?

  20. Choose Your Own Adventure • Re-search v. search • Memorable v. relevant • Results change v. stay the same • Hide change v. show change • Forget v. remember as forgettable • Merge v. identify old or new • Implementation issues • Evaluation issues

  21. Choose Your Own Adventure • Re-search v. search • Memorable v. relevant • Results change v. stay the same • Hide change v. show change • Forget v. remember as forgettable • Merge v. identify old or new • Implementation issues • Evaluation issues (Done)

  22. Hide Change v. Show Change • Why I think change should be hidden • Example: dynamic menus • How to prove • New results better, called the same or worse • Baseline for testing – 2 lists, change explicit • What if we should show change? • Memorability suggests changes to highlight • Other applications where want to hide change (Done)

  23. Memorable v. Relevant • Why I think memorability is important • Relevance at a future date is what matters • Necessary to hide change • How to prove • Baseline for lab study with target first • What if relevance is what’s important? • Mapping between memorable and relevant • Useful related work on implicit feedback (Done)

  24. Re-search v. Search • Why I think people repeat searches • Information seeking literature • Re-finding consistently reported as a problem • How to prove • Study shows prefer to follow known paths • Search log analysis • What if people just want to search? • Memorable results ranked first • Other domains where list consistency matters (Done)

  25. Merge v. Identify Old and New • Why I think results should be merged • Information need not necessarily one or other • People don’t like to do extra work • How to prove • Search log analysis • Look at what people do in longitudinal study • Lab study – timing becomes an issue • What if people want to identify query type? • Other applications where merging is useful (Done)

  26. Results Change v. Stay the Same • Why I think results change • How search engines work • Personalization and dynamic content • How to prove • Track query results • What if results don’t change? • Probably will in future applications • Existing applications where lists change (Done)

  27. Forget v. Remember as Forgettable • Why I think people forget • Visual analogy • How to prove • Lab study – Do people find new information? • Longitudinal study – Ever click on new result? • What if remember as forgettable? • Build better model of memorability • Highlight important changes (Done)

  28. Implementation Issues • Page of cached result may disappear • Multiple result pages • Identifying repeat queries • User identified • Search sessions are not repeat queries • Exact query may be forgotten (Done)

  29. Evaluation Issues • Various goals to test • Does a merged list look like the original? • Does merging make re-finding easier? • Is search improved overall? • Lab study • How to set up re-finding task? • Timing differences significant enough? • Longitudinal study – What to measure? • What are good baselines? (Done)

  30. Thank you! Jaime Teevan teevan@mit.edu

  31. Strategies for Finding Teleporting Orienteering

  32. Why Do People Orienteer? • Easier than saying what you want • You know where you are • You know what you find • The tools don’t work

  33. Structural Consistency Important All must be the same to re-find the information! New name

  34. Absolute Consistency Unnecessary New name Focus on search result lists

  35. Query Changes • Most changes are simple • Capitalization • Phrasing • Word ordering • Word form • New queries shorter • What about longer time horizons? • Recognition v. recall

  36. Result List Changes • Tracked 10 queries on Google for a year+ • 1.18 of top 10 disappear each week • Rate of change likely to increase, e.g.: • Raw personalization • Relevance feedback • People forget their queries • 28% of queries forgotten within an hour

  37. Example: “neon signs”

More Related