Magic Search Engine: Bridging the Gap Between Memorable and Relevant Results

1. Amazing The Re:Search Engine Jaime Teevan MIT, CSAIL

2. “Pick a card, any card.”

3. Abracadabra! Case 1Case 2Case 3Case 4Case 5Case 6

4. Your Card is GONE!

5. People Forget a Lot

6. Change Blindness http://www.usd.edu/psyc301/ChangeBlindness.htm

7. Change Blindness http://www.usd.edu/psyc301/ChangeBlindness.htm

8. Re:Search Engine ?

9. Merge Old and New Results Old Merged New

10. We still need magic!

11. Overview • Memorability study • Recognition study • Assumptions • Implementation issues • Evaluation issues • Choose your own adventure

12. Memorability Study • Participants issued self-selected query • After an hour, asked to fill out a survey • 129 people remembered something

13. Data Analysis • Probability of being remembered • Anything? # of words? # of fields? • Features • Result features: clicked, not clicked, last clicked, rank, dwell time, frequency of visit, etc. • Query features: query type, query length, # of search in session, elapsed time, etc. • Remembered rank v. real rank • Map remembered rank to real rank

14. “Memorability”

15. Remembered Results Ranked High

16. Recognition Study • Same set-up as Memorability Study • Follow-up survey: Results the same? • Case 1: New results • Case 2: Random 4 same • Case 3: Clicked to top • Case 4: Same results • Case 5: Intelligent merging • 165 people completed both steps 19% 38% 41% 66% 81%

17. Assumptions • Re-search v. search • Memorable v. relevant • Results change v. stay the same • Hide change v. show change • Forget v. remember as forgettable • Merge v. identify old or new Why? How to test? What if I’m wrong?

18. Implementation Issues • Page of cached result may disappear • Multiple result pages • Identifying repeat queries • Exact query may be forgotten • User identified • Search sessions are not repeat queries

19. Evaluation Issues • Various goals to test • Does a merged list look like the original? • Does merging make re-finding easier? • Is search improved overall? • Lab study • How to set up re-finding task? • Timing differences significant enough? • Longitudinal study – What to measure? • What are good baselines?

20. Choose Your Own Adventure • Re-search v. search • Memorable v. relevant • Results change v. stay the same • Hide change v. show change • Forget v. remember as forgettable • Merge v. identify old or new • Implementation issues • Evaluation issues

21. Choose Your Own Adventure • Re-search v. search • Memorable v. relevant • Results change v. stay the same • Hide change v. show change • Forget v. remember as forgettable • Merge v. identify old or new • Implementation issues • Evaluation issues (Done)

22. Hide Change v. Show Change • Why I think change should be hidden • Example: dynamic menus • How to prove • New results better, called the same or worse • Baseline for testing – 2 lists, change explicit • What if we should show change? • Memorability suggests changes to highlight • Other applications where want to hide change (Done)

23. Memorable v. Relevant • Why I think memorability is important • Relevance at a future date is what matters • Necessary to hide change • How to prove • Baseline for lab study with target first • What if relevance is what’s important? • Mapping between memorable and relevant • Useful related work on implicit feedback (Done)

24. Re-search v. Search • Why I think people repeat searches • Information seeking literature • Re-finding consistently reported as a problem • How to prove • Study shows prefer to follow known paths • Search log analysis • What if people just want to search? • Memorable results ranked first • Other domains where list consistency matters (Done)

25. Merge v. Identify Old and New • Why I think results should be merged • Information need not necessarily one or other • People don’t like to do extra work • How to prove • Search log analysis • Look at what people do in longitudinal study • Lab study – timing becomes an issue • What if people want to identify query type? • Other applications where merging is useful (Done)

26. Results Change v. Stay the Same • Why I think results change • How search engines work • Personalization and dynamic content • How to prove • Track query results • What if results don’t change? • Probably will in future applications • Existing applications where lists change (Done)

27. Forget v. Remember as Forgettable • Why I think people forget • Visual analogy • How to prove • Lab study – Do people find new information? • Longitudinal study – Ever click on new result? • What if remember as forgettable? • Build better model of memorability • Highlight important changes (Done)

28. Implementation Issues • Page of cached result may disappear • Multiple result pages • Identifying repeat queries • User identified • Search sessions are not repeat queries • Exact query may be forgotten (Done)

29. Evaluation Issues • Various goals to test • Does a merged list look like the original? • Does merging make re-finding easier? • Is search improved overall? • Lab study • How to set up re-finding task? • Timing differences significant enough? • Longitudinal study – What to measure? • What are good baselines? (Done)

30. Thank you! Jaime Teevan teevan@mit.edu

31. Strategies for Finding Teleporting Orienteering

32. Why Do People Orienteer? • Easier than saying what you want • You know where you are • You know what you find • The tools don’t work

33. Structural Consistency Important All must be the same to re-find the information! New name

34. Absolute Consistency Unnecessary New name Focus on search result lists

35. Query Changes • Most changes are simple • Capitalization • Phrasing • Word ordering • Word form • New queries shorter • What about longer time horizons? • Recognition v. recall

36. Result List Changes • Tracked 10 queries on Google for a year+ • 1.18 of top 10 disappear each week • Rate of change likely to increase, e.g.: • Raw personalization • Relevance feedback • People forget their queries • 28% of queries forgotten within an hour

37. Example: “neon signs”