Can People Collaborate to Improve the relevance of Search Results?

1. Florian Eiteljörge Can People Collaborate to Improve the relevance of Search Results? Florian Eiteljörge eiteljoerge@stud.uni-hannover.de 1

2. Florian Eiteljörge Outline • Web search & social search techniques • Phase one: Study setup & results • Phase two: Study setup & results • Discussion

3. Florian Eiteljörge Web Search • Search engines heavily used on internet • studies indicate: 50% of web search sessions fail • Idea: use social search techniques to improve web search

4. Florian Eiteljörge Social search techniques • Idea • people search for something and give (implicit) feedback by clicking on result items • Most clicked items seem to be mostly relevant – so they will be ranked higher next time. • Problem • users tend to click on the top result items • popular sites get even more popular, even if there are new high-quality pages that would be more relevant ("rich-get-richer" phenomenon)

5. Florian Eiteljörge What is the paper about? • Authors had three hypothesis related to social search techniques: • H1: Users will prefer to rate results at the top of the result lists, whether the results are randomized, or in the order that Google presents them. • H2: Users explicit relevance rankings are not biased by the rank of the result list [while implicit feedback is biased] • H3: For some types of queries people's collaborative effort can produce better ordering of search results. • The authors developed a search engine environment to capture user respond by presenting Google's top ten results in randomized order to test the above hypothesis

6. Florian Eiteljörge Study setup – phase one (rating) • 145 participants were invited by mail to rate search results for their relevance • participants had the possibility to rate any number of results of preselected queries in the most frequent categories (shopping, health, technology, business, computers, arts) • participants were free to choose categories and queries they wanted to rate • the result items were presented in random order • Google-like result item layout • relevance was measured on a 4-point scale: highly relevant, relevant, don’t know, not relevant • after rating queries, each participant was asked to answer a short survey to determine how experience in searching affects the relevance perception

7. Florian Eiteljörge Results first bar: percent of selection of the item for ratingsecond bar: percent of times when item was rated as highly relevant

8. Florian Eiteljörge Results – phase one • participants preferred to rate the first two items (H1 confirmed) • participants explicit feedback not biased in general (H2 mostly confirmed) • feedback for the first item is biased: rated highly relevant in 70% of the times (even if participants were told the order is randomized)

9. Florian Eiteljörge Study setup – phase two (evaluation) • 20 participants were invited to choose if they prefer the results based on the explicit user-feedback or the Google-results • the invited participants self-identified themselves as novice searchers • both result-lists were displayed side-by-side • the new ranking was created with the following formula:score = 3 x highly-relevant-count + 2 x relevant-count + don’t-know-count + (-1) x not-relevant-count

10. Florian Eiteljörge Results – phase two • in some categories the users rate result items very different from Googlee.g.: shopping (digital cameras, walking shoes) – a mean difference in ranking of 4.2 • in some categories users agree with the Google rankinge.g.: Business (Microsoft Bid for Yahoo, Online Advertisement) – a mean difference of 0.8 • 70% of the participants rated the user-based ordering higher than the Google-ordering; these participants chose to rate queries in the categories shopping, computers and arts • the other 30% preferred the Google-ranking while choosing to rate queries of the categories business and technology

11. Florian Eiteljörge Conclusion • people prefer the top result items • explicit feedback is not biased in general • in some categories the Google-ranking is very inconsistent to the users ranking

12. Florian Eiteljörge Discussion

13. Florian Eiteljörge Presentation based on • Morris MR, Horvitz E. SearchTogether: an interface for collaborative web search. Symposium on User Interface Software and Technology. 2007:3-12http://www.grouplens.org/system/files/p283-agrahri.pdf