1 / 18

Effects of Popularity and Quality on the Usage of Query Suggestions during Information Search

Effects of Popularity and Quality on the Usage of Query Suggestions during Information Search. Can users be induced to take bad query suggestions because they believe many others have used the suggestions in the past?. Diane Kelly, Amber Cushing, Maureen Dostert, Xi Niu and Karl Gyllstrom.

miracle
Download Presentation

Effects of Popularity and Quality on the Usage of Query Suggestions during Information Search

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Effects of Popularity and Quality on the Usage of Query Suggestions during Information Search Can users be induced to take bad query suggestions because they believe many others have used the suggestions in the past? Diane Kelly, Amber Cushing, Maureen Dostert, Xi Niu and Karl Gyllstrom University of North Carolina at Chapel Hill

  2. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Motivation & Background • Detrimental impact of social search features • Social influences on behavior • Psychology, sociology, and economics • Recommender systems, business and marketing • Query suggestions as a social search feature and system usage information • Users’ abilities to identify ‘good’ suggestions 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  3. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Research Questions • Are users influenced by usage information associated with recommended queries? • Can users distinguish between high and low quality query suggestions? • What are users’ perceptions of the usefulness of query suggestions and usage information for open search tasks? 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  4. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Motivation & Background • Use of query suggestions as idea tactics • A move to help users generate new approaches or solutions to information search problems • Potentially useful when user has a limited model of the problem space, is researching a difficult topic, and/or exhausts own ideas for queries 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  5. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Research Questions • What is the relationship among topic difficulty, users’ willingness to take recommendations and their abilities to distinguish between high and low quality queries? 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  6. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Method • Laboratory experiment with experimental search system • Twenty-three undergraduate subjects • Four assigned search topics (15 minutes each) • Closed collection of newspaper articles (3GB, or about 1 million documents) • 8 query suggestions per search topic 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  7. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Method High Quality Queries Low Quality Queries 0-9 Low Usage Information 20-40 High Usage Information 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  8. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Outcome Measures & Analysis • Suggestion Usage/Selection (Binary) • Post-Search Evaluations (5-point scale, 1=low; 5=high) • Query Quality • Confidence in Rating • Willingness to Recommend to Others • Topic Difficulty • Exit Interview and Manipulation Check • Analysis: Logistic regression, t-tests, open-coding 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  9. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Results: General Usage of Suggestions 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  10. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Results: Popularity Were subjects influenced by the usage information associated with recommended queries? No.Usage information (popularity) was not a significant predictor of selection behavior. 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  11. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Results: Popularity What about subjects’ post-search evaluations of query quality? Was there a difference in ratings according to popularity? No. Subjects’ mean ratings were similar. 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  12. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Results: Query Quality Could subjects distinguish between high and low quality query suggestions? Yes.Query quality was a significant predictor of selection behavior. 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  13. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Results: Query Quality What about subjects’ post-search evaluations of query quality? Was there a difference in ratings according to actual query quality? Yes. Subjects rated high quality queries significantly higher than low quality queries. *p<.05 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  14. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Results: Topic Difficulty What was the relationship among topic difficulty and subjects’ willingness to take recommendations? Yes. Subjects were significantly more likely to use suggestions when the topic was hard. 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  15. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Results: Subjects’ Perceptions What were subjects’ perceptions of the usefulness of query suggestions and usage information for open search tasks? • the usage information did not influence their selections that much and was not that important for this task • they ignored the usage information and did their own testing • the query suggestions stimulated thinking outside the box and helped them narrow their searches • the query suggestions were useful when they ran out of ideas or were unsure of how to start the search 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  16. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Summary of Findings • Subjects integrated the suggestions into their searching fairly quickly • The usage information did not influence subjects’ selection • Subjects selected more high quality queries than low quality queries and also rated these higher • Subjects took significantly more suggestions when searching for difficult topics • Suggestions seemed to function as an idea tactic • Task type mediates the effects of social influence on selection behaviors (suggestive) 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  17. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Implications • Design • Tailor suggestions to properties of task (type, difficulty, and stage) • Suggestion algorithms • Presentation methods • Query performance prediction • Method • Data collection and evaluation of suggestion use 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

  18. UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Thank You. Diane Kelly, dianek@email.unc.edu 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

More Related