1 / 42

The Good, the Bad, and the Random : An Eye-Tracking Study of Ad Quality in Web Search

The Good, the Bad, and the Random : An Eye-Tracking Study of Ad Quality in Web Search . Shuangze Tang Xu Tian. Outline . Introduction Related work Method Results Conclusion. 1. Introduction. SERP: Search Engine Result Page Organic search Vs. Sponsored search. 1. Introduction.

avery
Download Presentation

The Good, the Bad, and the Random : An Eye-Tracking Study of Ad Quality in Web Search

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Good, the Bad, and the Random:An Eye-Tracking Study of Ad Quality in Web Search Shuangze Tang XuTian

  2. Outline • Introduction • Related work • Method • Results • Conclusion

  3. 1. Introduction • SERP: Search Engine Result Page • Organic search Vs. Sponsored search

  4. 1. Introduction • Goal: How users distribute their visual attention on different components of a SERP during Web search tasks.

  5. 2. Related work • Web Search Behavior in General • The quality of the results and their presentation: • Using total time or overall search success (Turpin & Scholer @ SIGIR’06) • Search success is the same for both good and degraded system(Smith & Kantor @ SIGIR’08) • The Search type • Three general classes of user goals: Informational, Navigational and Transactional (Border et al. @SIGIR forum2002) • Searchers are more successful for common queries and common goals (Downey er al. @CIKM’08) • Individual differences • Search experts (White & Morris @SIGIR’07) and Domain experts (White et al. @WSDM’09)

  6. 2. Related work • Eye Tracking on SERPs • Searchers examine a SERP is influenced by the position and relevance of the results (Joachims et al. @ SIGIR’05; Pan et al. @ JCMC 2007; Guan& Cutrell @CHI’07 ) • Longer snippets lead to better search performance for information tasks (Gutrell & Guan @CHI’07) • Two different types of searchers : exhaustive and economic (Aula et al. @INTERACT’05) • The Influence of Ads • Sponsored links (10% to 23% of all links) were presented on a SERP (Hochstotter & Lewandowski @Inf. Sci. 2009) • Intergrating ads with the organic results does not increase their click through rate (Jensen & Spink @ IJIMA 2009)

  7. The Innovation • Former researches mainly focus on how visual attention distributed on 10 organic results • However… additional element in the page • We are focusing on other components

  8. The Innovation • Task Type • Informational Vs. Navigational • Quality of ad: • Good Vs. Bad • Sequence of Good &Bad ads

  9. 3 Methods • Instrument: EYE-TRACKING • Gaze-position: stand for visual attention. • Tools: • TobiiEyeTracker& Tobii Studio

  10. Right ADs 3Method Introduction Top Ads Search Engine Result Pages (SERP) Organic Result

  11. 3.1 Experimental design and Procedure • Variables • Procedure

  12. Task Variables • Task type (informational/navigational) • Quality of the ads (good/bad) shown on the SERPs • Block (G/B/R) the trial belongs to • Condition (GB/BG/RR) the participant was assigned to

  13. Task type example: • Navigational Task(They had to find specific pages) • Informational Task(They had to find factual information)

  14. SERP Elements and • SERP Generation • 10 results Not containing any special elements like maps, videos, images, or deep links • 3 top ads • 5 right rail ads • Related searches 20 of the 32 initial queries contained related searches

  15. SERP Generation(con’d) • Implement our own search interface show in former slide. • Initial task query, the interface showed a locally cached version of the first SERP for the query. • Sequent user-generated queries, the interface queried a commercial Web search engine. • A pool of good ads and of bad quality ads. • The static first SERPs for the initial task queries always contained the same ads from the appropriate pool. (Our focus) • For subsequent, ad from the appropriate pool were randomly selected and integrated into the SERP at runtime. (Not the focus in this experiment )

  16. More detail: Pool of GOOD Quality or BAD Quality Ads. Initial Task query Same Ads Sequent query Ads randomly and integrated

  17. Ad Quality • The good ads Select from the ads shown by commercial web search engines such as Bing, Google, and Yahoo, in response to the initial task queries. • The bad ads Select from the same commercial web search engines by generating queries using a subset of the terms occurring in the initial task queries

  18. Trail Sequences • A trail is one unit of the experiment starting from reading the task description until completing the task • The experiment was divided into 4 blocks, of 8 consecutive trails • Three types of blocks: Good(G), Bad(B) or Random (R) • Each participant was assigned to one of 3 conditions GB (GBGB), BG (BGBG), or RR (RRRR) • The order of the tasks in a 32 trails sequences was randomly assigned

  19. More in detail: • 3 types of Block • Good(G): 8 trails with mostly good ad quality. # of good ads > # of bad ads • Bad(B): 8 trails with mostly bad ad quality. # of good ads < # of bad ads • Random(R): 8 trails with good and bad ad quality equally. # of good ads = # of bad ads

  20. The Experiment divided into 4 blocks, of eight consecutive trails. To make the blocking effect less obvious to the participants, the ad quality in second trial of each G or B block is reversed. (1)All ads: either good or bad //(2)Each participant was assigned to ONE of the 3 conditions GB, BG, RR.//(3)Each condition contains 4 blocks of trials GBGB, BGBG, and RRRR (16 g +16 b) //(4) The order of tasks: randomly assigned. //(5)Each unique tasks was performed in all 3 conditions//(6)The participants saw 32 trials without and special description of block structure and the quality of ads.

  21. Summary of Independent Variables • The main independent variables for each trial: • Task type (informational/navigational) • Quality of the ads (good/bad) shown on the SERPs • Block (G/B/R) the trial belongs to • Condition (GB/BG/RR) the participant was assigned to

  22. Tasks Description (1) To every participant, solve the SAME set of 32 search tasks. • Half are navigational( Find specific pages) • Half are informational(Find factual information) (2) Each task had a description telling the participants what they should do. • Provide them with an initial query for each task (3)Cached results for each initial query. =>consistent initial set of result for each task (4)After the initial SERP was represented, participants were free to proceed as they wished.

  23. Procedure • Each task, the participants with a written task description and corresponding initial query. • Reading the description and query aloud. • Start first search query. (static page, locally stored) • To solve the task, they need navigate to an appropriate web page and point out the solution on it to the experimenter. • After finding solution, answer ”How good was the search engine for this task?” • Finished, fill in a study questionnaire asking about their WEB search experience and practices during the study.

  24. Apparatus • 17” LCD monitor1280x1024 pixels • BrowserInternet Explorer 7 with a windowsize of 1040x996 pixels • Tobii x50 eyetracker(50 Hz ) • software Tobii Studio. Video sample

  25. Participants • 38 participants produced valid eye-tracking data (out of 41), Recruited from a user study pool. • Age (26 ---60) (mean = 45.5, σ = 8.2) • Wide variety of backgrounds and professions. • 21 female and 17 male. • 13 unique task sequences. • 13 participants assigned to the GB condition, • 13 to BG • 12 to RR. • Overall, we got valid eye-tracking data for 1210 trials.

  26. 3.4 Measures • AOIs • (Areas of Interest) • All regions labeled in right pictures are AOIs

  27. 3.4 Measures(cont’d) • Fixation Impact fi(A) • Function: determine the amount of gaze an AOI A received. • From: detected using built-in algorithms of Tobii Studio. The algorithms generate a fixation if recorded gaze locations of at least 100 ms are close to each other. (35 pixel)

  28. 3.4 Measures(cont’d) • Clicks c(A) • Function: count the number of clicks on any links on the SERPs. • c(A) specifies the number of clicks aggregated for the AOI A. • e.g. The AOI top-ads containing all 3 top ads, c(top-ads) is the sum of clicks on any of the top ads.

  29. 3.4 Measures(cont’d) • Time on SEPR t • For each participant and task, the time on SERP t measures the time the participant spent on the FIRST STATIC SERP for a give task. • t including time(all views of the first static SERP as well as the time to first click. )

  30. 4.RESULT • Gaze on the first static SERP represents 88% of the total gaze on all SERPs • Focus on 3 aspects of gaze on SERPs: • Task type • ads quality • orders of good/bad ads

  31. 4.1 General Gaze Distribution on SERPs • A gaze heat map describing the distribution of visual attention. • This picture the well-know gaze distribution referred to as “Golden triangle” and “F-shaped pattern.”

  32. 4.1 General Gaze Distribution on SERPs The top ads receive as much attention as result found on the fold(at position 6,7) Most Visual attention was devoted to the top few organic results.

  33. 4.1 General Gaze Distribution on SERPs • Gaze and click : in general agreement, some differences. • There are more clicks that attention on top results • Top and right rail ads receive a higher fraction of visual attention than of clicks • top and right rail ads receive a higher fraction of visual attention than of clicks

  34. 4.2 Effects of Task Type Average fixation impact for SERP elements, broken down separately for informational and navigational tasks.

  35. 4.3 Effects of Ad Quality Twice visual attention to top ads when ads were of good quality No reliable effects of ad quality on this part Average fixation impact for SERP elements, broken down separately for SERPs containing good and bad ads.

  36. 4.4 Sequence Effects

  37. 4.4 Sequence Effects(click)

  38. 4.4 Discussion • The order of good/badad strongly affected the search behavior. • SERP has Random order of good/bad ads =>ignore • SERP has consistent Good ads => more attention • Predictability is an important factor

  39. 4.5 BlockingEffects Drop >30%

  40. 5. conclusion • the top few organic result which is stronger for informational than for navigational tasks. • Furthermore, the quality of ads has a significant influence on users search interaction. Good ads could attract more visual attention • Finally, gaze were strongly related to the order in which good and bad ads were presented. when ad quality varied randomly, participants are more likely ignore the ads, even though the ads were good on half of the trials.

  41. Future work • This research represents a first step in understanding how task, ad quality and sequence influence search interaction. • In our study, we focused on a specific static SERP composition which always consisted of 10 textual organic results with top and right rail ads. • next step: • Add more SERP compositions, e.g., images, maps or deep links • In addition, we would like to explore how the quality of ads interacts with the quality of organic results. • Finally, Develop more modelsof search processes and strategies

  42. Thanks!

More Related