Personalizing Search on Shared Devices

1. Personalizing Searchon Shared Devices Ryen White and Ahmed Hassan Awadallah Microsoft Research, USA Contact: ryenw@microsoft.com

2. Shared Device Search • 2011 Census: 75% of U.S. households have computer • In most homes that machine is shared between multiple people • Search engines use machine identifiers based on cookies, ids, etc. • Assumes 1:1 mapping from IDs to people for analysis and personalization Shared devices in households • Attributing activity to people (not machines) may improve personalization • Some early indications of effectiveness in prior work (Singla et al., 2014)

3. Is Shared Device Searching Common? • Analyzed comScore search data (all engines, en-US) • Both machine identifiers and person identifiers (users self-identify) • Aside: Within-session shared device search less common: 97% sessions = single user Variations in % machine ids = multi-user withdifferent profile sizes • 6 months = 66% • 3 months = 57% • 1 month= 44% Multi-user (66%)

4. Handling Shared Device Use • Limited current solutions in search engines (user sign-in) • However: Requires user effort to sign in, People don’t sign out so their signals mixed • Some solutions in other domains, e.g., streaming media • Ideally this would happen automatically without user needing to explicitly log in • Search activity attribution methods can help with this …

5. Activity Attribution Challenge • Given a stream of data from a machine identifier, attribute observed historic and new behavior to the correct person • Related work in signal processing and fraud detection • Applications for: Personalization, Advertising, Privacy protection • Question: What is upper bound on gain from attribution-based methods? • We perform ORACLE study with perfect knowledge of who is searching History of search activity on machine New query Which user? User 1 User 2 User 1 User 3 {k user clusters} “From devices to people: Attribution of search activity in multi-user settings” White et al., WWW2014

6. Key Contributions • Introduce attribution-based personalization (ABP) and estimate its value in ORACLE STUDY (perfect knowledge of who is searching) • Show machine vs. person is meaningful for an important application: predicting searchers’ future interests • Identify properties of interest models and queries for which ABP is best • Learn model to predict when to apply ABP on a per-query basis

7. Key Contributions • Introduce attribution-based personalization (ABP) and estimate its value in ORACLE STUDY (perfect knowledge of who is searching) • Show machine vs. person is meaningful for an important application: predicting searchers’ future interests • Identify properties of interest models and queries for which ABP is best • Learn model to predict when to apply ABP on a per-query basis

8. Attribution-Based Personalization (ABP) • Three phases: • Activity attribution and interest model construction for individuals from historic activity • Attribution of newly-observed activity to the correct searcher • Application of that searcher’s specific interest model for personalization

9. Building Interest Models • Build machine and person interest profiles based on the ODP hierarchy • Use result clicks • Category distributions can differ between people and machines, e.g., • Sports/Tennis largest in machine, but only highest for one searcher (B) • Some topics have broad interest, e.g., all searchers are interested in movies  Individualized models could matter • Question is how much and when do they matter most and least?

10. Key Contributions • Introduce attribution-based personalization (ABP) and estimate its value in ORACLE STUDY (perfect knowledge of who is searching) • Show machine vs. person is meaningful for an important application: predicting searchers’ future interests • Identify properties of interest models and queries for which ABP is best • Learn model to predict when to apply ABP on a per-query basis

11. Dataset Per machine or person: 6 months 1 month Evaluation Interest Model Building Time • Two years of comScore logs • Divided into two subsets: • Model building: 6mo of comScore search logs for model building (Jan13 - Jun13) • Evaluation: 1mo immediately following to evaluate predictions (Jul13) • Result clicks from each person/machine used to construct interest models • Machine click thresholds: • MODEL BUILDING: ≥ least 100 clicks • EVALUATION: ≥ 15 clicks

12. Prediction Task • Given a query and interest model, predict ODP categories of next click • Vary identifier type and match type • Identifier type: Machine- or person-based • Match type: All historic activity or on-task activity only • On-task search activity: On-task historic activity as clicks associated with queries with at least one non-stopword term in common with current query • On-task models more accurately reflects state-of-the-art in personalization(Bennett et al. SIGIR12; Teevan et al. WSDM11)

13. Prediction Task: Evaluation Metrics • Precision (P): Did the top predicted label == actual label (1 or 0)? • Recall (R): Did actual label appear in prediction? • F1 score: Harmonic mean of P and R • Reciprocal Rank: If actual label == predicted label, the score assigned was the reciprocal of the prediction rank position 1 ⁄ r, and 0 otherwise • Averaged over all queries in evaluation dataset

14. Given our evaluation set (𝑄)  {timestamp, machine identifier, person identifier, query, {result clicks}} for each query (𝑞) in 𝑄: Evaluation Method For each identifier type in {machine, person}: For each match type in {all, on-task}: • For each 𝑞 ∈ 𝑄: • If identifier type = machine: • If identifier type = person: If match type = all: If match type = on-task: If match type = all: If match type = on-task: Obtain all historic queries from the machine from the model building dataset Obtain all historic queries from the searcher from the model building dataset • Find all historic queries from machine with ≥ 1 non-stopword terms in common with 𝑞 in the model building data • Find all historic queries from searcher with ≥ 1 non-stopword terms in common with 𝑞 in the model building data • Get clicked results for each of the queries and assign ODP categories to the clicked results • Build an interest model (𝑢) comprising the normalized distribution of ODP categories from the assignment • Select top-weighted predicted label in 𝑢, denoted 𝑝𝑙1 • Compute the effectiveness of the method in relation to the ground truth • Average metric values for matchtype across all 𝑞 ∈ 𝑄 to compute the overall performance metrics

15. Prediction Results • Focus on machines w/ 2+ users in the rest of our analysis • Shared device searching is predictable accurately (White et al., WWW14) • Machine-based models are ourbaselines for each of the twomatch types • Gains in precision, F1, and RR • 11-15% in overall perf. • 19-43% for on-task perf. • Focus on F1 for remainder of analysis Recall slightly higher for machine  Machine-based models are asuperset of the person-based models

16. Key Contributions • Introduce attribution-based personalization (ABP) and estimate its value in ORACLE STUDY (perfect knowledge of who is searching) • Show machine vs. person is meaningful for an important application: predicting searchers’ future interests • Identify properties of interest models and queries for which ABP is best • Learn model to predict when to apply ABP on a per-query basis

17. Impact of Additional Factors • Properties of the interest models and query can influence utility of ABP • Model Properties • Model entropy: Entropy of the interest model (low, medium, high) • Relative model size: Fraction of machine-based model • Number of searchers on machine • Query Properties • Click entropy: Diversity of clicks (low, medium, high) • Popularity: Frequency of query (low, medium, high) • Topic: Top-level ODP category • Focus on two highlighted factors (see paper for rest) • Control for task effects by focusing on on-task model variants

18. Impact of Additional Factors • Compute the gains differentially based on features of models and the queries, e.g., • Model entropy, i.e., diversity of the category (c) model on the machine (m) • Query topic, i.e., top-level ODP category of the top-result for the query • When the machine-based model is more diverse, then person-based methods perform better  More benefit from focus • Topics for which specific users already represented (only small n interested) • Others where interests are more broad

19. Key Contributions • Introduce attribution-based personalization (ABP) and estimate its value in ORACLE STUDY (perfect knowledge of who is searching) • Show machine vs. person is meaningful for an important application: predicting searchers’ future interests • Identify properties of interest models and queries for which ABP is best • Learn model to predict when to apply ABP on a per-query basis

20. Applying Model and Query Properties • Train a model to learn when to apply ABP on a per-query basis • Featurized properties of the model and the query based on additional factors: • 130k evaluation queries from 2.5k people (1k machines) • 6mo/1mo build/test, MART-based classifier, 10 fold CV, 100 runs, Compute F1 • Labels: Positives: ABP > Machine-level, Negatives:ABP  Machine-level

21. Selective Application of ABP • Best: 21% ABP, 9% baseline, 70% tied • Predict which model best: • Strong predictive performance (acc.= 0.918) > marginal baseline (0.791) • Top features: MachineModelEntropy (max), RelativeModelSize (0.699 of max), QueryTopic (0.441 of max) • Applying prediction in personalization: Always apply best • ABP performance of 88-96% of the oracle • Much better than always applying ABP • Demonstrates the benefits of intelligently applying ABP for each query

22. Discussion • Shared device searching common • Oracle study showed clear utility from ABP • Focused on click prediction; Other applications need to be examined • Need to performance with automated ABP methods • Alternative self-identification methods need to be examined (e.g., sign-in) • Closer link between people and devices  impact on shared device usage?

23. Summary and Takeaway • Introduced attribution-based personalization, performed oracle study • Observe an increased accuracy in future interest predictions (11-19% in the F1-score, depending on match type) by applying this approach • Gains vary by model/query properties, with selective application of method • Significant opportunities to enhance personalization via tailored models • Future work: • More (non-oracle) studies with different ABP methods • ABP methods for truly personalized ranking and recommendation at scale

24. Shared Device Searching: Distribution • Distribution of users searching • Generally one dominant searcher (44-83% of queries) • Decreases with other users, but still by far the most active + many other less active searchers