Scalable Supervised Dimensionality Reduction using Clustering

1. Scalable Supervised Dimensionality Reduction using Clustering Troy Raeder, Claudia Perlich, Brian Dalessandro, OriStitelman, Foster Provost m6d

2. What we do 100 Million Browsers Who should we target for a product? Browsing General browsing cookies 100 Million URL’s Shopping at one of our campaign sites Does the ad have an effect? conversion Where should we advertise and at what price? What data should we pay for? 0.0001% to 1% baserate If M6 wins an auction we serve an ad Billions of Auctions per day Attribution? Ad Exchange

3. Agnostic Data A consumer’s online activity gets recorded like this: Purchases Encoded date1 3012L20 date 2 4199L30 … date n 3075L50 The Branded Web The Non-Branded Web Browsing History Hashed URL’s: date1 abkcc date2 kkllo date3 88iok date4 7uiol …

4. Our Model • Our goal: To identify people who are likely to purchase a particular product after seeing an ad. • Our Approach: A massive, sparse classification problem. • Data points: Individual cookies. • Features: are past visited URLs. • Class: Have you ever bought from Brand X? • Our system: Thousands of classification models, with Millions of features per model.

5. Dimensionality Reduction • Our high-dimensional classification models work really well in most contexts, but in some cases fewer dimensions are better. • Rare Events: Some campaigns get very few positives, making it hard to estimate meaningful coefficients. • Cold Start: At the very beginning of a campaign, we have seen fewer positive examples. Same problem. • Flexibility: There are some things that large models just can’t do (speed).

6. Dimensionality Reduction • There are a few obvious options for dimensionality reduction. • Hashing: Run each URL through a hash function, and spit out a specified number of buckets. • Categorization: We had both free and commercial website category data. Binary URL space  binary category space.www.baseball-reference.com Sports/Baseball/Major_League/Statistics • SVD: Singular Value Decomposition in Mahout to transform large, sparse feature space into small dense feature space. www.dmoz.org

7. Dimensionality Reduction • These are all good options, but could we do better? • Motivation: Guarantee sufficient representation in the data. • Intuition: combine similar URLs together. • How should we measure similarity between URLs? • Answer: Model parameters! • Result: supervised multi-task dimensionality reduction in the space of model parameters. • Basic idea: Hierarchical clustering of the URLs themselves.

8. Setup models U R L S Table entries are model parameters (Naïve Bayes)

9. Building the Algorithm • For hierarchical clustering, we need: • A feature space and a distance measure. • Pearson correlation in the space of model parameters. • A method for cutting the tree. • Popularity based.

10. Example Home Kids Health Home News Games & Videos

11. Experiments • We built models off data from 28 campaigns. • Our production cluster definitions have 4,318 features. • We tried to get each of the “challengers” as close to this as we possibly could. • We evaluate on Lift (5%) and AUC.

12. Results

13. Results

14. Results (in lab)

15. Results (in production)

16. Questions? • Thanks for coming!