1 / 45

TMLE ANALYSIS

TMLE ANALYSIS. The Causal effect of display advertising on conversion. NYC Predictive Analytics Ori Stitelman ( ori@media6degrees.com ) Brian Dalessandro Claudia Perlich foster provost August 11, 2011. QUESTION OF INTEREST. WHAT IS THE EFFECT OF DISPLAY ADVERTISING

gur
Download Presentation

TMLE ANALYSIS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TMLE ANALYSIS The Causal effect of display advertising on conversion • NYC Predictive Analytics • Ori Stitelman (ori@media6degrees.com) • Brian Dalessandro • Claudia Perlich • foster provost • August 11, 2011

  2. QUESTIONOF INTEREST WHAT IS THE EFFECT OF DISPLAY ADVERTISING ON CUSTOMER CONVERSION? ?

  3. WHAT ISTMLE? A semi-parametric method for estimating causal parameters that directly answer a business question of interest?

  4. WHAT ISTMLE? A semi-parametric method for estimating causal parameters that directly answer a business question of interest? Semi-parametric Realistic assumptions Causal Parameters Effect/Impact (NOT Coefficients) Directly Actionable

  5. OUTLINE • Background: Display advertising & • Peek at results • A/B Testing • Alternative Approaches (NICE) • Results • Conclusion

  6. THE BROWSER PROCESS 1. Observe people taking actions and visiting content 2. Use observed data to build list of prospects 3. Subsequently observe same browser surfing the web the next day 4. Browser visits a site where a display ad spot exists and bid requests are made 5. Auction is held for display spot 6. If auction is won display the ad 7. Observe browsers actions after displaying the ad

  7. Results Preview 1.05X 1.11X 0.92X 2.26X 2.62X 1.31X TELECOM COMPANY A TELECOM COMPANY B TELECOM COMPANY C

  8. GENERAL APPROACH ? 1. State Question P 2. Define Causal Assumption/Likelihood Ψ(P) 3. Define Parameter Ψ(Pn) 4. Estimate Parameter

  9. ? 1. STATE QUESTION What is the effect of DISPLAY ADVERTISING on customer CONVERSION? DISPLAY ADVERTISING Showing/Not showing a browser a display ad. CUSTOMER CONVERSION Visiting the advertisers website in the next 5 days.

  10. P 2. DEFINE CAUSAL ASSUMPTIONS/LIKELIHOOD O = (W,A,Y) ~ P0 W – Baseline Variables A – Binary Treatment (Ad) Y – Binary Outcome (Site Visit) gA • P0(O) = P(W)P(A|W)P(Y|A,W) QY QW

  11. DATA STRUCTURE: OUR VIEWERS Head Shape Color Sex CHARACTERISTICS (W) TREATMENT (A) Ad No Ad CONVERSION (Y) No Yes

  12. Ψ(P) 3. DEFINE PARAMETER LIKELIHOOD • P0(O) = P(W)P(A|W)P(Y|A,W) DISTRIBUTION UNDER INTERVENTION • P0,a(O) = P(W)P(Y|A=a,W) = • P0,a(O) = P(W)P(Ya|W)

  13. Ψ(P) 3. DEFINE PARAMETER • P0,a(O) = P(W)P(Ya|W) 1. ADDITIVE IMPACT • ΨAI(P)=E[YA=ad] – E[YA=no ad] 2. RELATIVE IMPACT • ΨRI(P)=E[YA=ad]/E[YA=no ad]

  14. Ψ(Pn) 4. ESTIMATE PARAMETER 1. PARAMETERS ARE COMBINATIONS OF TREATMENT SPECIFIC CONVERSION RATES • E[YA=ad]E[YA=no ad] 2. SO WE CAN COMBINE ESTIMATES OF THESE THESE RATES • φn,ad/φn,no ad • φn,ad-φn,no ad

  15. Ψ(Pn) 4. ESTIMATE PARAMETER • Optimal Experiment • A/B Testing • MLE Based Substitution Estimator (MLE) • Inverse Probability Estimators (IPTW) • Double Robust Estimating Equations (DR-IPW) • Targeted Maximum Likelihood Estimation (TMLE) NICE

  16. OPTIMALEXPERIMENT: Compare conversion rates of seeing an ad to conversion rate without ad for same individuals. 3.6 per 1,000 1.2 per 1,000

  17. OPTIMALEXPERIMENT: OR OBSERVE OUTCOME SHOW AD? BROWSER OR

  18. REALITY: OR OR OR OR

  19. OPTIMAL EXPERIMENT: Compare conversion rates of seeing an ad to conversion rate without ad for same individuals. 3.6 per 1,000 1.2 per 1,000

  20. COMMON APPROACH:A/B TESTING Since we can not both treat and not treat same individuals. Randomization is used to create “equivalent” groups to treat and not treat. 3.4 per 1,000 1.6 per 1,000

  21. SIGNIFICANT COSTSASSOCIATED WITH DOING A/B TESTING WELL • Cost of displaying PSAs to the control(untreated group). • Overhead cost of implementing A/B test and ensuring that it is done correctly (Kohavi et al.) • Wait time necessary to evaluate the results.

  22. Non INVASIVECAUSAL Estimation (NICE) Estimate The Effects In The Natural Environment (Observed Data)

  23. “WHAT IF”CAUSAL ANALYSIS ADJUSTING FOR CONFOUNDING Need to adjust for the fact that the group that saw the advertisement and the group that didn’t may be very different.

  24. TWO MACHINES:QY,n AND gA,n gA Pn(A| ) gA,n Pn(Y| ) Pn(Y| ) QY QY,n

  25. ESTIMATING QY AND gA • Many tools exist for estimating binary conditional distributions. • Logistic regression, SVM, GAM, Regression Trees, etc. • Data adaptive estimation methods that use cross validation. • SuperLearner (R package, Eric Polley) • Causal analyses will benefit from the advances in parallelized routines for data adaptive estimation.

  26. INVERSE PROBABILITY WEIGHTED ESTIMATORS (IPTW) Strategy Pn(A| ) • Adjust For Confounding Through gn. • Weight individuals that are unlikely to be shown an advertisement more than individuals that are likely to be shown an advertisement = 1/2 Estimator Pn(A| ) = 1/10

  27. MLE BASED SUBSTITUTION ESTIMATOR (MLE) Strategy • Adjust For Confounding Through Q. • Predict how each observed browser will behave had they been shown an ad and had they not been shown an ad. QY QY QY,n Estimator QY,n

  28. DOUBLE ROBUST ESTIMATORS What if QY,n or gA,n are broken? The MLE based estimator and IPTW rely on consistent estimates of Q and g respectively. gA P(A| ) gA,n P(Y| ) QY P(Y| ) QY,n

  29. AUGMENTED – IPTW (A-IPTW) Strategy • Adjust For Confounding Through Q and g. • Augments IPTW estimator with information from Q. • Alternatively adjusts MLE with information from g. OR Estimator

  30. TARGETED MAXIMUM LIKELIHOOD ESTIMATOR (TMLE) Strategy • Adjust For Confounding Through Q and g. • Predict how each observed browser will behave had they been shown ad and had they not been shown ad. • The new machine Q* is calibrated with concern for the parameter of interest. • R package “tmle” at CRAN QY Ψ(QY) Ψ(QY) Q*Y,n Estimator QY Q*Y,n

  31. CREATING Q*Yn • QYn is updated to Q*Yn using a clever covariate that is a function of g. • Update is done through the use of a parametric submodel. • Update is a univariate regression with the initial QYn as an offset. gA QY QY Ψ(QY) Q*Y,n

  32. SAMPLING/ANALYSIS • Select Prospects that we got a bid request for on day 0. • Observe if treated on day 1. For those treated A=1 and those not treated A=0. Collect W. • Create outcome window that is the next five days following treatment and observe if event occurs. • Estimate parameters using the methods previously described.

  33. RESULTS

  34. IPTW AND A-IPTW BLOW UP UNDER POSITIVITY VIOLATIONS

  35. METHOD VALIDATION: A/B Test vS TMLE

  36. METHOD VALIDATION: Negative Test

  37. gross conversion rates Are only part of the story -0.2% TELECOM COMPANY B TELECOM COMPANY C TELECOM COMPANY A

  38. Effectiveness varies by marketer & campaign 1.08X 3.77X 4.23X 1.08X B2B COMPANY A B2B COMPANY B

  39. Effective creative and targeting drive lift across marketers and verticals 1.11X 2.33X 1.17X 1.84X 1.34X 1.71X 3.00X 2.57X CAR RENTAL RESTAURANT AIRLINE HOTEL

  40. SUMMARY RESULTS Average relative LIFT OF 90% for m6d PROSPECTING For 25 out of 30 marketers, relative lift was higher for prospecting candidates than for retargeting candidates .

  41. CONCLUSIONS • Estimating causal effects allows one to directly estimate the impact of the display advertisement. • Causal effects can be estimated in the observed data. • Display advertising works in varied ways. Causal analysis allows us to estimate effects on a case by case basis. • All methods for estimating effects are not equal. • These methods may be used for assessing other types of causal effects. • What is next?

  42. Visit our blogmedia6degrees.com/blog

  43. OTHER RESOURCES FOR TMLE Enter “Slide Show” view and click image to link to site.

  44. Acknowledgements Edward Capriolo Brian Dalessandro Rod Hook Brian May Ryan Ottobre Claudia Perlich Tom Phillips Mark van derLaan Susan Gruber Eric Polley Foster Provost

  45. references • O. Stitelman, B. Dalessandro, C. Perlich, and F. Provost. Estimating The Effect Of Online Display Advertising On Browser Conversion. In Proceedings of KDD, Annual International Workshop on Data Mining and Audience Intelligence for Online Advertising, ADKDD ’11. • M. van derLaan and S. Rose. Targeted Learning: Causal Inference for Observational and Experimental Data. New York, NY: Springer Publishing Company, 2011. http://www.targetedlearningbook.com/ • ‘tmle’ R Package http://cran.r-project.org/web/packages/tmle/index.html • R. Kohavi and R. Longbotham. Unexpected results in online controlled experiments. ACM SIGKDD Explorations Newsletter, 12(2):31–35, 2010. • R. Lewis and D. Reiley. Does retail advertising work: Measuring the effects of advertising on sales via a controlled experiment on yahoo. Technical report, Working paper, 2010. • D. Chan, R. Ge, O. Gershony, T. Hesterberg, and D. Lambert. Evaluating online ad campaigns in a pipeline: causal models at scale. In Proceedings of KDD, KDD ’10, pages 7–16, New York, NY, USA, 2010. ACM.

More Related