Uplift Analysis with the Quadstone System

1. Uplift Analysiswith the Quadstone System Monday, 7th January 2005 7.30am PST / 10.30am EST / 3.30pm GMT / 16.30 CET Any trouble getting into the conference call: contact support@quadstone.com.

2. How to ask questions • Return to Meeting Manager • Use Chat

3. Agenda MOTIVATION Demo 1: Up-sell example (binary outcome) When is Uplift Modelling important? Demo 2: Deep-sell example (continuous outcome) TECHNICAL CONSIDERATIONS Practical considerations and guidelines Small population issues and extensions The quality measure: Qini TRIAL How to get a trial copy & datasets Uplift Analysis Nicholas J. Radcliffe Chief Technology Officer

4. “We have to find a way of making the important measurable, instead of making the measurable important” — Robert McNamara “I know half the money I spend on advertising is wasted,but I can never find out which half” — John Wanamaker

5. Demo 1: Up-sell example Binary outcome SCENARIO Mobile phone company 3G MMS Video phone promotion Some mass advertising - non-targeted customers can purchase Direct calling campaign to drive further sales Random 250k chosen from 10m base for trial c. 75k actually targeted; c. 175k as control

6. When is Uplift Modelling Important?

7. Two Separate Benefits • Not targeting people who are little affected • Reponse: Don’t spend money targeting or offer discounts to people who will buy anyway • Attrition: Don’t spend money trying to save people who will go anyway • Targeting people with low probability but high responsiveness • Response: Do spend money on people who aren’t very likely to buy if you do, but are very responsive to offers/contact • Attrition: Do spend money trying to save people who aren’t at huge risk of attrition, but can be made much more likely to stay

8. Pre-existing knowledge of product Many influences 20% off coupon When would a conventional model be misled? High pre-existing purchase rate

9. When are negative effects likely? • Sometimes, our actions actually drive customers away, especially when: attrition risk dissatisfied / angry customers risqué / offensive communications intrusive contact mechanisms forgotten standing charges

10. Demo 2: Deep-sell example Continuous outcome SCENARIO Grocery retailer Direct mail campaign to increase spend Weekly Spend measured in 12-week “pre”-period (AWS) Also in 6-week post period (AWSPostCampaign) Objective is difference: (PostMinusPreAWS) Random 250k chosen from 10m base for trial c. 75k actually targeted; c. 175k as control

11. Control Group Structure • Control group • Must be representative: technique will give misleading results otherwise • In practice, this means randomly select controls from target group • There must be enough of them All possible recipients Targets

12. Population Size • Population size • “Rule of 500”: to detect a x% difference (uplift), x% of the smaller population (usually controls) should ideally be at least 500 people • So if looking for 1% difference, control group needs to have at least 50,000 people • So consider longitudinal controls – contact half now, half later

13. Pruning and Validation • Pruning • Autopruning is implemented, based on qini variance • In practice, fairly unaggressive, so recommend manual pruning • Validation • Ordinary test-training fine if there is enough data • If not, consider k-way crossvalidation

14. Small Population Extensions • Bagging (oversampling method) and k-way cross-validation • Analysis candidate selection • useful if there are “too many” analysis candidates • Stronger pruning (variance-based) • Stratification • Not part of product, but potentially available as an extension if purchased

15. Return on Investment • Key thing is that Campaign ROI depends on the net effect (i.e. uplift) of action, not apparent response • (reduction in churn) × (value of saved people) – (cost of action) • (increase in purchase rate) × (value of purchase) – cost • (increase in spend) – (cost of action) etc. • Quadstone System has many suitable ROI FDL functions ( fx) built in (even without uplift license)

16. So how do youmeasure what’s important?

17. Quality Measure Considerations • Can only estimate uplift by segment • This is what we are used to with control groups • One person does not have a (knowable, measurable) uplift • Generalizing measures like classification error/accuracy or R2 doesn’t look promising • Rank statistics do seem more promising because they can sometimes be computed on a segmented basis

18. Negatively affected by action Positively affected by action Can we use/modify the Gini for Uplift? Overall uplift: x% x% Possibility of negative effects uplift 0% x% 100% % of customers targeted

19. Summary: When to use Uplift • Uplift modelling is just a better way of modelling the true effect of an action • Particularly relevant to: • Retention (where it’s the number/value of people you save that’s important • Up-sell, cross-sell, deep-sell (where it’s the incremental revenue or profit that’s important) • Risk management actions (where it’s the reduction in risk achieved that’s important)

20. Where to find out more • www.quadstone.com/system/uplift/ • For more in-depth training: our Uplift Analysis course. Contact support@quadstone.com

21. Questions and answers

22. After the webinar • These slides, the data and a four-week trial license are available via www.quadstone.com/training/webinars/ • Any problems or questions, contact support@quadstone.com

23. Building uplift models Ensure random control group exists Set partition field with P interpretation (1 for treated, 0 for control) Set objective (binary, continuous/discrete) Hit go Pruning Switch to test dataset Hit Autoprune Creating results field Use “Uplift as difference” Using difference viewers Crossdistribution Viewer places partition field on “ axis” automatically For view shown, drag count to depth, duplicate mean (ObjectiveField) and drag on to height Can configure which population is viewed by right-clicking on functions Using ROI Functions These are available under fx in Table Viewer when deriving new field. Uplift: Quick Reference

24. Upcoming webinars Thursday, 17th February 2005 Data Preparation in the Quadstone System Version 5 7.30am PST / 10.30am EST / 3.30pm GMT / 16.30 CET If there’s a webinar topic you’d like to see, please let us know via support@quadstone.com. www.quadstone.com/training/webinars/

25. Your feedback Suggestions or feedback? Please enter them in the feedback form or send them tosupport@quadstone.com

26. Modifying the Gini for Uplift? Unaffected by action Negatively affected by action x% Positively affected by action uplift 0% x% 100% % of customers targeted

27. Can’t do better than 100% sales: if 90% of control group purchases then maximum uplift = 10% 10% Can’t do worse than 0% sales: if 5% of control group purchases then maximum negative uplift = 5% The Shape of the Qini Curve ? Why is this flat? neutral –ve x% +ve uplift 0% x% 100% % of customers targeted