Evaluation of Prediction Models for Marketing Campaigns

1. Evaluation of Prediction Models for Marketing Campaigns Author: Saharon Rosset Advisor: Dr. Hsu Graduate: Lin Yan-Cheng

2. Abstract • Discuss model-evaluation criteria about their robustness • Ex. Response Rate in Customer retention

3. Agenda • Introduction • Model Evaluation • Planning Campaigns • Performance Measures • Prediction Model Performance • From Sample to Population • Confidence Intervals • Case Study • Conclusion • Opinion

4. Motivation • dealing with marketing applications the issue of evaluating prediction models is following twofold • Evaluation has to be statistically sound • Evaluate models should utilize from business perspective

5. Objective • To discuss some applicable model-evaluation and selection criteria

6. Model Evaluation • Evaluate the models’ performance on an independent test set • Adjust the models’ score to fit the full population distribution, in case it is expected to be different from the sample distribution used for training and test

7. Planning Campaigns • To measure by the amount of responders captured within the targeted population • The amount can be measured in two diff. way • Lift: How much better are we doing by using our model to select the target population relative to a random selection of the target population • RR: How frequently do we expect to encounter a responder when running our campaign?

8. Performance Measures • A, B : Total number of responders and non-responders, respectively • Aj, Bj: Total number of responders and non-responders, respectively, in the j-th top quantile. • j*(A+B)or(Aj+Bj): all cases in the j-th top quantile • A/(A+B): overall response rate

9. Measures at Pre-Specified Cutoff Points • Response Rate • RR(j) = Aj/(Aj+Bj) • Lift • Response Non-Response Ratio • RNR(j) = (Aj/A)/(Bj/B)

10. Comarison of Cut-Point Measures

11. Predicting Model Performance • Performance measures are usually calculated on a test sample data set • These measures need to be adjusted to the full population

12. From Sample to Population • A, B : the number of responders and non-responders in the FP (full population), respectively. • a, b : the number of responders and non-responders in the TS (Test Set), respectively. • ai, bi :the number of responders and non-responders in percentile i in the TS

13. Transformation • Extrapolate each percentile pair( ai, bi) in the TS to (Ai, Bi) in the FP • Ai = ai (A / a) • (Ai, Bi) does not add up to a FP percentile, TS percentiles are merged or split in order to attain FP percentiles

14. Confidence Intervals • Percentile point-estimators are not sufficient for evaluating the model predictive ability • confidence intervals for predict a model’s performance on future data

15. Case Study • Amdocs is a leading provider of CRM, Billing and Order Management solutions to the communications and IP industry worldwide • Consider a prediction model for a retention campaign, in which responders are potential churners and the overall response rate is the overall churn rate

16. Legacy model vs. New model • Initially legacy models RR at 10% was 2.75 times better than new model, but that was evaluated based on different test populations. Churn rate is 4.5 times in legacy models

17. RR vs. Lift vs. RNR

18. Conclusion • Discuss a few model-evaluation criteria about their robustness under changing population distributions • RR is a non-robust measure, Lift and RNR measures be commended to be used

19. Opinion • We need to consider the robustness of measure in our case before we conclude that.