Business Intelligence and Decision Modeling

1. Business Intelligence and Decision Modeling Week 12 Campaign Management: Testing & Performance

2. Outline • Testing and Control Groups • (Chap. 14) • Campaign Performance Measurement • (Chap. 12) • Analytics and Modeling Revisited • (Chap. 13)

3. About Testing • Interactive marketing Vs traditional advertising • Testing Vs market research • Testing Vs management resistance

4. Testing for Effectiveness • Copy (communications) • Execution (creative) • Package bundles and offers • Clickstreams (email and websites) • Channels

5. Testing Methodology • Sample size • Control group or benchmark • Test groups set aside for time length • Half-life …

6. Testing Methodology • Half-life Response 100% 50% 33% 25% Time

7. Testing Methodology:CI for a given sample • P +/- CI or • P +/- Z x SEpor • P +/- Z √(pq/n)

8. Testing Methodology:CI for a given sample • CI = +/- Z x √(pq/n) • If p = .04, n = 10,200 and Z = 1.96 • CI = 1.96 √(.04 x .96/ 10,200) • CI = .0038 • Thus UCL = .0438; LCL = .0362 Note: CL confidence level (Z), CI confidence interval (+/-), UCL and LCL Upper and Lower confidence limits

9. Testing Methodology:Sample size for a given CL • If CI = Z x √(pq/n) • Then CI = Z x √(pq/ √ n • √n = Z x √(pq) / CI • If p = .04, Z = 1.96 and CI = .0038 • .0362 < P < .0438 • √n = 1.96 √(.04 x .96)/ .0038 • √n = 101 or n = 10,215 • Two-tail test! Note: CL confidence level (Z), CI confidence interval (+/-), UCL and LCL Upper and Lower confidence limits

10. Two-tail Test for CL Z = 1.96 for 95% CL .025 .025

11. One-tail Test for Sample Z = 1.64 for 95% CL .05

12. Testing Methodology:Sample size for a given CL • If CI = Z x √(pq/n) • Then CI = Z x √(pq/ √ n • √n = Z x √(pq) / CI • If p = .04, Z = 1.64 and CL = .0038 • P > .0438 • √n = 1.64 √(.04 x .96)/ .0038 • √n = 84.57 or n = 7,152 • One-tail test! Note: CL confidence level (Z), CI confidence interval (+/-), UCL and LCL Upper and Lower confidence limits

13. Test against Control • One single item variable: 2 Groups • SPSS DM Control Package Test • SPSS Independent Sample T-test • Also available on Excel • One single item variable: 3+ groups • SPSS Compare means • Oneway ANOVA

14. Testing Interactive Marketing Treatments • Two or more variables: experimental design • Two-way ANOVA (Excel and SPSS)

15. Testing Treatments (2)

16. Testing Treatments (3)

17. Campaign Performance(Traditional and online)

18. Interactive Marketing Campaign Effectiveness • CPM, RPM, OPM (Cost, Revenue and Orders per M or thousand) • CPO (Cost per order) • CPR (Cost per Response)

19. Cost Measures • CPM = (Cost / Volume) * 1000 (19 000$ / 50 000) * 1000 = 380$ • CPM media ((Media + Insertion)/Circulation) * 1000 ((40 000$ + (10 * 1 000))/ 1 000 000) * 1000 = 60$ (Insertions are charged in CPM)

20. Response Measures • Response in % = (Response / Quant.) * 100 • RPM = (Response / Quant.)/1000 • OPM = (Orders/Quant./1000)

21. Email and Web Analytics

22. e-commerce/emailsCampaign Effectiveness • Delivered • Click to open (CTO) • Unique click / Click through rate (CTR) • Cost per impression (CPI) • Paid per click (PPC) • Cost per action (CPA) • Cost per lead (CPL)

23. Clickstream Analysis Search Preference Edit profile Welcome Main message Discount Offer Download Offer Second subject/ offer Links/ Directory Viral information

24. Clickstream Analysis

25. Clickstream Analysis

26. Email Performance

28. Top line from Chapter 14-1Testing • Everyone agrees that testing is essential, but few people do it. • Use half-life in direct mail testing. • Consider both the short-term and long-term effect of everything that you do. • The formula for a control group size is Control group size = 500÷anticipated response rate. • In e-mail testing, the results come back in 24 hours. You can test the subject line, calls to action, the offer, the price, and the day of the week. You can also personalize or the arrangement of the home page. • Use your best previous promotion as your control. • Define what was best about the control e-mail: Was it opens? Clicks? The lower rate of unsubscribes or the high conversions? Be sure you have a concrete definition of best. • The best test is the single-variable test. Test only one thing with each e-mail promotion.

29. Top line from Chapter 14-2Testing • Start with a quick subject line test. Create a random 10 percent of your file, dividing it into six equal segments. Try one subject line on each segment. Mail the test. Tomorrow you will know which was best. Use that. • Before any test, write down a statement of what you’re trying to prove or disprove by the test. Separate significant results from the noise. • If you are doing a lot of tests (and you should), many tests will fail to prove anything significant. • Set aside time to study every test’s results. One of e-mail marketers’ great sins is the failure to study the results of previous tests. • Mistakes you can make in testing: • Too many changes at once. • Looking only at conversions. Look at opens and what customers click on. • Testing by committee rather than by the customers. • Keep a log of your mailing statistics. Tests have to be compared with the results of other tests to know whether you are doing better or worse. • Test by the demographics or behavior of your audience.

30. Top line from Chapter 12-1Campaign Management • Database marketing is conducted by campaigns: groups of messages with a common theme such as “Pre-Christmas Mailing.” • Each campaign stands on its own and can be analyzed by its open, click, and conversion rates. • There are many different types of customer and subscriber communications, including promotions, transactions, triggers, reminders, thank yous, surveys, welcomes, and reactivations. • Successful direct mail response rates are around 2.6 percent. E-mail conversion rates average about 0.11 percent. • Direct mail names can be rented legally and ethically. E-mail subscribers have to agree to receive your messages. • The average e-mail marketer sends about 50 campaigns a month. The median is 28. • Typical e-mail subscribers get about 10 e-mails per month from each company they have given their name to.

31. Top line from Chapter 12-2Campaign Management • The average open rate for e-mails is about 11 percent. That means that almost 90 percent of e-mails never get read by anybody. • For those e-mails that get opened, subscribers click on about 24 percent of them. A click means that the subscriber is actually reading the message. • Conversions (sales) average about 4.5 percent per click. They average about 0.11 percent per delivered e-mail. • The TV–e-mail paradox: we can’t accurately measure what a TV ad does, so we spend a lot of money on TV. We can accurately measure what e-mail does, so we spend very little. • E-mails usually produce more off-e-mail sales than sales within the e-mail. From this we derive the off-e-mail multiplier. • Online sales usually have a higher profit margin than retail store sales. • Monthly campaign reports are essential to successful e-mail marketing.

32. Top line from Chapter 13 -1 • Analytics helps you to predict which recipients of your direct mail will buy your products, and which are not likely to buy. At $500 per thousand pieces, analytics can save you a lot of money. • Analytics is not as useful for e-mail marketing. The cost of appending data and the modeling often results in a loss, since the cost of mailing is only $6 per thousand. • Predictive models are based on previous promotions. You add demographic data (age, income, value of home, etc.) to a sample of your file and determine the differences between responders and non-responders. • Predictive modeling uses multiple regressions. It results in an algorithm—a mathematical formula that can be used to “score” any direct mailing file that has demographics appended, and predict, before you mail, which ones are going to respond. • Modeling does not always work. Sometimes what makes people buy is not based on demographics.

33. Top line from Chapter 13 -3 • CHAID is very useful for dividing your database into segments containing people with different interests and response rates. • Descriptive analytics is useful for advertising campaigns, but seldom useful for direct mail. • Clickstream data analysis can be very useful in planning the layout of a Web site or an e-mail. • Key performance indicators (KPIs) can help you determine the relative success of e-mail programs.