1 / 23

The Art of Testing

The Art of Testing. August 2012. Agenda. This is a Test. This is only a Test. Testing in Direct Mail Show Me the Numbers Oooops ! Pop… TEST…. Valid or Invalid – you be the Judge!. What Is a test?.

ila-pate
Download Presentation

The Art of Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Art of Testing August 2012

  2. Agenda • This is a Test. This is only a Test. • Testing in Direct Mail • Show Me the Numbers • Oooops! • Pop… TEST…. Valid or Invalid – you be the Judge!

  3. What Is a test? • A procedure intended to establish the quality, performance, or reliability of something, esp. before it is taken into widespread use. • A/B testing, split testing or bucket testing compares a control sample to other samples in order to discover how to improve response or conversion rates

  4. What Is NOT a Test?

  5. NO, REALLY - A TEST Is NOT… …..Comparing results of two different campaigns mailed during two different times of the year. A TEST Is NOT… … Comparing results of two mailings at two different organizations. A TEST Is NOT… … Mailing two completely different packages, with different offers, and different asks , with different inserts.

  6. Testing in Direct Mail

  7. Testing in Direct Mail Low Risk/ Lower Reward Higher Risk/Higher Reward

  8. Testing in Direct Mail: Knowing When & What to Test • When – • Quantity is RIGHT • Timing is RIGHT • There is Rollout potential • What – • Tests should be designed to meet both station goals and budget. • All tests should start with your hypothesis!

  9. Splitting the File • RANDOM- • Tips on pulling a random file: • Team Approach has a “Random”” field! • EQUAL: • The Test Bucket and the Control bucket need an equal # of records (which may mean a third bucket (“fall out”) if it is not a 50/50 split! • CODED: • Each segment needs a unique code for tracking purposes. • Don’t forget those pesky follow-ups!

  10. Set yourself up for Statistical Success… • Final Objective: Statistic validity • What makes a test statistically valid? • Significance • Sample Size • What does it mean to be statistically significant? • Results are repeatable, not by random chance

  11. Significance • Significance measures the likelihood of repeatable results. • Determined by mathematical formulas and expressed as a percentage (the confidence level) • 95% Confidence Level is best practice • Significance tests differ for Response Rates & Average Gifts • A winning test does not necessarily mean you have a winning package • Free significance calculators are available online

  12. Significance Calculators

  13. Sample Size • Why is sample size important? • Measuring significance • Minimizing risk & controlling costs • How do we determine sample size? • In general: There is no “right” quantity to test • The larger the sample size, the better • Rule of 100 • Statistical Calculations

  14. Sample Size: Rule of 100 “100 responses will make this test statistically significant.” • Some say that this rule-of-thumb has never proved them wrong. • Some believe this statement underestimates the actual sample size that is needed.

  15. Sample Size Calculator:Response Rate

  16. Sample Size: Statistical Calculations • Response Rate: N = 31.38 * p * (1-p)/(significant effect)² • Average Gift: N = 31.38 * σ² /(significant effect)² • Have no fear! • Many free calculators are available online

  17. Rule of 100 vs. CalculationsHead to head test… • Scenario 1 • Control RR: 2% • Expected Lift: 25% • Calculations show that… • Responses needed: 308 • Control/Test Quantity: 12,302 • Scenario 2 • Control RR: 2% • Expected Lift: 47% • Calculations show that… • Responses needed: 102 • Control/Test Quantity: 3,481 Results: In scenario 1, the Rule of 100 underestimates the quantity needed. In scenario 2, the Rule of 100 holds true.

  18. Quick Anatomy Takeaways The goal is to set yourself up for statistically valid results from the start • Make sure that you have an adequate sample size • To determine the right size you need to know the expected response rate, effect, and confidence level. • Run the test… • One variable at a time; multivariate testing increases sample size. • Results are In! Check for significance. • Use the calculators to see if your test was significant at the 95% confidence level.

  19. OOOPS!Mistakes that Muddle

  20. Mistakes that Muddle • Knowing when something “just isn’t right.” • Uncovering what that something is.

  21. YOU BE THE JUDGE!

  22. Questions? Thank You!!!

More Related