1 / 17

The ABC of AB Testing: Evidence-based Decision Making

Learn why testing is crucial, what you can test, how to form hypotheses, interpret results, and mind your p-values. Make informed decisions based on evidence and improve your digital engagement strategies.

cneil
Download Presentation

The ABC of AB Testing: Evidence-based Decision Making

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The ABC of AB testing Jamie Woolley, digital engagement strategist 19 JUNE 2019 • COMMUNITY CONFERENCE 2019

  2. Why test at all? • Find out what works… • … and what doesn’t • Make decisions based on evidence • Avoid arguments • COMMUNITY CONFERENCE 2019

  3. What can you test? • Almost anything... • Subject lines • Email content • Site architecture • Site usability • Donation forms • Landing pages • Facebook ads • Google Adwords • Images • COMMUNITY CONFERENCE 2019

  4. Know your goals • Decide what you’re trying to improve • Open rates • Click rates • Form completion • Traffic • COMMUNITY CONFERENCE 2019

  5. How to test Form your hypothesis — what do you predict will happen? Create your variants Decide sample size and test duration Test! Interpret results • COMMUNITY CONFERENCE 2019

  6. Take baby steps • COMMUNITY CONFERENCE 2019

  7. Multivariate testing • COMMUNITY CONFERENCE 2019

  8. AB testing is iterative • MORE LIKE THESE RESULTS • YOU WON’T SEE THESE RESULTS… • COMMUNITY CONFERENCE 2019

  9. Email testing • COMMUNITY CONFERENCE 2019

  10. Website testing • COMMUNITY CONFERENCE 2019

  11. Statistics – run! COMMUNITY CONFERENCE 2019

  12. Statistical significance COMMUNITY CONFERENCE 2019

  13. Mind your p-values • Check for statistical significance, a measure of confidence in your results • The lower the p-value, the more confident you can be that your results are due to your test variables • The standard measure is 0.05 or lower, so you can be at least 95% confident your hypothesis is correct COMMUNITY CONFERENCE 2019

  14. No significant difference p-value is greater than 0.05 so results could be due to other factors - boo COMMUNITY CONFERENCE 2019

  15. Success p-value is less than 0.05 so you can accept your hypothesis - hooray! COMMUNITY CONFERENCE 2019

  16. Testing does a lot, but not everything • AB testing can’t tell you why — but you can make educated guesses • It’s not suitable for large redesigns as it won’t be clear what influenced the results • Testing won’t tell you if you’re addressing the right problem, so clear goals are important • Don’t let testing dictate creative design decisions COMMUNITY CONFERENCE 2019

  17. Thank You. Jamie Woolleyjme.woolley@gmail.comjamiewoolley.org • COMMUNITY CONFERENCE 2019

More Related