1 / 10

A/B Testing SaaS Products

A/B Testing SaaS Products. ProductCamp Provo March 29 th , 2014 Nate Carrier. A/B Testing: An Introduction. 50/50 randomized split between two experiences Used in Web development Internet marketing SaaS Products. Why A/B Testing?. Web development and internet marketing

upton
Download Presentation

A/B Testing SaaS Products

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A/B Testing SaaS Products ProductCamp Provo March 29th, 2014 Nate Carrier

  2. A/B Testing: An Introduction • 50/50 randomized split between two experiences • Used in • Web development • Internet marketing • SaaS Products

  3. Why A/B Testing? • Web development and internet marketing • What metrics do you try to improve? • Conversion • Sales • ROI (on ad spend) • Engagement • SaaS (cloud-based software) • Improve user experience (UX) • Increase engagement • Drive long-term profitability

  4. When Should You A/B Test? • Before introducing a new feature • Small day-to-day improvement • When you want to improve the customer exp. • When you want to increase sales/subscriptions • All the time!

  5. How to A/B Test • Define goal / question • Why are you running the test? • Identify metrics • How do you identify a successful test? • Design test experience • Set up data collection • Google analytics (very limited), Adobe Marketing Cloud, custom, etc. • Analyze data

  6. Let’s Analyze Data • Test and Control have different # of users! • Google Analytics • Email: ab@productcampprovo.org • Pw: ProductCamp Provo (with the space) • Shortcut: A/B Test on Voting • Excel Data • http://bit.ly/1mdQYJz • Limited data pushed into website database • User, Test Group, Post (at vote level)

  7. Some Ideas of What to Look For • Difference between Test & Control on: • Votes per Visitor • Visit duration • Sessions voted for (somewhat time consuming)

  8. What Insights Have You Found? • How do the test and control groups differ? • Number of votes per visitor • Test > Control • Visit duration • Test < Control • Different sessions voted for?

  9. Statistical Significance • Provides confidence in result • Insignificant: diff caused by random variation • Significant: most likely caused by something • Analytics and statistics only reveal correlation • Is that a bad thing? • Can require more data than we can get • Requires more skill to calculate • R

  10. Connect with Me @nate_carrier linkedin.com/in/natecarrier njcarrier@gmail.com

More Related