350 likes | 462 Views
A/B Testing – can you afford not to? #13NTCab. Hugh Dwyer, International Rescue Committee Jacob Colie, Stage Coach Digital Tom Williamson Jackson River. Today’s Agenda. Introductions What is A/B Testing? Testing Goals – what should we test? Testing versus Experimenting
E N D
A/B Testing – can you afford not to? #13NTCab Hugh Dwyer, International Rescue Committee Jacob Colie, Stage Coach Digital Tom Williamson Jackson River
Today’s Agenda • Introductions • What is A/B Testing? • Testing Goals – what should we test? • Testing versus Experimenting • Crafting an A/B Test Plan • Selling your plan • Build, Measure, Learn • Testing Tools • Resources & Downloads • QA A/B Testing
What is A/B Testing? A/B Testing: A/B testing is an experimental approach to web design which aims to identify changes to web pages that increase or maximize an outcome of interest (e.g., donation conversion). As the name implies, two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior. A/B Testing
Standard Donation Page A/B Testing
A/B Test Donation Page A/B Testing
What is A/B Testing? Multivariate Testing Multivariate testing is similar to A/B testing, but tests more than two different versions at the same time. A/B Testing
Multivariate A/B Testing
Challenges • What are some common challenges to doing A/B Testing? • Resources – technology, time and people • Data sampling size • Is my list big enough? • Are my results significant? • “Do no harm” • Lack of existing test data to build on • Risk vs. Reward • To do A/B testing you need to have an Entrepreneurial Spirit! A/B Testing
A/B Testing Channels • Channel Testing • Email • Advertising • Web • Offline/Direct Mail • Integrated (On and Offline) • Segmentation A/B Testing
Testing Goals – what should we test? Channels and Audience Segments Snail Mail Email Search Advertising New Audiences Leads New Donors Multi-Year Donors Website/Landing Environment A/B Testing
Testing Goals – what should we test? • Metrics and KPIs • Establish benchmark metrics: what are you trying to beat? • Average time on site/page • Open rates • Click-through rates • Form conversion (donations, email signups, etc.) • Average Gift • Cost to acquire • Cost per lead A/B Testing
Testing Goals – what should we test? Evaluating options • Alignment with strategy, context • Examine conversion funnels and donor lifecycle: where are your challenges? • Consider potential returns, & calculate ROI • Where can you move the needle the most? • Quick wins and low hanging fruit A/B Testing
Testing Goals – what should we test? Variables Price points Offer elements Modal windows Page layout Products Design elements Video Usability CTA Multi-step forms Default ask CTA Positioning Ask string Keywords Premiums Images Copy Benefits Subject lines Timing Frequency Communications strategies Testimonials Persuasion assets Content Achievements Expert ratings Security seals Cross-channel Media mentions A/B Testing
Testing Goals – what should we test? • Considerations • Evolutionary vs. revolutionary changes • Frequency and pace of testing • Short-term vs. long-term • Easy road vs. hard road • Embrace (calculated) risks A/B Testing
Testing vs. EXPERIMENTS A/B Testing
When to test? When to experiment? A/B Testing
GOALS Accelerate your REVENUE Provide VALUE for your users A/B Testing
The DIGITAL REVENUE CYCLE A/B Testing
BUILD. MEASURE. LEARN* * Read THE LEAN STARTUP A/B Testing
BUILD an MVP A/B Testing
MEASURE revenue and engagement A/B Testing
LEARN what did and didn’t work A/B Testing
Baseline IDEAL A/B Testing
vs. A/B Testing
short-term vs. LONG-TERM A/B Testing
5% vs. 35% A/B Testing
Examples A/B Testing
MERCY CORPS Retention A/B Testing
IRC Rescue Gifts A/B Testing
Getting started with A/B Testing • KISS – Keep it simple silly • Define metrics to test • Sustainer Acquisition • Average Gift • Gift String • Conversion %, etc. • Identify your top 3-5 items to test, and schedule them over Xmonths! • Decide how you will test each scenario • Control group • Minimum data required to validate • Resources necessary • Internal communications • Implement each test • Build, measure, learn – be patient A/B Testing
Individual Test Cycle A/B Testing
Selling your plan Developing a business case Recommendation Objectives Inputs Analysis Plan A/B Testing
Resources & Downloads • Presentation Materials • PowerPoint - www.jacksonriver.com/nten/ab_testing • Third Party Tools: • Optimizely - www.optimizely.com • Visual Website Optimizer - http://visualwebsiteoptimizer.com • Google Content Experiments • 8 Seconds - http://www.8seconds.net A/B Testing
Questions and Answers • Contact info • Hugh Dwyer • Sr. Director, Information & Operations, International Rescue Committee • hugh.dwyer@rescue.org • Jacob Colie • Senior Strategist & Principal, Stagecoach Digital • jacob@stagecoachdigital.com • @jacobcolie • Tom Williamson • CEO, Jackson River • tom.williamson@jacksonriver.com • @tomwilliamson, @jacksonriver A/B Testing
Evaluate This Session! Enter for a chance to win an NTEN engraved mini iPad! Or, search by session title at www.nten.org/ntc/eval