1 / 26

Learning from Obama: Redesigning Analytics

Learning from Obama: Redesigning Analytics. The fundraising challenge. In 2008, Obama campaign raised $750 million Would not be enough in 2012. $750 million?. Not impressed. The fundraising challenge. But fundraising was proving more difficult in 2012 than in 2008

natala
Download Presentation

Learning from Obama: Redesigning Analytics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Learning from Obama: Redesigning Analytics

  2. The fundraising challenge • In 2008, Obama campaign raised $750 million • Would not be enough in 2012 $750 million? Not impressed.

  3. The fundraising challenge • But fundraising was proving more difficult in 2012 than in 2008 • President less available for fundraising events • In early campaign, we saw average online donation was half of what it had been in 2008 • We had to be smarter, and more innovative

  4. Overview • A/B testing in Obama’s digital department • Lessons learned • Don’t trust your gut • Foster a culture of testing • Make it personal

  5. Winning with A/B Testing

  6. Example: Draft language

  7. What impact can testing have? Test sends Full send (in millions) • $2.2 million additional revenue from sending best draft vs. worst, or $1.5 million additional from sending best vs. average

  8. Test every element • After testing drafts and subject lines, we would split the remaining list and run additional tests • Example: Unsubscribe language

  9. No, really. Test every element. • We also were always running tests in the background via personalized content

  10. Then, keep testing • Example: how much email should we send? +6 emails per week

  11. The results • Campaign raised over one billion dollars • Raised over half a billion dollars online • Over 4 million Americans donated • Recruited tens of thousands of volunteers, publicized thousands of events and rallies • Did I mention raising >$500 million online? • Conservatively, testing probably resulted in ~$200 million in additional revenue

  12. Lessons

  13. Lesson #1 Don’t Trust Your Gut

  14. Don’t trust your gut • We don’t have all the answers • Conventional wisdom is often wrong • Long-held best practices are often wrong • You are not your audience • There was this thing called the Email Derby… • If even the experts are bad at predicting a winning message, it shows just how important testing is.

  15. Experiments: Ugly vs. Pretty • We tried making our emails prettier • That failed • So we asked: what about ugly? • Ugly yellow highlighting got us better results

  16. Lesson #2 Foster a culture of testing

  17. The culture of testing • Check your ego at the door • Use every opportunity to test something • Compare against yourself, not against your competitors or “the industry” • Are you doing better this month than last month? • Are you doing better than you would have otherwise?

  18. When in doubt, test • In a culture of testing, all questions are answered empirically • Example: With the ugly yellow highlighting, we worried about the novelty factor • Maybe highlighting would only work for a short time before people started ignoring it (or being irritated by it). • We decided to do a multi-stage test across three consecutive emails

  19. The ugly highlighting experiment • Experimental design: • Determined through this test that novelty was indeed a factor Second Email Third Email First Email Group 1 Group 1 Group 1 Group 2 Group 2 Group 2 Group 3 Group 3 Group 3 Group 4 Group 4 Group 4 Group 5 Group 5 Group 5 Group 6 Group 6 Group 6 Group 7 Group 7 Group 7 Group 8 Group 8 Group 8

  20. Lesson #3 Use data to make the user experience more personal

  21. Big data ≠ big brother • Testing allows you to listen to your user base • Let them tell you what they like • Whether through A/B testing or behavioral segmentation, optimization gives them a better experience • Usually, the interactions that are the most human are the ones that win

  22. Be human! • In general, we founds shorter, less formal emails and subject lines did best. • Classic example: “Hey” • When we dropped a mild curse word into a subject line, it usually won • “Hell yes, I like Obamacare” • “Let’s win the damn election” • “Pretty damn cool”

  23. Good segmentation: behavioral • Behavioral segmentation was much more effective than demographic segmentation • Donor vs. non-donor • High-dollar vs. low-dollar • Volunteer status • What issues do people say they care about? • After using A/B tests to create a winning message, we could tweak it slightly for various behavioral groups and get better results

  24. Experiments: Personalization • Adding “drop-in sentences” that reference people’s past behavior can increase conversion rates • Example: asking recent donors for more money • Added sentence significantly raised donation rate • Confirmed in several similar experiments …it's going to take a lot more of us to match them. Will you donate $25 or more today? …it's going to take a lot more of us to match them.You stepped up recently to help out -- thank you. We all need to dig a little deeper if we're going to win, so I'm asking you to pitch in again. Will you donate $25 or more today?

  25. Conclusions

  26. Conclusions • Test everything, especially your gut instinct • Foster a culture of testing • Use data to make it personal

More Related