Building a Web Recommendation and Customization Platform

1. Building a Web Recommendation and Customization Platform September 2013

2. Big Fish Company Overview Big Fish is the world’s largest producer and distributor of casual games • Founded in 2002, Big Fish has distributed over 2 billion casual games and launchesA New Game Everyday!® for PC & Mac • 3,000+ games in the Big Fish catalog from 140+ exclusive game developer partners worldwide • Global business with games sold in 150+ countries • Over 100 million people have played games in the Mystery Case Files®franchise • Big Fish has published over 300+ unique mobile games on iOS, Android and other platforms • Offices located in Seattle, WA; Oakland, CA; and Luxembourg

3. Our Distribution Catalog Our catalog of 3,000+ casual games on many platforms are distributed through our multiple channels

4. Big Fish by the Numbers 2 billion+ Total PC/Mac games downloaded from Big Fish’s leading online marketplace 140+ Million Mobile games downloaded 100 million+ People who have played a Mystery Case Files® game (flagship brand) 10 million+ Unique visitors per month to Big Fish’s web portals 2 million+ Retail units sold annually (e.g. Wal-Mart, Target, Tesco and other large international retailers) 1 million+ Games distributed per day worldwide 3,000+ Unique games in the Big Fish catalog 600+ Game developer partners worldwide 300+ Unique games for smartphones and tablets 150+ Countries where customers purchase games by Big Fish 40+ Percentage of revenues outside North America 3 Offices (Seattle, WA; Oakland, CA; Luxembourg) 1 #1 PC/Mac casual game app store in the world

5. Agenda • Business Requirements • Project Requirements • Data Solutions Evaluated • Evaluation Results • Technical Details and Implementation • Outcome and Results

6. Game Recommendation System • Components: • Recommendation engine • Purchase and download tracking • Failsafe logic • Backfilling ‘light’ recommendations • Global recommendations • Service down fall-through Game Recommendation

7. Business Requirements • Increase sales • Increase Customer Lifetime Value • Consistent results between platforms • Targeted game marketing • Never show a product the user cannot buy Game Recommendation

8. Standard Homepage (Default)

9. Intelligent Recommendations

10. Big FishApplication Generic Recommendations (Not Logged In) Targeted Recommendations (Logged In) Game Recommendation

11. Project Requirements • Ability to meet traffic demands for 3 years • Real-time purchase and download tracking • Updates to the customers recommendations automatically happen • Tracking of all algorithms and clicks • Able to rebalance shards without Engineering Game Recommendation

12. Data Solutions Evaluated • Existing MySQL database • MySQL Master / Slave topology with 1.5BB row table • NoSQL • Redis • Hadoop • Riak • Membase • Membase + MySQL (hybrid) • Cassandra • Hbase Game Recommendation

13. Evaluation Results General questions and concerns each solution • Data consistency • Disaster Recovery (Backups) • Failover • Performance • Persistence • Risk to move to a new system • Expertise • Cost Game Recommendation

15. Evaluation Test - Metrics Lookup Directory Inserts and Lookups per Second Single Database Performance VS Sharded Database Performance Game Recommendation

16. Sharding by the Numbers • New design utilized composite PK based on customer, game and language • Lookups were all primary key; very performant • Write load is light; do not suffer from mid-table inserts Current Numbers: • Database: 75 queries per second with 912 µs response time • Memcache: 600 requests per second with 185 µs response time • Database is over 100GB with 2.14BB+ records • Grows at a rate of 750,000+ records per day Game Recommendation

17. Sharding Implementation • Lookup service database has 2 tables: • One for shard definitions • One for mapping the customer to the shard • Leverage 2 stored procedures for data integrity and acid compliance: • One locates which shard receives next customer • One gets customer shard information or creates it Game Recommendation

18. Outcome and Results • From inception to code release ~ 6 months • Launched all backend services in Q4 2011 • Performed 6 iterations updating modules and pages on our site to utilize the system • Deployed recommendations to Big Fish Application Game Recommendation

19. Content Customization System • Content: • Ability to change assets on the website per customer • Customization: • Leveraging customer data and behavioral patterns to create a personalized experience Content Customization

20. Business Requirements • Increase revenue through customized experience • Admin tool for internal marketing teams • Ability to support 100% delta change per day • Integrate with all Big Fish products and services Content Customization

21. Existing customer homepage

22. New Customer Homepage

23. No Campaign Triggered

24. Campaign Triggered

25. Project Requirements • Long term scalability • Support up to 250MM+ records (tested 750MM) • Performant system for reads and writes • Enable the business while reducing Engineering and Operations maintenance costs Content Customization

26. Data Solutions Evaluated • Traditional replication deployment • Sharded replication deployment • Solves / provides workarounds for most problems • MySQL Cluster (NDB) • Ability to quickly load data and retain high read rates • Auto-sharding • In-memory solution; removed memcache layer • Reduced Engineering QA efforts • Enterprise support (Oracle technology, MCM toolset) Content Customization

27. Evaluation Results • Ability to load ~180,000 records per second using 4 MySQL threads, batch size of 10,000 records • Insert 100MM in less than 10 minutes • Tuned Cluster to be write intensive resulting in 150,000 inserts/sec and 3,000 reads/sec • ~27,000 reads/sec Content Customization

28. System Architecture http://www.mysql.com/why-mysql/case-studies/mysql-cs-bigfish.html

29. NDB Connectors Reads Writes Content Customization

30. Outcome and Results • From inception to code release ~ 4 months • Deployed Q2 2013 • MySQL Cluster supports 10-12M requests per day (1-1 mapping for page views) • Phased launch: • Support all International sites • Support US site Content Customization

31. Overall Recommendation Platform Results • Enabled the business to increase revenue • Nominal maintenance (< 1%) • Five 9’s availability

32. Questions?

33. Thank You!