eHarmony in Cloud

1. eHarmony in Cloud Subtitle Brian Ko

2. eHarmony • Online subscription-based matchmaking service • Available in United States, Canada, Australia and United Kingdom. • On average, 236 members in US marry every day. • More than 20 million registered users. 1

3. Why Cloud? • Problem exceeds the limits of the data center and data warehouse environment. • Leverage EC2 and Hadoop to scale data 2

4. Finding match • Model Creation 3

5. Find matching • Matching 4

6. Find Matching • Predicative Model Scores 5

7. Requirement • All the matches, scores, and user information should be archived daily • Ready for 10X growth • Possible O(n2) problem • Need to support set of models becoming more complex 6

8. Challenge • Current architecture is multi-tiered with a relational back-end • Scoring is DB join intensive • Data need constant archiving • Matches, match scores, user attributes at time of match creation • Model validation is done at a later time across many days • Need a non-DB solution 7

9. Solution • Open Source Java implementation of Google’s MapReduce framework • Distributes work across vast amounts of data • Hadoop Distributed File System (HDFS) provides reliability through replication • Automatic re-execution on failure/distribution • Scale horizontally on commodity hardware 8

10. Slide 9 • Simple Storage Service (S3) provides cheap unlimited storage. • Elastic Cloud Computing (EC2) enables horizontal scaling by adding servers on demand. 9

11. MapReduce • A large server farm can use MapReduce to process huge dataset. • Map step • Master node takes the input • Chops it up into smaller sub-problems • Distributes those to worker nodes. • Reduce step • Master node takes the answers to all the sub-problems • Combines them in a way to get the output  10

12. Why Hadoop • Mapper and Reducer are written by you • Hadoop provides • Parallelization • Shuffle and sort 11

13. Actual Process • Upload to S3 and start EC2 Cluster 13

14. Actual Process • Process and archive 14

15. Amazon Elastic MapReduce • It is a web service • EC2 cluster is managed for you behind the scenes • Starts Hadoop implementation of the MapReduce framework on Amazon EC2 • Each step can read and write data directly from and to S3 • Based on Hadoop 0.18.3 15

16. Elastic MapReduce • No need to explicitly allocate, start and shutdown EC2 instances • Individual jobs were managed by a remote script running on master node (no longer required) • Jobs are arranged into a job flow, created with a single command • Status of a job flow and all its steps are accessible by a REST service 16

17. Before Elastic Map Reduce • Allocate/Verify cluster • Push application to cluster • Run a control script on the master • Kick off each job step on the master • Create and detect a job completion token • Shut the cluster down 17

18. After Elastic MapReduce • With Elastic MapReduce we can do all this with a single local command • Uses jar and conf files stored on S3 • Various monitoring tools for EC2 and S3 are provided 18

19. Development Environment • Cheap to set up on Amazon • Quick setup - Number of servers is controlled by a config variable • Identical to production • Separate development account recommended 19

20. Cost comparison • Average EC2 and S3 Cost • Each run is 2 to 3 hours • $1200/month for EC2 • $100/month for S3 • Projected in-house cost • $5000/month for a local cluster of 50 nodes running 24/7 • A new company needs to add data center and operation personnel expense 20

21. Summary • Dev tools really easy to work with and just work right out of the box • Standard Hadoop AMI worked great • Easy to write unit tests for MapReduce • Hadoop community support is great. • EC2/S3/EMR are cost effective

22. The End 5 minutes of question time starts now!

23. Questions 4 minutes left!

24. Questions 3 minutes left!

25. Questions 2 minutes left!

26. Questions 1 minute left!

27. Questions 30 seconds left!

28. Questions TIME IS UP!