Spark - Shark Data Analytics Stack on a Hadoop Cluster

1. Spark - Shark Data Analytics Stack on a Hadoop Cluster April 22, 2013

2. Big Data Week Data Science Group April 23, 2013

3. Michael Malak Data Analytics Senior Engineer at Time Warner Cable Technicaltidbit.com

4. Chris Deptula Senior Big Data Consultant 317.840.2935 chris.deptula@openbi.com @chrisdeptula http://www.openbi.com

5. Michael Walker Managing Partner 720.373.2200 m@rosebt.com http://www.rosebt.com

6. Agenda The Big Data Problem Spark Ecosystem NFL Data Science Use Case Visualizing Data

7. The Big Data Problem

8. Speed Kills in Data Science

10. Hype Cycle for Emerging Tech 2012

11. Hype Cycle for Big Data 2012

12. Evolution of DW Architecture

13. Emerging DW Architecture

14. Next-Generation Data Architecture

15. Big Data Ecosystem Parts

16. DW Database Systems MQ 2013

17. Total Enterprise Data Growth 2005-2015

18. Structured vs Unstructured Data

19. Modern DW/BI Analytical Ecosystems

20. Big Data Ecosystem Parts

21. The Internet of Things

22. Big Data 4 V's

23. New World of Databases

24. New World of Databases

25. Hadoop

26. Hadoop

27. Hadoop

28. Big Data Vendor Focused on Hadoop and NoSQL Revenue 2012

29. Big Data Analytics Infrastructure

30. The Spark Ecosystem

31. Agenda • What Hadoop gives us • What everyone is complaining about in 2013 • Spark • Berkeley Team • BDAS (Berkeley Data Analytics Stack) • RDDs (Resilient Distributed Datasets) • Shark • Spark Streaming • Other Spark subsystems technicaltidbit.com

32. What Hadoop Gives Us HDFS Map/Reduce technicaltidbit.com

33. Hadoop: HDFS Image from mark.chmarny.com technicaltidbit.com

34. Hadoop: Map/Reduce Image from blog.octo.com Image from people.apache.org/~rdonkin technicaltidbit.com

35. Map/Reduce Tools Pig Script HiveQL Hbase App Hive Pig Hadoop Linux technicaltidbit.com

36. Hadoop Distribution Dogs in the Race Hadoop Distribution Query Tool Apache Drill Stinger technicaltidbit.com

37. Other Open Source Solutions Druid Spark technicaltidbit.com

38. Not just caching, but streaming 1st generation: HDFS 2nd generation: Caching & “Push” Map/Reduce 3rd generation: Streaming technicaltidbit.com

39. Berkeley Team Image from Ian Stoica’s slides from Strata 2013 presentation 40 students 8 faculty 3 staff software engineers Silicon Valley style skunkworks office space 2 years into 6 year program technicaltidbit.com

40. BDAS(Berkeley Data Analytics Stack) Bagel App Shark App Spark Streaming App Bagel Shark Spark Streaming Spark App Spark Hadoop/HDFS Mesos Linux technicaltidbit.com

41. RDDs(Resilient Distributed Dataset) Image from MateiZaharia’s paper technicaltidbit.com

42. RDDs: Laziness x => x.startsWith(“ERROR”) lines = spark.textFile(“hdfs://...”) errors = lines.filter(_.startsWith(“ERROR”)) .map(_.split(‘\t’)(2)) .filter(_.contains(“foo”)) cnt = errors.count All Lazy Action! technicaltidbit.com

43. RDDs: Transformations vs. Actions Transformations map(func) filter(func) flatMap(func) sample(withReplacement, frac, seed) union(otherDataset) groupByKey[K,V](func) reduceByKey[K,V](func) join[K,V,W](otherDataset) cogroup[K,V,W1,W2](other1, other2) cartesian[U](otherDataset) sortByKey[K,V] Actions reduce(func) collect() count() take(n) first() saveAsTextFile(path) saveAsSequenceFile(path) foreach(func) [K,V] in Scala same as <K,V> templates in C++, Java technicaltidbit.com

44. Hive vs. Shark Shark HiveQL HiveQL HDFS files + HDFS files RDDs technicaltidbit.com

45. Shark: Copy from HDFS to RDD CREATE TABLE wiki_small_in_mem TBLPROPERTIES ("shark.cache" = "true") AS SELECT * FROM wiki; CREATE TABLE wiki_cached AS SELECT * FROM wiki; Creates a table that is stored in a cluster’s memory using RDD.cache(). technicaltidbit.com

46. Shark: Just a Shim Shark Images from ReynoldXin’s presentation technicaltidbit.com

47. What about “Big Data”? PB TB Shark Effectiveness GB MB KB technicaltidbit.com

48. Median Hadoop job input size Image from Reynold Xin’s presentation technicaltidbit.com

49. Spark Streaming: Motivation HDFS x1,000,000 clients technicaltidbit.com

50. Spark Streaming: DStream DStream RDD {{“id”: “hercman”}, “eventType”: “buyGoods”}} {{“id”: “hercman”}, “eventType”: “buyGoods”}} {{“id”: “shewolf”}, “eventType”: “error”}} 2 sec RDD {{“id”: “shewolf”}, “eventType”: “error”}} 2 sec . . . RDD {{“id”: “catlover”}, “eventType”: “buyGoods”}} {{“id”: “hercman”}, “eventType”: “logOff”}} 2 sec “A series of small batches” technicaltidbit.com