A Characterization of Big Data Benchmarks

1. A Characterization of Big Data Benchmarks Wen.Xiong Zhibin Yu, Zhendong Bei, Juanjuan Zhao, Fan Zhang, Yubin Zou, Xue Bai, Ye Li, Chengzhong Xu Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences

2. Agenda • Background • Motivation • Methodology • Evaluation • Conclusion • Future work

3. ETI Confidential Background • Requirements of a benchmark suite • Characteristics of different workload-input pairs • Spatio-temporal data in a real world system

4. ETI Confidential Background (1/3) • Requirements of a benchmark suite • a benchmark suite should contain workloads that represent a wide range of application domains. • workloads in a benchmark suite should be as diverse as possible. • a benchmark suite should not have redundant workloads in itself, keeping simulation or measure time as short as possible.

5. ETI Confidential Background (1/3) • simulation time between different numbers of workload-input pairs After removing redundancy, it can decrease 30% number of workload-input pairs and %40 simulation time.

6. ETI Confidential Background (2/3) • Characteristics of different workload-input pairs • Characteristics of workloads as the size of input data set changing • Stable • Unstable

7. ETI Confidential Background (3/3) • Spatio-temporal data in Shenzhen Transportation System • GPS trajectory data of taxicabs, 30000+ taxicabs, 90 millions GPS points per day. • Smart card data in metro transportation system, 15+ millions smart cards, 12+ millions transaction records per day.

8. ETI Confidential Background (3/3) • 2000 square kilometers, 18 millions of people. • road network in Shenzhen contains 73515 vertices and 101794 road segments.

9. ETI Confidential Motivation • Remove redundancy of a typical benchmark suite • Provide a benchmark suite for spatio-temporal data

10. ETI Confidential Motivation (1/2) • Remove redundancy of a typical benchmark suite • To decrease experiment time of benchmarking the objective system by minimizing the number of typical workload-input pairs.

11. ETI Confidential Motivation (2/2) • Provide a benchmark suite for spatio-temporal data • Representative workloads in our benchmark suite are as follows: • transaction count (hotregion) • spatiotemporal origin destination (sztod) • map matching • hotspot monitoring • spatiotemporal secondary sort

12. ETI Confidential Methodology • Typical MapReduce-based workloads • Micro architecture level metrics • Principal component analysis (PCA) • Hierarchical clustering and K-means clustering

13. ETI Confidential Methodology • Typical MapReduce-based workloads (1/2):

14. ETI Confidential Methodology • Typical MapReduce-based workloads (2/2):

15. ETI Confidential Methodology • Micro architecture level metrics are as follows: • Instruction per cycle (IPC) • L1 instruction cache miss ratio • L2 instruction cache miss ratio • Last level cache miss ratio • Branch prediction per instruction • Branch miss prediction per instruction • Off-chip bandwidth utilization

16. ETI Confidential Methodology • Principal Component Analysis: • It can reduce program characteristics while controlling the amount of information that is thrown away.

17. ETI Confidential Methodology • Hierarchical clustering • Hierarchical clustering is a "bottom up" approach: each observation starts in its own cluster, and workload-input pairs of clusters are merged as one moves up the hierarchy. It is useful in simultaneously looking at multiple clustering possibilities, and we can use a dendrogram for selecting desired number of clusters. • K-means clustering • K-means clustering aims to partition n workloads-input pairs into k clusters in which each workload-input pair belongs to the cluster with the nearest mean, where K is a value specified by user.

18. ETI Confidential Evaluation (instruction per cycle) The IPC of these sixteen workloads are range from 0.72 to 0.96, with an average value of 0.85. Wordcount has the lowest IPC value and hotregion has highest value among these workloads. ETI Confidential 03/10/2014 18

19. ETI Confidential Evaluation (L1 ICache miss ratio) The cache miss ratios of these typical workloads are range from 3.9% to 19.8%, with an average value of 8.9%. svm has the lowest L1 instruction cache miss ratio and hive-aggre has the highest L1 instruction cache miss ratio. ETI Confidential 03/10/2014 19

20. ETI Confidential Evaluation (L2 ICache miss ratio) The cache misses value of these workloads are range from 23.7% to 64.9%. On average, workloads from DCBench in right side have larger L2 instruction miss rate then workloads from HiBench in the left side. Overall, the L2 cache is ineffective in our experiment platform. ETI Confidential 03/10/2014 20

21. ETI Confidential Evaluation (branch prediction per instruction ) These values are range from 0.18 to 0.23, with an average value of 0.21. Hotregion has the lowest value of branch prediction per instruction while nutchindexing has the highest value of branch prediction per instruction. ETI Confidential 03/10/2014 21

22. ETI Confidential Evaluation (branch missprediction ratio ) These ratios are range from 1.5% to 5.6%, with an average value of 2.7%. Pagerank has the lowest branch miss prediction ratio while nutch indexing has the highest branch miss prediction ratio. The results show that the branch predictor of our processor matches these typical MapReduce based applications. ETI Confidential 03/10/2014 22

23. ETI Confidential Evaluation (off-chip bandwidth utilization) Among these workloads we evaluated, terasort is the only one that has the highest utilization ratio with a value of 14%. Overall, in our experiment platform, processors significantly over-provision off-chip bandwidth for these typical workloads. ETI Confidential 03/10/2014 23

24. ETI Confidential Evaluation (Hierarchical clustering )

25. ETI Confidential Evaluation (Hierarchical clustering ) • strong cluster, three workload-input pairs of same workload clustered together. • weak cluster, two workload-input pairs of same workload clustered together. • non cluster, no workload-input pairs of same workload clustered together.

26. ETI Confidential Evaluation(K-means clustering) • Seclecting 8 workload-input pairs via K-means clustering

27. ETI Confidential Evaluation(K-means clustering) sort-60G can be taken as the representative workload-input pair of its group including eight members.

28. ETI Confidential Conclusion • Redundancy exists in these pioneering benchmark suites • Such as sort and terasort. • The workload behavior of trajectory data analysis applications is dramatically affected by their input data sets.

29. ETI Confidential Future work • Conduct similarity analysis in workload-input pairs at a larger scale. • More metrics and larger input size • Fully implement a big data benchmark suite for spatio-temporal data • Data model, data generator and typical workload-input pairs.

30. Thank You !!!