Column-Oriented Storage Techniques for MapReduce

1. Column-Oriented Storage Techniques for MapReduce Avrilia Floratou (University of Wisconsin – Madison) Jignesh M. Patel (University of Wisconsin – Madison) Eugene J. Shekita (While at IBM Almaden Research Center) Sandeep Tata (IBM Almaden Research Center) Presented by: Luyang Zhang && Yuguan Li

2. Motivation Databases MapReduce Column – Oriented Storage Performance Programmability Fault tolerance

3. Column-Oriented Storage • Benefits: • Column-Oriented organizations are more efficient when an aggregate needs to be computed over many rows but only for a notably smaller subset of all columns of data. • Column-Oriented organizations are more efficient when new values of a column are supplied for all rows at once. • Column data is of uniform type, which provides some opportunity for storage size optimization. (e.g. Compression)

4. Questions • How can columnar-storage operate efficiently on top of a DFS (HDFS)? • Is it easy to apply well-studied techniques from the database field to the Map-Reduce framework given that: • It processes one tuple at a time. • It does not use a restricted set of operators. • It is used to process complex data types. How to incorporate columnar–storage into an existing MR system (Hadoop) without changing its core parts?

5. Challenges In Hadoop, it is often convenient to use complex types like arrays, maps, and nested records to model data. --- which leads to a high deserialization cost and lack of effective column-oriented compression techniques. Serialization: data structure in memory  bytes that can be transmitted Deserialization: bytes  data structure in memory (Since hadoop is written in Java, more complex than C++ )

6. Challenges Compression: Although the column data seems to be more similar and share a high compression ratio, the complex type makes some existed technologies cannot be applied to Hadoop. Programming API: Some technologies are not feasible for hand-coded mapreduce function.

7. Outline Column-Oriented Storage Lazy Tuple Construction Compression Experimental Evaluation Conclusions

8. Column-Oriented Storage in Hadoop • Main Idea: Store each column of the dataset in a separate file • Problems: • How can we generate roughly equal sized splits so that a job can be effectively parallelized over the cluster? • How do we make sure that the corresponding values from different columns in the dataset are co-located on the same node running the map task?

9. Column-Oriented Storage in Hadoop Horizontally Partitioning Into split-directories /data/2013-03-26/s1 /data/2013-03-26/ 1st node /data/2013-03-26/s2 2nd node Introduce new InputFormat/OutputFormat : ColumnInputFormat (CIF) ColumnOutputFormat(COF)

10. ColumnInputFormat V.S RCFile Format • RCFile Format: • Avoid Replication and Co-location problem • Using Pax instead of a true column-oriented format, all columns will be packed in a single row-group as a split. • Efficient I/O elimination become difficult. • Metadata need additional space overhead. • CIF: • Need to tackle Replication and Co-location • Efficient I/O elimination • Consider adding a column to a dataset.

11. Replication and Co-location Node A Node B Node C Node D HDFS Replication Policy CPP Introduce a new column placement policy (CPP) Can be assigned to “dfs.block.replicator.classname”

12. Example Map Method if (age < 35) return name Record Age Name ColumnInputFormat.setColumns(job,”Age”,”Name”) Joe 23 23 Joe 32 32 David David John 45 What if age > 35? Can we avoid reading and deserializing the name field? Mary 30 50 Ann

13. Outline Column-Oriented Storage Lazy Tuple Construction Compression Experiments Conclusions

14. Lazy Tuple Construction Mapper ( NullWritable key, LazyRecordvalue) { String name; int age = value.get(“age”); if (age < 35) name = value.get(“name”); } Mapper ( NullWritable key, Recordvalue) { String name; intage = value.get(“age”); if (age < 35) name = value.get(“name”); } Deserialization of each record field is deferred to the point where it is actually accessed, i.e. when the get() methods are called. *:Deserialize only those columns that are actually accessed in map function

15. LazyRecordimplements Record name age lastPos skip lastPos =curPos curPos Why do we need these: “ Without lastPos pointer, each nextRecord call would require all the columns to be deserialized to extract the length information to update their respective curPos pointer.”

16. Skip List (Logical Behavior) Skip 100 Records R1 R1 R100 R100 Skip 10 R1 R1 R10 R10 R20 R20 ... R90 R90 R100 R100 R1 R1 R2 R2 R10 R10 R20 R20 ... R90 R90 ... R99 R99 ...

17. Example Name if (age < 35) return name Skip10 = 1002 Skip100 = 9017 Joe Age 0 David 10 rows 23 1 … 39 100 rows 2 Jane Skip Bytes 45 … Skip 10 = 868 … … 10 rows 102 30 Mary Ann John

18. Example if (age < 35) return hobbies Info Skip10 = 2013 Skip100 = 19400 Age “hobbies”: tennis “friends” : Ann, Nick 23 10 rows “friends” : George 39 … 45 100 rows Null … … Skip 10 = 1246 … 10 rows 30 “hobbies”: tennis, golf …

19. Outline Column-Oriented Storage Lazy Record Construction Compression Experiments Conclusions

20. Compression Dictionary Compressed Skip Lists Compressed Blocks Dictionary “hobbies” : 0 “friends” : 1 # Records in B1 Skip10 = 210 Skip100 = 1709 LZO/ZLIB compressed block RID : 0 - 9 0 : {tennis} 1 : {Ann, Nick} Skip Bytes B1 10 rows 1: {George} … # Records in B2 100 rows Null LZO/ZLIB compressed block RID : 10 - 35 … B2 Decompress Skip 10 = 304 … 10 rows 0: {tennis, golf}

21. Outline Column-Oriented Storage Lazy Record Construction Compression Experiments Conclusions

22. RCFile Metadata Joe, David Row Group 1 23, 32 {“hobbies”: {tennis} “friends”: {Ann, Nick}}, {“friends”:{George}} Metadata John, Smith Row Group 2 45, 65 {“hobbies”: {tennis, golf}}, {“hobbies”: {swimming} “friends”: {Helen}}

23. Experimental Setup • 42 node cluster • Each node: • 2 quad-core 2.4GHz sockets • 32 GB main memory • four 500GB HDD • Network : 1Gbit ethernet switch

24. Overhead of Columnar Storage Single node experiment Synthetic Dataset 57GB 13 columns 6 Integers, 6 Strings, 1 Map Query Select *

25. Benefits of Column-Oriented Storage Single node experiment Query Projection of different columns

26. Workload Schema Query If( urlcontains “ibm.com/jp” ) find all the distinct encodings reported by the page URLInfo { String url String srcUrl time fetchTime String inlink[] Map <String,String[]> metadata Map <String, String> annotations byte[] content } Dataset : 6.4 TB Query Selectivity : 6%

27. Comparison of Column-Layouts (Map phase) SEQ: 754 sec

28. Comparison of Column-Layouts (Map phase) 3040

29. Comparison of Column – Layouts (Total job) SEQ: 806 sec

30. Conclusions Describe a new column-oriented binary storage format in MapReduce. Introduce skip list layout. Describe the implementation of lazy record construction. Show that lightweight dictionary compression for complex columns can be beneficial.