1 / 28

BASIC Regenerating Codes for Distributed Storage System s

BASIC Regenerating Codes for Distributed Storage System s. Kenneth Shum (Joint work with Minghua Chen, Hanxu Hou and Hui Li ). Window Azure data centers. Inside a data center. http://technoblimp.com. Data distribution. Encode and distribute a data file to n storage nodes.

kineks
Download Presentation

BASIC Regenerating Codes for Distributed Storage System s

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. BASIC Regenerating Codes for Distributed Storage Systems Kenneth Shum (Joint work with Minghua Chen, Hanxu Hou and Hui Li)

  2. Window Azure data centers kshum

  3. Inside a data center http://technoblimp.com kshum

  4. Data distribution • Encode and distribute a data file to n storage nodes. Data File: “INC” kshum

  5. Data collector • Data collector can retrieve the whole file by downloading from any k storage nodes. “INC”  kshum

  6. Three kinds of disk failures • Transient error due to noise corruption • repeat the disk access request • Disk sector error • partial failure • detected and masked by the operating system • Catastrophic error • total failure due to disk controller for instance • the whole disk is regarded as erased Aug 2013 kshum 6

  7. Frequency of node failures Figure from “XORing elephants: novel erasure codes for Big Data” by Sathiamoorthy et al. Number of failed nodes over a single month in a 3000 node production cluster of Facebook. Aug 2013 7

  8. Outline of this talk • Repetition scheme • Traditional erasure-correcting codes • Reed-Solomon codes • Network-coding-based scheme • BASIC regenerating codes Aug 2013 kshum 8

  9. Distributed storage system • Encode a data file and distribute it to ndisks • (n,k) recovery property • The data file can be rebuilt from any kdisks. • Repair • If a node fails, we regenerate a new node by connecting and downloading data from any d surviving disks. • Aim at minimizing the repair bandwidth(Dimakis et al 2007). • A coding scheme with the above properties is called a regenerating code. kshum

  10. Repetition scheme • GFS: Replicate data 3 times • Gmail: Replicate data 21 times kshum

  11. 2x Repetition scheme Divide the datafile into 2 parts 1G A A, B 1G Data Collector B 1G A 1G Cannot toleratedouble disk failures B

  12. 1G Repair is easy for repetition-based system New node A A B A Repair bandwidth =1G B

  13. Reed-Solomon Code Divide the file into 2 parts A A, B Data Collector B A+B It can toleratedouble disk failures A+2B Aug 2013 13

  14. Repair requires essentially decoding the whole file A A New node 1G B 1G A+B Repair bandwidth = 2G A+2B Aug 2013 kshum 14

  15.    BASIC regeneration code Binary AdditionShiftImplementableConvolutional Divide the datafile into 4 parts 0.5G 0.5G 0.5G 0.5G Utilization of bit-wise shift in storage was proposed byPiret and Krol (1983), andQureshi, Foh and Cai (2012).

  16. Download from nodes 1 and 2     1G Data Collector 0.5G 1G 0.5G 0.5G 0.5G Aug 2013 16

  17. Download from nodes 1 and 3     1G Data Collector 0.5G 0.5G 0.5G 1G 0.5G Aug 2013 17

  18. Download from nodes 1 and 4     1G Data Collector 0.5G 0.5G 0.5G 0.5G 1G Aug 2013 18

  19. Download from nodes 2 and 3     1G Data Collector 0.5G 0.5G 0.5G 1G 0.5G Aug 2013 19

  20. Download from nodes 2 and 4     1G Data Collector 0.5G 0.5G 0.5G 0.5G 1G Aug 2013 20

  21. Download from nodes 3 and 4     Data Collector 0.5G 1G 0.5G 0.5G 0.5G 1G Aug 2013 21

  22. Zigzag decoding à laGollakata and Katabi (2008) What to solvefor P1and P2. P1  P2 P1P2 P1  P2’ P1P2’ Aug 2013 kshum 22

  23. Repair of BASIC regenerating code New node XOR Repair bandwidth=1.5 G  Bitwise shift and XOR   Bitwise shift and XOR 

  24. Interference alignment Repair of BASIC regenerating code  Decode the blueand red packets byzigzag decoding   

  25. Comparison of the three examples Aug 2013 kshum 25

  26. Summary • We can reduce repair bandwidth by network coding. • BASIC regenerating codes • A failed storage node can be repaired by simple bit-wise shift and XOR operations. • Small storage overhead due to shifting. Aug 2013 kshum 26

  27. References • Piret and Krol, MDS convolution codes, IEEE Trans. of Information Theory, 1983. • Dimakis, Brighten, Wainwright and Ramchandran, Network coding for distributed storage systems, INFOCOM, 2007. • Gollakata and Katabi, Zigzag decoding: combating hidden terminals in wireless networks, Proc. in the ACM Sigcomm, 2008. • Qureshi, Foh, and Cai, Optimal solution for the index coding problem using network coding over GF(2), Proc. IEEE Conf. on Sensor Mesh and Ad Hoc Comm. and Network, 2012. • Sung and Gong, A zigzag decodable code with MDS property for distributed storage systems, Proc. IEEE Symp. on Information Theory, 2013. • Hou, Shum, Chen and Li, BASIC regenerating code: binary addition and shift for exact repair, Proc. IEEE Symp. on Information Theory, 2013. kshum

  28. Two modes of repair • Exact repair • The content of the new node is exactly the same as the content of the failed node • Functional repair • only requires that the (n,k) recovery property is preserved. kshum

More Related