Exact Regenerating Codes on Hierarchical Codes

1. Exact Regenerating Codes on Hierarchical Codes Ernst Biersack Eurecom France Joint work and Zhen Huang

2. Outline • :: Introduction and motivation • :: Hierarchical Codes • :: Regenerating Codes • :: Combining Hierarchical Codes and Regenerating Codes • :: Conclusion

3. Motivation: Elements of a P2P backup system • Performance metrics: • Storage efficiency: how much redundant information do you store? From Julian Monteiro

4. Motivation: Network Bandwidth is a scarce resource Our first objective is to find erasure codes that consume less communication bandwidth, i.e. have better efficiency factor ρ - Network communication bandwidth cannot be “put aside” for later use A second objective should be to adopt repair policies that provide a smooth utilization of the communication bandwidth

5. Hierarchical Codes Regenerating Codes ER-Hierarchical Codes

6. - A particular way to build erasure codes is linear codes Linear Codes: Overview original fragments c1,1 [c1,1 c1,2 c1,3 c1,4] o1 Linear combination p1 c1,2 [c2,1 c2,2 c2,3 c2,4] o2 p2 c1,3 p1 [c3,1 c3,2 c3,3 c3,4] p3 o3 c1,4 parity fragment [c4,1 c4,2 c4,3 c4,4] p4 o4 p5 P = CO O=C-1P p6 If C is invertible, i.e. the coefficient vectors are linearly independent, we can reconstruct the original fragments. If coefficients are chosen randomly in GF(216), the matrix is invertible with a very high probability.

7. let us try to change the way the code is built: p5 o1 o2 o3 p1 p2 p3 p5 p6 p4 p7 o1 o2 o3 p1 p2 p3 p4 p6 p7 o4 o4 Hierarchical codes: Idea There are sets of 4 parity fragments that are not sufficient to reconstruct the original file. traditional erasure code Hierarchical code

8. The repair degree determines the efficiency factor ρ Hierarchical codes : Repair degree Failure ρ=4 ρ=2

9. HC-(k,h) • k original blocks • h redundant blocks Hierarchical codes: Recursive Construction

10. Hierarchical codes: Theory

11. Hierarchical codes: Repair • What if p_1 and p_3 are lost? • Use p_2 , 1 out of {p_7, p_8} and 1 out {p_4, p_5, p_6} • need 3 blocks • What if p_1, p_2, and p_3 are lost? • Use ….. • need ???? blocks In HC, the earlier we repair the repair is often “cheaper”

12. Two possible instances of a 64+64 hierarchical code 64+64 hierarchical codes: Reliability vs Cost - Lower repair cost comes at the prices of reduced reliability

13. Hierarchical Codes Regenerating Codes ER-Hierarchical Codes

14. What happens if… p’4 p1 p2 p5 p7 upon a repair we contact more than k peers? Every peer stores a parity block larger (or equal) than the usual parity fragment (i.e. 1/k of the file size)? Regenerating Codes: Idea o1 b1 d>k o2 o3 o4 |block|≥|file|/k p8 Regenerating codes (by G. Dimakis) give the answer: the repair communication requirements are much smaller.

15. - regenerating codes are controlled by two additional parameters beyond k and h :: d the repair degree :: ithe block expansion index k ≤ d ≤ k+h-1 0 ≤ i ≤ k-1 Regenerating codes: Performance - if we consider a regenerating code with k=32 and h=32: classical erasure codes MBR: Minimum-Bandwidth Regenerating MSR: Minimum-Storage Regenerating

16. - k=32 and h=32 and a stored file of 1MB: Additional space Regenerating codes: Performance Communication is impressively reduced with small amount of extra storage.

17. Communication Regenerating codes: A new dimension in the trade-off • RC(k,h,d,i,) • k original pieces • h additional pieces • d repair degree • i block expansion factor Erasure Codes Storage Replication Regenerating codes can be seen as a generalization of replication and RSE that allow to more flexibly trade off communication and storage requirements.

18. Regenerating codes: Want to know more See http://csi.usc.edu/~dimakis/StorageWiki/doku.php A wiki on Coding for Distributed Storage maintained by Alexandros G. Dimakis

19. Hierarchical Codes Regenerating Codes ER-Hierarchical Codes

20. Can we combine Hierarchical codes and Regenerating Codes? • Yes: • ER-Hierarchical Codes combine concepts of Hierarchical Codes and Regenerating Codes, namely that • most parity blocks are linear combinations of only a small subset of all original blocks and that • a storage block consists of α fragments, while a repair block has only β fragments, with , β < α ER-Hierarchical Codes

21. How to transform Hierarchical code into ER-Hierarchical Code? ER-Hierarchical Codes: Construction

22. ER-Hierarchical Codes: Construction

23. In HC we would need to download 4 blocks of size 1 each •  4 units of traffic • In ER-HC we now download 5 fragments of size ½ each •  2.5 units of traffic ER-Hierarchical Codes: Repair

24. ER-Hierarchical Codes: Traffic reduction (analysis) • ER-HC reduces the traffic by more than • 85% as compared to RSE and Regenerating Codes • 40% compared to Hierarchical codes Reg Code is MSR with d=k+1

25. ER-Hierarchical Codes: Repair Strategies

26. ER-Hierarchical Codes: Performance (simulation) In HC and ER-HC , the earlier we repair the “cheaper” the repair; is not the case for RG and RSE

27. Have presented some new codes that • greatly reduce the communications overhead • Regenerating codes apply principles of network coding to distributed storage and allow to trade off storage space for communications bandwidth • As compared to RSE codes • Regenerating codes increase the repair degree (number of nodes that must be contacted for repair) but significantly reduce the amount of data downloaded from each node • Hierarchical codes significantly reduce the repair degree while keeping the amount of data transferred by each node the same (as RSE) • Combining Regenerating Codes and Hierarchical Codes makes us win at both fronts • Reduces repair degree and the amount of data transmitted by each node Conclusion

28. Future work • Further exploit the possibilities offered by ER-Hierarchical Codes • Study the relationship between coding and repair policies for systems with churn • Reactive repair results in repair burst • Proactive repair has smoother repair traffic but does unnecessary repairs. If repairs are cheap, as they are for ER-HC, proactive repair becomes much more attractive since the “earlier we repair”, the cheaper a repair

29. Thanks Questions?