340 likes | 470 Views
Secure Distributed Storage: Recent Results and Open Problems. Gregory Chockler 1 , Rachid Guerraoui 2 , Idit Keidar 3 and Marko Vukoli ć 2. 1 IBM Research, Haifa 2 EPFL 3 Technion.
E N D
Secure Distributed Storage:Recent Results and Open Problems Gregory Chockler1, Rachid Guerraoui2, Idit Keidar3 and Marko Vukolić2 1 IBM Research, Haifa 2 EPFL 3 Technion Dagstuhl: From Security to Dependability September 12, 2006
Distributed Storage base objects / servers clients c2 c1 c3 DISTRIBUTED STORAGE G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 2
Secure Distributed Storage • Availability / Liveness • Wait-Freedom - despite asynchrony, crash and arbitrary failures of both clients and base objects • Weaker notions (e.g., obstruction-freedom) • Consistency / Safety • Atomicity, Regularity, Safety • Performance • Latency, Storage-requirements, Message size, ... • Best-case and Worst-case G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 3
In which setting? • Failure setting • Up to t base objects may fail • b may be Byzantine (arbitrary faulty), 0 ≤ b ≤ t • The rest may crash • Asynchony, Reliable point2point channels • Different model properties • Intercommunication among base objects possible? • Self-verifying data (e.g., digital signatures)? • Number of clients and base objects? G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 4
Outline • Simple distributed storage (b = 0) • Byzantine base object failures (b > 0) • Byzantine client failures • Performance • Summary and open problems G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 5
A simple storage [ABD95] • MWMR atomic wait-free storage • b = 0, any number of clients may crash • Assumes S ≥ 2t + 1 base objects • Not secure (i.e., not resilient to Byzantine failures) but crucial for understanding other implementations G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 6
ABD Read and Write writer WRITE(v) ts = highest rcvd ts + 1 wait for S-t replies Get_ts w=<ts,v> wait for S-t replies Single Writer (SW) reply with local <ts,v> bo1 bo2 reply with local ts bo3 Regular reader READ() Writeback <ts,v> READ_request Select <ts,v> with highest ts return(v) t=1; S=2t+1=3 G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 7
ABD and arbitrary failures • If we naively use ABD to handle Byzantine failures • Vulnerabilities in every round • 1st round of READ • Byzantine b.o. may return arbitrary value with the highest timestamp • 2nd round of READ • Byzantine reader may writeback any value (atomic case) • 1st round of WRITE • Skipping timestamps • 2nd round of WRITE • Poisonous WRITEs G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 8
Outline • Simple distributed storage (b = 0) • Byzantine base object failures (b > 0) • Byzantine client failures • Performance • Summary and open problems G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 9
Byzantine server failures • Optimal resilience [MAD02] • S≥ 2t + b + 1 • ABD is optimally resilient for b = 0 • Optimal resilience is one of the most desirable goals • [MR98] • Optimally resilient ABD-like implementation (b=t) • Tolerates reader Byzantine failures • Assumes self-veriying data (digital signatures) • Issues • Poisonous writes (malicious writers) • Self-verifying data G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 10
Self-verifying data • Powerful • Preclude Byzantine processes from forging values • Heavyweight • Requires setup, key distribution • Difficult in large systems • Recent solutions do not use self-verifying data • Main principle: need (at least) b+1 confirmations G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 11
SBQ-L algorithm [MAD02] • The first MWMR atomic optimally resilient storage • w/o self-verifying data • S≥2t+b+1 • Servers maintain a list of pending readers • Servers push concurrent updates to readers • Instead of reading back (Jay’s talk, see later) • Readers return highest ts value when this is confirmed by t+b+1 b.o. • Avoid write-back • But relies on servers to propagate data in case of client failures • Skipping timestamps G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 12
Non-skipping timestamps • [BD04] • Idea: choose b+1st highest timestamp • Drawback: not optimally resilient • ([BD04] requires at least 2t+2b+1 base objects) • [CT06] • Optimally resilient non-skipping timestamps • Using treshold cryptography • Not a lightweigth solution • Lightweight non-skipping timestamps? • with optimal resilience • deterministically G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 13
Outline • Simple distributed storage (b = 0) • Byzantine base object failures (b > 0) • Byzantine client failures • Performance • Summary and open Problems G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 14
Goals • Malicious writers • Prevent inconsistent (poisonous) writes • Not interested in preventing writing arbitrary values • Assume that authenticated writer may write any value • Malicious readers • Prevent all sorts of malicious behavior G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 15
Malicious writers • Poisonous writes • Broadcast [BT85] • Writers’ signatures + echoing among servers [MAD02] • Asynchronous verifyable information dispersal [CT06] • These techniques rely on intercommunication among base objects • i.e., on some variant of reliable broadcast • Cannot be applied in pure shared memory model • [GWGR04] • Hash of written data is stored at low bits of the timestamp (simplified) G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 16
Malicious readers • Intuitive idea: prevent readers from writing • Works in crash case, wait-free for regular semantics • [ACKM04] • SWMR, Wait-free, safe storage • SWMR, FW-terminating, regular storage • Both optimally resilient • Problem - regular wait-free storage? • Either we allow readers to write [ACKM 06], [GLV06], [BD06] • Or base objects store entire version history • Is this necessary? G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 17
Regular wait-free storage [CGK06] writer bo1 bo2 bo3 bo4 bo5 reader Base objects store limited number of ts/value pairs (e.g., last 2) Readers do not modify the state of base objects G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 18
Malicious readers: atomic case • How to prevent malicious readers in pure shared memory model (atomic case)? • No existing solution (w/o self-verifying data) G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 19
Outline • Simple distributed storage (b = 0) • Byzantine base object failures (b > 0) • Byzantine client failures • Performance • Summary and open problems G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 20
Latency • Frequently considered the most important perfromance metric • Ideally we would like all operations to be fast (1 round-trip) • Crash failures: ABD SWMR regular • all operations fast + optimal resilience • W/O self-verifying data – for minimal possible latency • 2 important optimization directions • Worst-case latency • Best-case latency • Optimize for synchronous periods, w/o concurrency, with few failures [ACKM04, GWGR04, GLV06, ...] • Avoid write backs in atomic case [GWGR04, GLV06] G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 21
Worst-case latency • Not all operations can be fast with optimal resilience (even in the SWSR safe case) • WRITE [ACKM04]; READ ([GV06]) • 2 rounds worst-case latency (pure shared memory model) • [GV06]: SWMR regular wait-free optimally resilient storage • Do not care about optimal resilience? • All READS/WRITES can be fast! (atomic SWMR wait-free) • Need many base objects, limit number of readers [DGLV05] G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 22
High-resolution timestamps [GV06] • HRts: information about readers’ logical time included in the timestamp • Readers write their local logical time to base objects • No impact on reilience to client malicious failures • [GV06] tolerates any number of Byzantine readers • Allow careful filtering of responses from b.o. • To quickly resolve ambiguities raised by malicious b.o. • Enables READ to complete in only 2 rounds • HRts allow combining optimal resilience and optimal latency • Deterministically, w/o self-verifying data • HRts are not necessary for safety! • [ACKM04] – SWMR safe storage (READ takes b+1 rounds) G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 23
High-Resolution Timestamps [GV06] writer WRITE(v) wait for S-t replies inc(ts) wait for S-t replies HRts = ts pw=<ts,v> WRITE completes! w=<HRts,v> reply with tsri bo1 tsr1[1..R] 1 2 1 bo2 1 2 tsr2[1..R] 0 bo3 1 2 0 tsr3[1..R] bo4 Send ACK 8 7 7 tsr4 pw w r1 r2 r3 t=b=1; S=2t+b+1=4; R=3 G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 24
Storage requirements • Presented solutions use full replication • Each base object stored entire replicated value • Erasure coding [GWGR04], [CT06] • Saves storage at base objects • Encode data in S chunks (when writing) • one distinct chunk stored per base object • any m chunks can reconstruct other S-m chunks • Used in conjunction with hash functions • To identify which chunks correspond to the same original data • Appealing to use m=b+1 • we need b+1 confirmations anyway • Orthogonal to many emulations that use full replication G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 25
Outline • Simple distributed storage (b = 0) • Byzantine base object failures (b > 0) • Byzantine client failures • Performance • Summary and open problems G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 26
Summary Response to attacks on ABD: • Resource exhaustion attacks (b.o.) • Non-skipping timestamps • Poisonous writes (writers) • Reliable broadcast + signatures • Hash included in the timestamp • Malicious responses from b.o. • Self-verifying data • b+1 distinct confirmations (or more) • Push values to readers • Inconsistent write backs (readers) • Self-verifying data • Avoid writebacks (when possible) • Allow readers to write metadata only G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 27
Summary (cont’d) • Best-case optimal latency • Optimize for synchronous periods, no contention, few failures • Worst-case optimal latency • High-Resolution timestamps (optimal resilience) • More base objects -> better latency • Storage requirements • Erasure coding + hashes • Bounded implementations (not discussed) • When do readers write? • Atomic (crash case – shared memory) • Regular (Byzantine - to achieve wait-freedom with constrained storage) • Safe (Byzantine – to combine optimal resilience and worst-case latency) G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 28
Open Problems • Related to optimal resilience • Worst-case latency of robust atomic READ? • Malicious readers in pure shared memory model? • Lightweight non-skipping timestamps (deterministic)? • Best-case optimality versus worst-case one? • Not orthogonal [GLV06], more research needed... • Other performance metrics? • Message complexity, message size, ... G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 29
References [ABD95] H. Attiya, A. Bar-Noy, and D. Dolev. Sharing memory robustly in message-passing systems. Journal of the ACM, 42(1):124–142, 1995. [ACKM04] Ittai Abraham, Gregory V. Chockler, Idit Keidar, and Dahlia Malkhi. Byzantine disk paxos: optimal resilience with Byzantine shared memory. Distributed Computing, 18(5):387–408, 2006. [ACKM05] Ittai Abraham, Gregory Chockler, Idit Keidar and Dahlia Malkhi. Wait-Free Regular Storage from Byzantine Components. In Information Processsing Letters,2006. [BD04] Rida Bazzi and Yin Ding. Non-skipping timestamps for Byzantine data storage systems. In Proceedings of the 18th International Symposium on Distributed Computing, volume 3274/2004 of Lecture Nodes in Computer Science, pages 405–419, Oct 2004. [BD06] Rida A. Bazzi, Yin Ding. Bounded Wait-Free f-resilient Atomic Byzantine Data Storage Systems for an Unbounded Number of Clients. To appear in Proceedings of the 20th International Conference on Distributed Computing, 2006. [BT85] Gabriel Bracha and Sam Toueg. Asynchronous consensus and broadcast protocols. Journal of the ACM, 32(4):824–840, October 1985. [CGK06] Gregory V. Chockler, Rachid Guerraoui, and Idit Keidar. Distributed computing with constrained memory. Technical Report 2006. [CT06] Christian Cachin and Stefano Tessaro. Optimal resilience for erasure-coded Byzantine distributed storage. In IEEE International Conference on Dependable Systems and Networks (DSN ’06), 2006. [DGLV05] P. Dutta, R. Guerraoui, R. R. Levy, and M. Vukolic. How Fast can a Distributed Atomic Read be? EPFL/LPD Technical Report LPD-REPORT-2005-001, Lausanne, Switzerland, 2005. Preliminary version appeared in Proceedings of the twenty-third annual ACM symposium on Principles of distributed computing (PODC’04), 2004. G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 30
References [GLV06] Rachid Guerraoui, Ron R. Levy, and Marko Vukolic. Lucky Read/Write Access to Robust Atomic Storage. In IEEE International Conference on Dependable Systems and Networks (DSN ’06), 2006. The full version of this paper is available as a EPFL/LPD technical report (LPD-REPORT-2005-005) with the same title. [GV06] Rachid Guerraoui and Marko Vukolic. How Fast Can a Very Robust Read Be? In 25th ACM Symposium on Principles of Distributed Computing (PODC’06), 2006. The full version of this paper is available as a EPFL/LPD technical report (LPD-REPORT-2006-008) with the same title. [GWGR04] G. Goodson and J. Wylie and G. Ganger and M. Reiter. Efficient Byzantine-Tolerant Erasure-Coded Storage. In IEEE International Conference on Dependable Systems and Networks (DSN ’04), pages 135–144, 2004. [MAD02] J.-P. Martin, L. Alvisi, and M. Dahlin. Minimal Byzantine storage. In Proceedings of the 16th International Conference on Distributed Computing, pages 311–325. Springer-Verlag, 2002. [MR98] D. Malkhi and M. Reiter. Byzantine quorum systems. Distrib. Comput., 11(4):203–213, 1998. G. Chockler, R. Guerraoui, I. Keidar, M. Vukolic. Secure Distributed Storage: Recent Results and Open Problems (Dagstuhl Talk)Slide 31