1 / 38

Robustness in the Salus scalable block store

Robustness in the Salus scalable block store. Yang Wang, Manos Kapritsos, Zuocheng Ren, Prince Mahajan, Jeevitha Kirubanandam, Lorenzo Alvisi, and Mike Dahlin University of Texas at Austin. Scalable and robust storage. More hardware. More failures. More complex software.

haruki
Download Presentation

Robustness in the Salus scalable block store

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Robustness in the Salus scalable block store Yang Wang, Manos Kapritsos, Zuocheng Ren, Prince Mahajan, Jeevitha Kirubanandam, Lorenzo Alvisi, and Mike Dahlin University of Texas at Austin

  2. Scalable and robust storage More hardware More failures More complex software

  3. Achieving both is hard Strong protections (End-to-end checks, BFT, Depot, …) BFT: read from f+1 nodes Consistency Challenge: vs Parallelism Read from 1 node Scalable systems (GFS/Bigtable, HDFS/HBase, WAS, Spanner, FDS, …..)

  4. Salus • Scalability: • Thousands of servers • Robustness: • Tolerate disk/memory corruptions, CPU errors, … • Do NOT hurt performance/scalability. • Usage: • Provide remote disks to users (Amazon EBS)

  5. Outline • Challenges • Salus’ overview • Solutions • Pipelined commit • Active storage • Scalable end-to-end checks • Evaluation

  6. Challenge: Parallelism vs Consistency Clients Infrequent metadata transfer Parallel data transfer Storage servers Metadata server State-of-the-art scalable architecture (GFS/Bigtable, HDFS/HBase, WAS, …) Data is replicated for durability and availability

  7. Challenges • Write in parallel and in order • Eliminate single points of failure • Write: prevent a single node from corrupting data • Read: read safely from one node • Do not increase replication cost

  8. Write in parallel and in order Clients Metadata server Write 1 Write 2 Data servers Write 2 is committed but write 1 is not. Not allowed for block store.

  9. Prevent a single node from corrupting data Computation nodes: • Data forwarding, garbage collection, etc • Tablet server (Bigtable), Region server (HBase), etc Clients Metadata server Data servers Single point of failure

  10. Read safely from one node Clients Metadata server Data servers Single point of failure

  11. Do not increase replication cost • Industrial systems: • Write to f+1 nodes and read from one node • BFT systems: • Write to 2f+1 nodes and read from f+1 nodes

  12. Outline • Challenges • Salus’ overview • Solutions • Pipelined commit • Active storage • Scalable end-to-end checks • Evaluation

  13. Salus’ approach Ensure robustness techniques do not hurt scalability Start from a scalable architecture (Bigtable/HBase)

  14. Salus’ interface and model • Disk-like interface: • A fixed number …. • Single writer • Barrier semantic • Failure model: • Byzantine but not malicious

  15. Salus’ key ideas • Pipelined commit • Guarantee ordering despite parallel writes • Active storage • Prevent a computation node from corrupting data • End-to-end verification • Read safely from one node

  16. Salus’ key ideas End-to-end verification Clients Pipelined commit Metadata server Active storage

  17. Outline • Challenges • Salus’ overview • Solutions • Pipelined commit • Active storage • Scalable end-to-end checks • Evaluation

  18. Pipelined commit • Goal: barrier semantic • A request can be marked as a barrier. • All previous ones must be executed before it. • Naïve solution: • The client blocks at a barrier: lose parallelism • A weaker version of distributed transaction • Well-known solution: two phase commit (2PC)

  19. Pipelined commit – 2PC Previous leader Servers Committed Batch i 1 3 Prepared 1 2 3 Leader 2 Client Batch i+1 Prepared 4 5 4 5 Leader

  20. Pipelined commit – 2PC Previous leader Servers Batch i-1 committed Batch i 1 3 Commit 1 2 3 Leader 2 Client Batch i committed Batch i+1 Commit 4 5 4 5 Leader

  21. Pipelined commit - challenge • Is 2PC slow? • Additional network messages? Disk is the bottleneck. • Additional disk write? Let’s eliminate that. • Challenge: whether to commit a write after recovery 1 3 2 is prepared. Should it be committed? Both cases are possible. 2 • Salus’ solution: ask other nodes

  22. Active Storage • Goal: a single node cannot corrupt data • Well-known solution: BFT replication • Problem: 2f+1 replication cost • Salus’ solution: use f+1 replicas • Require unanimous consent of the whole quorum • How about availability if one replica fails? • If one replica fails, replace the whole quorum

  23. Active Storage Computation node Storage nodes

  24. Active Storage Computation nodes • Unanimous consent: • All updates must be agreed by f+1 computation nodes. • Additional benefit: • Collocate computation and storage: save network bw Storage nodes

  25. Active Storage Computation nodes • What if one computation node fails? • Problem: we may not know which one is faulty. • Replace the whole quorum Storage nodes

  26. Active Storage Computation nodes • What if one computation node fails? • Problem: we may not know which one is faulty. • Replace the whole quorum • The new quorum must agree on the states. Storage nodes

  27. Active Storage • Does it provide BFT with f+1 replication? • No …. • During recovery, may accept stale states if: • The client fails; • At least one storage node provides stale states; • All other storage nodes are not available. • 2f+1 replicas can eliminate this case: • Is it worth adding f replicas to eliminate that?

  28. End-to-end verification • Goal: read safely from one node • The client should be able to verify the reply. • If corrupted, the client retries another node. • Well-known solution: Merkle tree • Problem: scalability • Salus’ solution: • Single writer • Distribute the tree among servers

  29. End-to-end verification Client maintains the top tree. Server 1 Server 3 Server 2 Server 4 Client does not need to store anything persistently. It can rebuild the top tree from the servers.

  30. Recovery • Pipelined commit • How to ensure write order after recovery? • Active storage: • How to agree on the current states? • End-to-end verification • How to rebuild Merkle tree?

  31. Discussion – why HBase? • It’s a popular architecture • Bigtable: Google • HBase: Facebook, Yahoo, … • Windows Azure Storage: Microsoft • It’s open source. • Why two layers? • Necessary if storage layer is append-only • Why append-only storage layer? • Better random write performance • Easy to scale

  32. Discussion – multiple writers?

  33. Lessons • Strong checking makes debugging easier.

  34. Outline • Challenges • Salus’ overview • Solutions • Pipelined commit • Active storage • Scalable end-to-end checks • Evaluation

  35. Evaluation

  36. Evaluation

  37. Evaluation

  38. Read safely from one node • Read is executed on one node: • Maximize parallelism • Minimize latency • If that node experiences corruptions, …

More Related