The SMART Way to Migrate Replicated Stateful Services

1. The SMART Way to Migrate Replicated Stateful Services Jacob R. Lorch, Atul Adya, Bill Bolosky, Ronnie Chaiken, John Douceur, Jon Howell Microsoft Research First EuroSys Conference 19 April 2006

2. C A Replicated services Replicated stateful services stateful B Paxos • Problem: Machine failure leads to unavailability • Solution: Replicate the service for fault tolerance • Problem: Replica state can become inconsistent • Solution: Use replicated state machine approach The SMART Way to Migrate Replicated Stateful Services

3. Migrating replicated services A C E B D • Migration: Changing the configuration – the set of machines running replicas • Uses of migration • Replace failed machines for long-term fault tolerance • Load balancing • Increasing or decreasing number of replicas The SMART Way to Migrate Replicated Stateful Services

4. Limitations of current approaches Limitations of current approachesaddressed by SMART • Can remove non-failed machines • Enables autonomic migration, i.e., migration without human involvement • Enables load balancing • Can do concurrent request processing • Can perform arbitrary migrations, even ones replacing entire configuration • Completely described in our paper • Cannot remove non-failed machines without creating window of vulnerability • Can only remove known-failed machines • Cannot use migration for load balancing • Cannot process requests in parallel The SMART Way to Migrate Replicated Stateful Services

5. Outline • Introduction • Background on Paxos • Limitations of existing approaches • SMART: Service Migration And Replication Technique • Configuration-specific replicas • Shared execution modules • Implementation and evaluation • Conclusions The SMART Way to Migrate Replicated Stateful Services

6. Background on Paxos The SMART Way to Migrate Replicated Stateful Services

7. Background: Paxos overview Paxos protocol • Goal: Every service replica runs the same sequence of requests • Deterministic service ensures state changes and replies are consistent • Approach: Paxos assigns requests to virtual “slots” • No two replicas assign different requests to same slot • Each replica executes requests in slot order A B C … … … requests: 1 2 3 4 5 6 1 2 3 4 5 6 slots: 1 2 3 4 5 6 The SMART Way to Migrate Replicated Stateful Services

8. Background: Paxos protocol • One replica is the leader • Clients send requests to the leader • Leader proposes a request by sending PROPOSE message to all replicas • Each replica logs it and sends a LOGGED message to the leader • When leader receives LOGGED messages from a majority, it decides it and sends a DECIDED message Z B C A PROPOSE PROPOSE DECIDED DECIDED LOGGED LOGGED Req client server replicas The SMART Way to Migrate Replicated Stateful Services

9. Background: Paxos leader change • If leader fails, another replica “elects” itself • New leader must poll replicas and hear replies from a majority • Ensures it learns enough about previous leaders’ actions to avoid conflicting proposals B A C Reply Poll Poll The SMART Way to Migrate Replicated Stateful Services

10. D C Background: Paxos migration α • Service state includes current configuration • Request that changes that part of the state migrates the service • Configuration after request n responsible for requests n+αand beyond Service state A 79 80 81 82 83 84 85 B 79 80 81 82 83 84 85 79 80 81 82 83 A, B, D A, B, C 84 85 The SMART Way to Migrate Replicated Stateful Services

11. Rationale for α • With α=1, slot n can change the configuration responsible for slot n+1 • Leader can’t propose slot n+1 until n is decided • Doesn’t know who to make proposal to, let alone whether it can make proposal at all • Prevents pipelining of requests • Request may wait a network round trip and a disk write Z B C A Req PROPOSE PROPOSE LOGGED LOGGED Req The SMART Way to Migrate Replicated Stateful Services

12. Limitations of existing approaches The SMART Way to Migrate Replicated Stateful Services

13. No request pipelining • Leader change is complicated • How to ensure that new leader knows the right configuration to poll? • How to handle some outstanding proposals being from one configuration and some from another? • Other problems • To avoid this complexity, current approaches use α=1 • But, this prevents request pipelining The SMART Way to Migrate Replicated Stateful Services

14. Window of vulnerability • Removing a machine creates window of vulnerability • Effectively, it induces a failure of the removed replica • Consequently, service can become permanently unavailable even if less than half the machines fail • Considered acceptable since machines only removed when known to a human to have permanently failed • Not suitable for autonomic migration using imperfect failure detectors, or for load balancing B C D A Poll Poll DECIDED PROPOSE PROPOSE DECIDED LOGGED LOGGED The SMART Way to Migrate Replicated Stateful Services

15. SMART The SMART Way to Migrate Replicated Stateful Services

16. Configuration-specific replicas • Each configuration has its own set of replicas and its own separate instance of Paxos • Simplifies leader change so we can pipeline requests • Election always happens in a static configuration • No window of vulnerability because a replica can remain alive until next configuration is established Replica 1A Replica 1B Replica 1C Replica 2A Replica 2B Replica 2D A B C D The SMART Way to Migrate Replicated Stateful Services

17. SMART migration protocol FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED JOIN JOIN JOIN FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED FINISHED Replica 1A Replica 1B Replica 1C • After creating new configuration, send JOIN msgs • After executing request n+α-1, send FINISHED msgs • Tells new replicas where they can get starting state • Makes up for possibly lost JOIN messages • When a majority of successor configuration have their starting state, replica kills itself • If a machine misses this phase, it can still join later READY READY READY READY PREPARE READY READY JOIN Replica 2A Replica 2B Replica 2D A B C D JOIN-REQ The SMART Way to Migrate Replicated Stateful Services

18. Shared execution modules Agreement 1A Replica 1A Replica 1B Agreement 1B Replica 1C Agreement 1C • Configuration-specific replicas have a downside • One copy of service state for each replica • Need to copy state to new replicas • Solution:Shared execution modules • Divide replica into agreement and execution modules • One execution module for all replicas on machine Execution 1A Execution 1B Execution 1C Agreement 2A Replica 2A Agreement 2B Replica 2B Agreement 2D Replica 2D Execution 2A Execution 2B Execution 2D Execution Execution Execution Execution A B C D The SMART Way to Migrate Replicated Stateful Services

19. Implementation and evaluation • SMART implemented in a replicated state machine library, LibSMART • Lets you build a service as if it were single-machine, then turns it into a replicated, migratable service • Farsite distributed file system service ported to LibSMART • Straightforward because LibSMART uses BFT interface • Experimental results using simple key/value service • Pipelining reduces average client latency by 14% • Migration happens quickly, so clients only see a bit of extra latency, less than 30 ms The SMART Way to Migrate Replicated Stateful Services

20. Conclusions • Migration is useful for replicated services • Long-term fault tolerance, load balancing • Current approaches to migration have limitations • SMART removes these limitations by using configuration-specific replicas • Can remove live machines, enabling autonomic migration and load balancing • Can overlap processing of concurrent requests • SMART is practical • Implementation supports large, complex file system service The SMART Way to Migrate Replicated Stateful Services