1 / 48

Understanding our Isis Example Program

Understanding our Isis Example Program. Ken Birman. Code for full program. using System; using System.Collections.Generic ; using System.Linq ; using System.Text ; using Isis; using System.Threading ; namespace ConsoleApplication3 { public class tuple { public int rank;

kenley
Download Presentation

Understanding our Isis Example Program

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Understanding our Isis Example Program Ken Birman

  2. Code for full program using System; usingSystem.Collections.Generic; usingSystem.Linq; usingSystem.Text; using Isis; usingSystem.Threading; namespace ConsoleApplication3 { publicclasstuple { publicint rank; publicint value; public tuple(int r, int v) { rank = r; value = v; } } classProgram { staticList<tuple> database = newList<tuple>(); publicconstint UPDATE = 0; publicconstint LOOKUP = 1; staticvoid Main(string[] args) { IsisSystem.Start(); Group g = newGroup("foo"); intmyRank = 0; bool go = false, dbfull = false; ; g.ViewHandlers += (ViewHandler)delegate(View v) { Console.WriteLine("New View: " + v); myRank = v.GetMyRank(); if (v.members.Length == 3) go = true; }; g.Handlers[UPDATE] += (Action<int,int>)delegate(int rank, int n) { database.Add(newtuple(n, rank)); Console.WriteLine("New tuple: " + rank + "/" + n); if (database.Count() == 15) dbfull = true; }; g.Handlers[LOOKUP] += (Action<int>)delegate(intarg) { Console.WriteLine("=== Query for arg=" + arg); List<int> answer = newList<int>(); int index = 0; foreach (tupletpin database) if (index++ % 3 == myRank) { Console.WriteLine("Looking at " + tp.rank + "/" + tp.value); if (tp.rank == arg) { Console.WriteLine("Including " + tp.rank + "/" + tp.value); answer.Add(tp.value); } } g.Reply(answer); }; g.Join(); while (!go) Thread.Sleep(10); for (int n = 0; n < 5; n++) g.OrderedSend(UPDATE, myRank, n); while (!dbfull) Thread.Sleep(10); if(myRank == 1) for (int n = 0; n < 3; n++) { List<List<int>> results = newList<List<int>>(); g.OrderedQuery(Group.ALL, LOOKUP, n, newIsis.EOLMarker(), results); Console.WriteLine("\r\nAnswers for Query rank=" + n); foreach (List<int> list in results) foreach (int value in list) Console.Write(value + " "); } IsisSystem.WaitForever(); } } }

  3. What the program does • Forms a process group • When there are 3 members, each of the 3 simultaneously sends updates to the group • They store these updates in “tuple” objects, as a list, called the “database” list. • Then 2 members wait (Isis.WaitForEver) while the 3rd member sends queries to the list • The queries are designed to select a subset of the data • We “subdivide” the work of searching the list

  4. What the program does • First member startsthe system • If several start atsame time, a Consensus within Isis2 decides which will be the first member • When it calls g.Join(),group is created • We see a first Viewreport staticvoid Main(string[] args) { IsisSystem.Start(); Group g = newGroup("foo"); . . . g.Join(); while (!go) Thread.Sleep(1); . . . }

  5. Consensus “in action”: IsisSystem.Start() • Step 1: Discovery • “Is anyone running Isis?” • Step 2: Collision • They hear each other! • Consensus: The process withthe smallest id is pickedby fault-tolerant algorithm • Step 3: A starts first. B and C are added next Copy A B C Step 1 {A, B, C} Step 2 {A, B, C} {A, B, C} Add B Add C B added C added

  6. Notice: A “wins the race” A is active before B,C • IsisSystem.Start() finishes first in A • B and C join “in background” but user code is already running in A Copy A B C Step 1 {A, B, C} Step 2 {A, B, C} {A, B, C} Add B Add C B added C added

  7. FLP impact? • It is impossible to guarantee progress for a correct consensus protocol that tolerates even one fault • Clearly Isis2 startup is solving consensus • Thus we can guarantee that startup will succeed • However, we can make it “very likely” that it will succeed, and this is enough

  8. What does A do now? Type signature g.ViewHandlers+= (ViewHandler)delegate(View v) { . . . Code for when a new membership view is defined . . . }; g.Handlers[UPDATE] += (Action<int,int>)delegate(int rank, int n) { . . . Code for doing an UPDATE action . . . }; g.Handlers[LOOKUP] += (Action<int>)delegate(intarg) { . . . Code for doing a LOOKUP action . . . }; g.Join(); Anonymous method

  9. “Delegate” concept • In C++ every method must have a name • In Java and C# you don’t need to name a method if you are happy to use “anonymous” naming • The compiler takes the “delegate” and creates a method with a name the compiler itself assigns • Then it compiles this method • So we are simply registering a method with Isis2

  10. What does A do now? g.ViewHandlers+= (ViewHandler)delegate(View v) { . . . Code for when a new membership view is defined . . . }; g.Handlers[UPDATE] += (Action<int,int>)delegate(int rank, int n) { . . . Code for doing an UPDATE action . . . }; g.Handlers[LOOKUP] += (Action<int>)delegate(intarg) { . . . Code for doing a LOOKUP action . . . }; g.Join();

  11. Variables in a delegate • Arguments are provided during the “upcall” • But a delegate can also access variables that were accessible in the method where it is placed • Thus our delegates can use variables created by the Main method. • We use this to access the “database” variable

  12. Access to the database variable Declared “outside” classProgram { staticList<tuple> database = newList<tuple>(); publicconstint UPDATE = 0; publicconstint LOOKUP = 1; staticvoid Main(string[] args) { . . . g.Handlers[UPDATE] += (Action<int,int>)delegate(int rank, int n) { database.Add(newtuple(n, rank)); Console.WriteLine("New tuple: " + rank + "/" + n); if (database.Count() == 15) dbfull = true; }; . . . } }

  13. Access to the database variable classProgram { staticList<tuple> database = newList<tuple>(); publicconstint UPDATE = 0; publicconstint LOOKUP = 1; staticvoid Main(string[] args) { . . . g.Handlers[UPDATE] += (Action<int,int>)delegate(int rank, int n) { database.Add(newtuple(n, rank)); Console.WriteLine("New tuple: " + rank + "/" + n); if (database.Count() == 15) dbfull = true; }; . . . } } Can be used “inside”

  14. But the methods don’t run yet! g.ViewHandlers+= (ViewHandler)delegate(View v) { . . . Code for when a new membership view is defined . . . }; g.Handlers[UPDATE] += (Action<int,int>)delegate(int rank, int n) { . . . Code for doing an UPDATE action . . . }; g.Handlers[LOOKUP] += (Action<int>)delegate(intarg) { . . . Code for doing a LOOKUP action . . . }; g.Join(); If group does not exist, creates it. A will run this first

  15. What the program does • So… the three members try to start Isis2, but A “wins” • As a result, A runs the g.Join() first • A “new view” event occurs with 1 member Isis: Searching for the Isis ORACLE... New View: View[gname=<foo>, gaddr=(0:224.0.69.120/0:0), viewid=0; 1 members={ [(5656)]}, hasFailed=[*-], nextIncomingMSGID={ <0:0> 0:0 }, StableTo={ ** 0 }, joining={ [(5656)]}, leaving={ []}, IamLeader = True g.ViewHandlers += (ViewHandler)delegate(View v) { Console.WriteLine("New View: " + v); myRank= v.GetMyRank(); if(v.members.Length == 3) go = true; };

  16. What is in the View object? • Array of members (Address[n]), in v.members • Who just joined (v.joined), left/failed (v.leaving) • View id number, increments for each new view • Group address: an Address object • Both groups and individual processes have Addresses • The “rank” of the member: 0..n-1 in the v.members v.GetMyRank(), v.GetRankOf(someone)

  17. What is in the View object? • Group “foo” has Address (0:224.0.69.120/0:0) • There is one member: (5656). It just joined. • If “long format” is used, we would also see the host IPv4 address: (5656:123.64.88.2:2345/1212) • The viewid is 0: the group was just created Isis: Searching for the Isis ORACLE... New View: View[gname=<foo>, gaddr=(0:224.0.69.120/0:0), viewid=0; 1 members={ [(5656)]}, hasFailed=[*-], nextIncomingMSGID={ <0:0> 0:0 }, StableTo={ ** 0 }, joining={ [(5656)]}, leaving={ []}, IamLeader = True

  18. Next B and C join the group • New views are reported • Viewid 0: 1 member: A • Viewid 1: 2 members: A, B • Viewid 2: 3 members: A, B, C • Notice that ALL group members see the same view events, starting when they join • “Old” members have smaller rank than “new” ones

  19. State transfer • Our sample program didn’t transfer any state to the joining member. • Homework task: Add state transfer • State: contents of the List<Address> database • In an Isis2 group, the state consists only of the state-machine replicated data you associate with the group • You do not need to include every variable! • In our sample program, the state was “empty” until after 3 members join, so state transfer was not needed

  20. What did the program do next? • Look closely at the View event handler • Each member • Notes its “rank”, in the variable myRank • myRank=0 in A, 1 in B, 2 in C • Sets boolean “go” to true when 3 members are in the View that was just reported g.ViewHandlers += (ViewHandler)delegate(View v) { Console.WriteLine("New View: " + v); myRank= v.GetMyRank(); if(v.members.Length == 3) go = true; };

  21. What did the program do next? • Now look at the Main method after g.Join • Every member loops, sleeping for 10ms at a time • So it checks go 100 times/second • When go becomes true, we pass the “barrier” • Homework: Replace “go” with a C# Semaphore: • Semaphore go = new Semaphore(0, int.MaxValue); • go.WaitOne() …. go.Release(1) while(!go) Thread.Sleep(10);

  22. What did the program do next? • Now look at the Main method after go==true • All three members concurrently begin to call g.OrderedSend! for (int n = 0; n < 5; n++) g.OrderedSend(UPDATE, myRank, n);

  23. Isis2 has many multicast options • g.SafeSend: Paxos, guarantees • Total ordering, all-or-nothing delivery, durability even if all members crash. Logged into a disk file • … this is stronger than we need! Our data is stored in memory (in the “database” List<tuple> variable) • g.OrderedSend: Optimistic early delivery • Total ordering, all-or-nothing among processes that do not crash. No logging, much faster than SafeSend • … this is what we are using

  24. Isis2 has many multicast options • g.Send: Optimistic, reliable, FIFO • First message sent is first message delivered • But ordered only if sent by the same sender process: A sends X, then A sends Y. X delivered before Y • g.CausalSend: Optimistic, causal, reliable • Causal ordering • First message sent is first delivered, even if different senders (e.g. process B receives X from A, then sends Y. X delivered before Y because B sent Y after receiving X) • g.RawSend: No guarantees, not even reliable

  25. With g.Send… • Our multicasts could arrive in different orders • All database copies would have identical content but perhaps the order in the list would differ • We saw this on Tuesday • With OrderedSend, all databases look the same • State machine replication in “practice” • Homework: In what ways does our application rely on ordering? Would the code break if we used g.Send and not g.OrderedSend? • Homework: Do we know what the ordering in database will be before the program ran, or are there multiple possible outcomes?

  26. Receiving the multicasts • The members get upcalls to the multicast event handler. In fact these are done on a different thread: the “multicast and view delivery thread” • dbfull becomes true when database has 15 items. • Homework: replace dbfull with a Semaphore g.Handlers[UPDATE] += (Action<int,int>)delegate(int rank, int n) { database.Add(newtuple(rank, n)); Console.WriteLine("New tuple: " + rank + "/" + n); if (database.Count() == 15) dbfull = true; };

  27. What is a “tuple”? • A simple object that stores a rank and a value publicclasstuple { publicint rank; publicint value; public tuple(int r, int v) { rank = r; value = v; } }

  28. Could we put tuples in messages? • Yes… but we would need to provide “Marshalling” code, and a null constructor, like this: • Then must call Msg.RegisterType(typeof(tuple), 0); • The 0 is a “unique type id” in range 0…128 [AutoMarshalled] publicclasstuple { publicint rank; publicint value; public tuple(int r, int v) { rank = r; value = v; } publictuple() { } }

  29. Can we put tuples in messages? • Homework • Modify the test program to send tuples • Store the received tuples into the database • Thought question: Does UPDATE call “new tuple”?

  30. What happens next? • Recall that tuple 15 sets dbfull to true • Look at Main again • … all three programs (A, B and C) pass the while g.OrderedSend(UPDATE, myRank, n); while (!dbfull) Thread.Sleep(10);

  31. What happens next? • Recall that tuple 15 sets dbfull to true • Look at Main again • … all three programs (A, B and C) pass the while g.OrderedSend(UPDATE, myRank, n); while (!dbfull) Thread.Sleep(10);

  32. What happens next? • Only member 1 (B) executes the last code: • B creates a “results” list, then issues an OrderedQuery asking for ALL responses • It does this three times: with n=0, n=1, n=2 • Each time it simply prints the list of responses if(myRank== 1) for (int n = 0; n < 3; n++) { List<List<int>> results = newList<List<int>>(); g.OrderedQuery(Group.ALL, LOOKUP, n, newIsis.EOLMarker(), results); Console.WriteLine("\r\nAnswers for Query rank=" + n); foreach(List<int> list in results) foreach(int value in list) Console.Write(value + " "); }

  33. What is a List<List<int>>? • In Java, C#, C++ we can create types that include other types are “arguments”. • These are called Generics • The results variable is a List: it will have one value in it for each member of the group • … and that value will be of type List<int> • Each group member sends a List in g.Reply

  34. Finally… the query handler • Look at the LOOKUP code: g.Handlers[LOOKUP] += (Action<int>)delegate(intarg) { Console.WriteLine("=== Query for arg=" + arg); List<int> answer = newList<int>(); int index = 0; foreach (tupletpin database) if (index++ % 3 == myRank) { Console.WriteLine("Looking at " + tp.rank + "/" + tp.value); if (tp.rank == arg) { Console.WriteLine("Including " + tp.rank + "/" + tp.value); answer.Add(tp.value); } } g.Reply(answer); };

  35. Finally… the query handler • … simplified version: g.Handlers[LOOKUP] += (Action<int>)delegate(intarg) { Console.WriteLine("=== Query for arg=" + arg); List<int> answer = new List<int>(); int index = 0; foreach (tupletpin database) if (index++ % 3 == myRank) { Console.WriteLine("Looking at " + tp.rank + "/" + tp.value); if (tp.rank == arg) { Console.WriteLine("Including " + tp.rank + "/" + tp.value); answer.Add(tp.value); } } g.Reply(answer); };

  36. Each member… • Has identical data in the database (g.OrderedSend). • … but has a different value of myRank: 0, 1 or 2

  37. Homework question • Exactly what can we say about the order we will see for tuples in database? • Why are the copies identical in A, B and C? • Did it matter that we only set go=true after A,B and C all joined? • Change the code to set go=true as soon as there are two members, A and B • Now use state transfer to fix the “bug” this causes • Will items sent by the same sender be in the sender order? (0,0)… (0,1)… (0,2)… (0,3)… (0,4)

  38. Finally… the query handler • Look at the LOOKUP code: g.Handlers[LOOKUP] += (Action<int>)delegate(intarg) { Console.WriteLine("=== Query for arg=" + arg); List<int> answer = newList<int>(); int index = 0; foreach (tupletpin database) if (index++ % 3 == myRank) { Console.WriteLine("Looking at " + tp.rank + "/" + tp.value); if (tp.rank == arg) { Console.WriteLine("Including " + tp.rank + "/" + tp.value); answer.Add(tp.value); } } g.Reply(answer); };

  39. Finally… the query handler • Look at the LOOKUP code: g.Handlers[LOOKUP] += (Action<int>)delegate(intarg) { Console.WriteLine("=== Query for arg=" + arg); List<int> answer = new List<int>(); int index = 0; foreach (tupletpin database) if (index++ % 3 == myRank) { Console.WriteLine("Looking at " + tp.rank + "/" + tp.value); if (tp.rank == arg) { Console.WriteLine("Including " + tp.rank + "/" + tp.value); answer.Add(tp.value); } } g.Reply(answer); };

  40. A, B and C each scan 5 tuples • Each one looks for rank==n, where n was from the Query sender (n=0… then n=1… then n=2) • How many items will each one find? • Thought question: how many items have rank=0? • … 5 have rank=0 • But which member will “see” these tuples?

  41. … we cannot know! • Perhaps “luck” will cause all of these to be scanned by A! But perhaps in a second run, A will see none • The situation depends on the data in database, and the order was not fully determined • Thus until we know the order, we cannot be sure which items A will look at, and until we know that, we cannot know how many will have rank=0

  42. … but we DO know that • Every tuple is scanned exactly once • And thus every rank=0 tuple will be seen by some member. • Example: perhaps A finds (0,1) and (0,3) • … and B finds none • … while C finds (0,0), (0,2) and (0,4) • Each sends its own List<int>: A sends { 1, 3 }, etc

  43. Back to the leader (process B) • Again, look at code in Main • Results is a “List of List<int>” if(myRank== 1) for (int n = 0; n < 3; n++) { List<List<int>> results = newList<List<int>>(); g.OrderedQuery(Group.ALL, LOOKUP, n, newIsis.EOLMarker(), results); Console.WriteLine("\r\nAnswersfor Query rank=" + n); foreach(List<int> list in results) foreach(int value in list) Console.Write(value + " "); }

  44. Back to the leader (process B) • Again, look at code in Main • Perhaps results will have {1,3},{},{0,2,4} • So output will be 1 3 0 2 4 if(myRank== 1) for (int n = 0; n < 3; n++) { List<List<int>> results = new List<List<int>>(); g.OrderedQuery(Group.ALL, LOOKUP, n, new Isis.EOLMarker(), results); Console.WriteLine("\r\nAnswers for rank=" + n); foreach(List<int> list in results) foreach(int value in list) Console.Write(value + " "); }

  45. What does “consistency” mean? • Every tuple is scanned by one group member • We know that there is a 0, 1, 2, 3, 4 value for each rank value 0, 1, 2 • So a consistent answer requires that we see 0..4 for each Query, but the program might not print in order • Homework: what we can say about the ordering within each List<int> reply element?

  46. Which result came first? • In the results list we do not know which part of the result came from A, which from B and which from C • Homework: • Add a second value to the reply giving the rank of the member that sent each list • Modify the output to show this information Answers for Query rank=0 [Member with myRank=0 sent {1,2}] [Member with myRank=2 sent {}] [Member with myRank=1 sent {0,3,4}]

  47. Using the View • Homework: Modify the program so that if the View contains N members when the Query is done, each does 1/N of the work • Currently code is “hard coded” to always use N=3 • What if a failure causes loss of a Reply? • Homework: Modify the replies to include value of N • Modify the Main program to sense missing reply and “retry” the Query: Missing a reply… must retry for rank=0 • Homework: Test your fault-tolerance logic (hint: consider using “IsisSystem.Terminate()” to “crash” a member)

  48. Code for full program using System; usingSystem.Collections.Generic; usingSystem.Linq; usingSystem.Text; using Isis; usingSystem.Threading; namespace ConsoleApplication3 { publicclasstuple { publicint rank; publicint value; public tuple(int r, int v) { rank = r; value = v; } } classProgram { staticList<tuple> database = newList<tuple>(); publicconstint UPDATE = 0; publicconstint LOOKUP = 1; staticvoid Main(string[] args) { IsisSystem.Start(); Group g = newGroup("foo"); intmyRank = 0; bool go = false, dbfull = false; ; g.ViewHandlers += (ViewHandler)delegate(View v) { Console.WriteLine("New View: " + v); myRank = v.GetMyRank(); if (v.members.Length == 3) go = true; }; g.Handlers[UPDATE] += (Action<int,int>)delegate(int rank, int n) { database.Add(newtuple(n, rank)); Console.WriteLine("New tuple: " + rank + "/" + n); if (database.Count() == 15) dbfull = true; }; g.Handlers[LOOKUP] += (Action<int>)delegate(intarg) { Console.WriteLine("=== Query for arg=" + arg); List<int> answer = newList<int>(); int index = 0; foreach (tupletpin database) if (index++ % 3 == myRank) { Console.WriteLine("Looking at " + tp.rank + "/" + tp.value); if (tp.rank == arg) { Console.WriteLine("Including " + tp.rank + "/" + tp.value); answer.Add(tp.value); } } g.Reply(answer); }; g.Join(); while (!go) Thread.Sleep(10); for (int n = 0; n < 5; n++) g.OrderedSend(UPDATE, myRank, n); while (!dbfull) Thread.Sleep(10); if(myRank == 1) for (int n = 0; n < 3; n++) { List<List<int>> results = newList<List<int>>(); g.OrderedQuery(Group.ALL, LOOKUP, n, newIsis.EOLMarker(), results); Console.WriteLine("\r\nAnswers for Query rank=" + n); foreach (List<int> list in results) foreach (int value in list) Console.Write(value + " "); } IsisSystem.WaitForever(); } } }

More Related