Isis 2 Process Groups

Isis2 Process Groups Cornell University Ken Birman

Isis2 Process Groups • The most important concept in the Isis2 system • A group • Has a name (like “myGroup”) • Has members (while active) • Interacts with those members by delivering events to them through upcalls to pre-registered methods • The “state” (the data associated with the group) lives within the application programs you build, not in Isis2

Key Concept:Join or Create When a program starts, it isn’t a member of any groups. Calling IsisSystem.Start() activates Isis2. Then the program would normally join or create groups

Group g = new Group(“myGroup”); • Creates an Isis2 object that will be your interface to the group. • The new Group() does not join or create the group. • The new Group object isn’t active yet. That gives you time to attach handlers to it before you join or create it • As soon as the join or create occurs, events might start to be received and reported to the application

Handlers • These are the important types of event handlers • ViewHander: Used to sense membership changes • Message Handlers: Used to process incoming point to point (unicast) messages and incoming multicasts • Checkpoint creation: Used to make a checkpoint of group state, which can be saved into a log file (for persistency) or sent to a joining member (to initialize it) • After registering handlers, you call g.Join() or g.Create() to bind this instance to others that may be active

Key Concept:View The View object reports the status of the group. It contains v.members[]: An array of member Addresses v.joiners[], v.leavers[]: Members that just joined or left v.viewid: An integer counter, incremented for each new view v.gname: name of the associated group v.offset: when a member joins, it can provide a value (the “offset”) for use in sending it initial state. The idea is to allow you to “top off” the state if some part of the state was pre-transferred before the join. E.g. suppose the group state is an append-only log. You could pre-transfer 10Mb and have the offset be 10Mb. At join time only the tail of the log needs to be transferred. The offset could also represent the number of updates that have been applied to a database.

Example: A view handler • This view handler will be called each time membership changes, including when the group is created or initially joined, and when the program leaves the group or “terminates” it • The example updates the title on the console window in Windows g.ViewHandlers+= (ViewHandler)delegate(View v) { Console.Title = "[" + g.gname+ "]: " + Isis.Isis.my_address + ": ViewId=" + v.viewid + ", MyRank=" + v.GetMyRank(); };

Key Concept:Address Within Isis2, each program has an associated Address. The Address includes the IPv4 address of the host on which it is running, port numbers for contacting it, and a process id. Isis2 groups also have addresses. You can print an Address in short or long-format. Call a.ToStringVerboseFormat() for long format.

myAddress, my rank • After IsisSystem.Start() IsisSystem.GetMyAddress() returns your own address. • Used to look yourself up in a View to learn your “rank” within that view. Use v.GetMyRank() • If a process has rank r in view v, this just means that v.members[r].Equals(myAddress) • Note: Compare Addresses with .Equals(), not ==

Address of some other process? • You can obtain this from • The view of a group you both belong to • The process could write it down for you in a file • If you know the machine on which the process will run, you can use new Address(hostIPv4, pid); • In RunAsWorkeryou can use 0 for the pid. • We’ll learn about this later.

Uses of the Address? • You can send a message directly to a process • You can query a process • You can tell Isis2 about a failure, if the process belongs to a group to which you also belong.

Key Concept: Consistency The central property that makes Isis2 awesome!

Consistency • One aspect of consistency is that each member of a group sees the same membership views, in the identical order • This is useful for coordinating actions • Notice that since each has its own rank, it is easy to subdivide a data set using the rank • v.members.Length: the total number of members. Call it N. • v.GetMyRank(), v.GetRankOf(a): gets rank values as integers in the range [0 .. (N-1)]

Simple thought problem 0 1 2 • A group has view v. The group maintains an array of data with R rows. If we want each member to do roughly equal work, how would you do that? If the group had N members (3 in this example) the caller will need to merge the 3 subreplies to obtain the answer to the query. Thus work occurs both in the group and in the caller. The caller aggregates the answers

Why does this work? • Key insight: when they receive the query, all members have the same current view for the group • A virtual synchrony guarantee • … and all members receive the Query. • So if we divide the work this way, each member knows what its share of the work is! • What about failures?

Overcome failureswith redundancy • How do we generate redundant replies? • Can the query caller “tell” if it is missing part of the answer? • Not without help: When Query is issued, we don’t know what the membership will be at the time of delivery • … so have the reply include “This is reply k of R”. The caller can build an array of R replies, and if it got one copy of each, the set of replies is complete.

Redundant replies: Option A One Reply but it contains two answers • Have each member handle two sets of data • g.Reply(# members, myRank, primary, secondary) If the caller is told how many members were in the group, and which parts of the answer this reply includes, it can merge the results and discard duplicates. It will either have a full set of results, or if any are missing, we know we had (unlikely) two “correlated” failures.

Redundant replies: Option B • In a large enough group, we can have 3 members as primary reply senders, 3 more as backups • The primary and backup members send Replies • The non-participants call g.NullReply(); • Caller will get 6 responses, each marked “1 of 3” etc, with each duplicated. Keeps just one • Either option is perfectly reasonable. With option B we can also spread the work around if a big group will receive lots of concurrent queries • Later when we learn about sharded data, the DHT, and subset queries, we’ll discover an option C…

Message handler • The actions taken when a message arrives are determined by the code in the message handler • This handler gets a query that passes in a string, and then replies by telling the caller how many members were in the group, its own rank, and sending back its share of the Query answer. myGroup.Handlers[2] += (Action<string>)delegate(string name) { . . . Compute my share of the reply . . . g.Reply(GetView().members.Length, GetMyRank(), myShare); };

Message handler • Notice that each handler • Has an associated request code. A small integer in the range [0..127]. We use request codes to simplify the application logic. • Has a “type signature”: the arguments to the upcall and their types. For example, string, int, etc. • You can specify application-defined types too, but if so must register them with Isis2 before using them • An incoming message must match a handler with the proper request code, and must exactly match the given types. The type match is highly restrictive.

Key Concept:Action. Delegate. In C#, an Action is a special type that denotes a method. It can specify the types of arguments, as in this example: Action<string> A delegate is a kind of anonymous method. It should match the Action type signature. A delegate doesn’t need a name (C# makes one up). Example: delegate(string name)

Key Concept:Tell Isis2 about a type Isis2 needs to be able to put object into messages and extract them from messages You can supply types of your own but must register them first.

Example: A user-defined type • Each type is given a small integer “type code” [0..127] • This example is automatically marshalled by Isis2: all public data fields will be transmitted • AutoMarshalled types must have a public constructor that takes no arguments. [AutoMarshalled] publicclassmyData { publicint d; publicmyData() { } publicmyData(int v) { d = v; } publicoverridestringToString() { returnbprint(d); } } Msg.RegisterType(typeof(myData), 0);

More about user-defined types • You can also define classes in which only some fields will be marshalled • Provide a method ToBArray() that Isis2 can call, and that will convert the object into a byte-array representation • Provide a constructor myData(byte[] ba) to convert a byte-array back into the object • You can use C# serialization. The Isis2Msg class has methods that will convert to and from byte arrays this quite efficiently. They handle endian ordering.

Key Concept:state transfer When a member joins a group, we use “state transfer” to initialize it Your code will decide what this state contains You register a “checkpoint maker” and “checkpoint loader” methods.

Example: Checkpoint methods • This code sends two messages, one with an integer in it, one with a double. The loader methods accept the incoming checkpoint data. • The example is very inefficient. A single call to SendChkpt could have many data objects in it (including complex objects such as arrays, lists, etc) g.MakeChkpt+= (Isis.ChkptMaker)delegate(Viewnv) { g.SendChkpt(istuff); g.SendChkpt(dstuff); g.EndOfChkpt(); }; g.LoadChkpt += (loadichkpt)delegate(int what) { istuff= what; }; g.LoadChkpt += (loaddchkpt)delegate(double what) { dstuff= what; };

Fancier checkpointing ideas • Consider pre-transferring as much state as you can prior to the join. Then use the offset argument to g.Join(); it will show up as the v.offset value in the checkpoint maker, which can send just the delta! • It is ok to call SendChkpt() more than once, but we do that mostly if the data will be large. Normally, one call is enough. • Isis2 lets you specify a “checkpoint choser” method. If you do, you can control which existing member sends state for a particular joining member.

Out of Band Transfer • A very good way to pre-transfer data is this: • Call IsisSystem.Start(); • Join a helper group that has no state in it. • Using the Isis.Client() method, do an RPC request to some member of the “data-heavy” group you plan to join. Ask it to do an “out of band” transfer to the joining member. (We’ll discuss the OOB layer soon) • Once this data has arrived, dispose of the Client handle and do a g.Join(offset) for the data heavy group, specifying an offset that tells how much data you already have. • Only transfer this remaining delta during the join itself.

Key Concept:Unicast Only legal after g.Join() has succeeded. g.P2PSend(dest, rcode, args): sends a point-to-point message (FIFO in this case) to a single specified member. The request code and args must match some handler within the group.

Key Concept:Unicast Query Only legal after g.Join() has succeeded. g.P2PQuery(dest, rcode, args, EOL, p2presults): sends a point-to-point query (FIFO in this case) to a single specified member. It calls g.Reply(results), and the result value(s) are returned in the “p2presults” list.

Key Concept:EOLMarker A special object used by Isis2 as a separator Separates the list of arguments being sent from the results: end of list marker.

Key Concept:Multicast Only legal after g.Join() has succeeded. g.Send(rcode, args): sends a multicast (FIFO in this case) to all members. The request code and args must match some handler within the group.

Key Concept:Query Only legal after g.Join() has succeeded. Nr = g.Query(N, rcode, args, EOL, qresults): sends a multicast (FIFO in this case) to all members. N of them (you can specify ALL, or any number  0) call g.Reply(results), and the values collected are returned in the lists of qresults. Nr gives the actual number of replies that came back.

Key Concept:Timeout Every query must have an associated timeout, but you can let Isis2 define a default value If you specify it, you provide the amount of time (in milliseconds) to wait for a reply, and the action to take if the timer expires: new Isis.Timeout(2000, Isis.Timeout.TO_NULLREPLY) Possible actions are: TO_ABORT: On timeout, throw an “abort reply” exception. TO_FAILURE: On timeout, declare the slow responder to have failed. TO_NULLREPLY: Ignore slow responders

Key Concept:Failure Isis2 considers a process to have failed if: The process doesn’t respond to messages for ISIS_TIMEOUT seconds (default: 45) You tell Isis2 that the process has failed by calling Isis.ProcessFailed() (this is legal only if the failed process is a member of some group to which your process also belongs). The process terminated gracefully, ideally by calling Isis.Shutdown().

Key Concept: Threads Modern systems use threads. Isis2 uses them extensively

Threads in Isis2 • C# makes it easy to use threads, and Isis2 definitely does… • Normally will have 8 or so system threads (they spend most of their time sleeping) • … plus 2 event delivery threads per group (one for views and multicast messages, the other for unicast messages). Implication: if the thread that delivers a multicast blocks for some reason, no more messages can be delivered until it wakes up! • … a few more are created from time to time as it runs. • The system runs best if you dedicate 1 or 2 cores to it, but no more than 2.

Key Concept:WaitForever() If the main thread exits, Isis2 automatically shuts down. What should you do if your main thread has no additional work to do? You call IsisSystem.WaitForever(); This blocks until something calls IsisSystem.Shutdown();

Key Concept:Client of a group For most purposes, processes join the groups that they participate in or use But you can also be a “client” of a group

Client c = new Client(“group name”); • A Client is a process “bound” to some member of the group, picked at random. • Allows the process to issue unicast (point-to-point) requests to the member (that member could relay them within the group if the logic is coded appropriately). • The member must call g.AllowClientRequests(rcode) for each request code that can accept external client-initiated requests this way. • A client should dispose of the handle (c.Dispose()) if it wishes to Join the group later.

Key Concept: System groups Isis2 has two of them

ORACLE and ISISMEMBERS • The system itself creates two groups. You can’t access them directly. • ORACLE: a small group that manages group membership on behalf of the system. It runs in the first processes you launch and has ISIS_ORACLESIZE members (typically 3). The load on the ORACLE is normally very light. • ISISMEMBERS: A group containing all processes using this instance of Isis2.

Our first complete Isis program • Let’s switch to Visual Studio now and develop a complete Isis2 program that maintains a list of (key,value) tuples (strings and doubles), updates them, includes state transfer, and queries them. • For queries, we’ll have each of the N members do 1/N’th of the work.

Isis 2 Process Groups