Technical Stream Session 5: JGroups

1. Distributed Systems Technical Stream Session 5:JGroups CSC 253 Gordon Blair, François Taïani

2. Overview of the Session • What is JGroups? • JGroups’ Architecture • The Channel Class • The Protocol Stack Infrastructure • The Building Blocks • Comparison with JMS G. Blair/ F. Taiani

3. What is JGroups? • A toolkit for reliable multicast communication • Open source (LGPL license from Free Software Foundation) • Implemented in Java • Used in industrial strength products like JBoss • to implement the JMS API, see last week TS lecture • Key Feature: Flexible Protocol Stack • Can be tailored to application needs and network characteristics • Can be extended • Provides a wide spectrum of properties. Focuses on reliability. • You only pay for what you really need • JGroups was created by Belan Ban when at Cornell Uni (US) • Incidentally the home of Isis (see group comm lecture last week) • Where Werner Vogel (Amazon’s CTO) worked for a long time • You know where to apply for your next summer job! G. Blair/ F. Taiani

4. higher level semantics building blocks fine tuning, adaptation, porting protocol stacks JGroups Architecture Channels socket like Network (JVM Networking API) G. Blair/ F. Taiani

5. properties of the channel (see later) sender's address (filled up by protocol stack) all group members I receive my own message JGroups API: Channels • Design by Simplicity • One core class: org.jgroups.Channel String props="UDP:PING:FD:STABLE:NAKACK:UNICAST:" + "FRAG:FLUSH:GMS:VIEW_ENFORCER:" + "STATE_TRANSFER:QUEUE"; Message send_msg;Object recv_msg;Channelchannel=new JChannel(props); channel.connect("MyGroup"); send_msg=new Message(null, null, "Hello world");channel.send(send_msg); recv_msg=channel.receive(0); System.out.println("Received " + recv_msg); channel.disconnect();channel.close(); G. Blair/ F. Taiani

6. Protocol Stacks • Each Channel instance sits on top of a protocol stack • Stack content defined by the property string of the Channel: • Channel myChannel = new JChannel("LAYER1:LAYER2") • Protocol stack composed out of "layers" • Messages go up and down the layer stack • Each layer can modify, reorder, pass, drop oradd a header to messages Channel Channel Channel Layer1 Layer1 Layer1 Layer2 Layer2 Layer2 Network G. Blair/ F. Taiani

7. Protocol Layers (1) • ChannelmyChan = new JChannel("UDP:PING:FD:GMS"); • Stack contains layers UDP, PING, FD, and GMS (bottom-up) • Corresponds to classes: org.jgroups.protocols.UDP org.jgroups.protocols.PING org.jgroups.protocols.FD org.jgroups.protocols.GMS • UDP: IP multicast transport based on UDP • PING: initial membership (used by GMS) • FD: Failure detection (heartbeat protocol) • GMS: Group membership protocol. • Some expertise needed • Syntactically any combination possible • Does not necessarily make sense • For instance GMS does not work w/o PING Channel GMS FD PING UDP G. Blair/ F. Taiani

8. Protocol Layers (2) • Example of other protocol layers • CAUSAL: causal ordering layer using vector clocks • TOTAL: total ordering layer using a message sequencer • NAKACK: negative ACKs (NAKs), paired with positive ACKs • STABLE: computes the broadcast messages that are stable • PERF: measures time taken by layers to process a message • COMPRESS: compresses the payload of a message • Constrains must be obeyed • Usually same layers needed in all group members • Not always the case, see PERF for instance • Position constrains • PERF must be top • Dependencies • GMS needs PING, STABLE needs NAKACK G. Blair/ F. Taiani

9. Building Blocks Channel Building Blocks • Channel class very simple • Similar to sockets, but group behaviour • Message based. No reply/request concept. • Active (pull-style) message reception • Explicit threading needed on top of channel for passive reception • Building Blocks (org.jgroups.blocks package) • Higher level classes on top of Channel • Provides higher level programming abstractions • Examples • PullPushAdapter: passive reception • RpcDispatcher: remote invocation • VotingAdapter:distributed voting G. Blair/ F. Taiani

10. PullPushAdapter: Example public class PullPushTest implements MessageListener { public static void main(String args[]) throws ... { Channelchannel=new JChannel(); channel.connect("PullPushTest"); PullPushAdapteradapter=new PullPushAdapter(channel); adapter.setListener(this); for(int i=0; i < 10; i++) { System.out.println("Sending msg #" + i); channel.send(new Message(null, null, "Hello "+ i)); Thread.currentThread().sleep(1000); } adapter.stop(); channel.close(); } public void receive(Message msg) { System.out.println("Received msg: " + msg); } } G. Blair/ F. Taiani

11. Small Quizz • Imagine 5processes execute the previous code • How many "Hello i" will be printed in total? • How many would this be with nprocesses? • First let's think about the "Hello 0" message • Process 1 sends it to all group members • All 5 members receive and print "Hello 0" from process 1 • Process 2 sends it to all group member too • All 5 members receive and print "Hello 0" from process 2 • Dito for process 3,4,5: As a whole 5x5 = 25 • 10 "Hello i" messages: 250 as a whole! • Home exercise: general formulae with n processes G. Blair/ F. Taiani

12. RpcDispatcher public class RpcDispatcherTest { public int print(int number) { return number * 2; } public static void main(String[] args) throws ... { Channelchannel=new JChannel(); channel.connect("RpcDispatcherTestGroup"); RpcDispatcherdisp= new RpcDispatcher(channel, null, null, this); RspListrsp_list= disp.callRemoteMethods(null,"print", new Integer(i), GroupRequest.GET_ALL, 0); System.out.println("Responses: " +rsp_list); channel.close(); disp.stop(); } } G. Blair/ F. Taiani

13. RpcDispatcher: Quizz • Assuming 3 processes execute the previous program • How many times will the print method be invoked? • Answer: • For each callRemoteMethods, print executed by all 3 processes • callRemoteMethods is called by all 3 processes • Total number of times print is invoked: 3x3 = 9 • Previous code completely symmetric • Possible only to have one group member perform an invocation • One group member can act as a proxy to external clients • Clients may contact any of the group members G. Blair/ F. Taiani

14. RpcDispatcher (2) • Provides “Group” Remote Procedure Call behaviour • Looks up the invoked method (here “print”) • In example collects answers from all group members • Each group member is potentially both a client and a server! • Not completely transparent • No stub or skeleton to hide remote invocation • Arguments passed as an array of Object • Return behaviour can be adapted • GET_ABS_MAJORITY: return majority (of all members, may block) • GET_ALL: return all responses • GET_FIRST: return only first response • GET_MAJORITY: return majority (of all non-faulty members) • GET_N: return n responses (may block) G. Blair/ F. Taiani

15. Comparison with JMS • JMS is a standardised API • Various implementations • JGroups is a library • Has its own API (not JMS compliant) • Only one implementation • Can be (and is) used to implement the JMS API (in JBoss) • JMS is a Message Oriented Middleware • Higher level that plain group communication as in JGroups • E.g. persistence, durability, transactions • JMS assumes a server based architecture • JGroups can be used in fully decentralised manner (or not) G. Blair/ F. Taiani

16. To Dig Further • The JGroups Web site • http://www.jgroups.org • JGroups-ME: JGroups for mobile devices • http://www.jgroups-me.org/ G. Blair/ F. Taiani

17. Expected Learning Outcomes At the end of this 5th Technical Session: • You should know what JGroups is about • You should appreciate the basic working of the Channel class • You should understand JGroups’ protocol stack mechanism • You should be able to compare JGroups to JMS G. Blair/ F. Taiani