Transactional Mobility in Distributed Content-Based Publish/Subscribe Systems

1. MIDDLEWARE SYSTEMS RESEARCH GROUP Transactional Mobility in Distributed Content-Based Publish/Subscribe Systems Songlin Hu*, Vinod Muthusamy+, Guoli Li+, Hans-Arno Jacobsen+ * Chinese Academy of Sciences, Beijing + University of Toronto June 25, 2009 29th Int’l Conference on Distributed Computing Systems(ICDCS 2009) • http://padres.msrg.toronto.edu

2. Many adaptive distributed applications require reprovisioning of software components Grid computingplatforms Stream processingengines Multiplayer onlinegames Distributed content-based publish/subscribe Mobileenvironments Mobile agentframeworks Process executionengines Transactional Mobility in Pub/Sub

3. Movement (reprovisioning) of software components should be well-behaved • No lost or duplicated messages • Other components not disrupted • Movement is transparent to both moving and stationary components • Isolated from operations by other components Transactional Mobility in Pub/Sub

4. Contributions • Define properties for mobile clients in a distributed publish/subscribe system • Develop protocols that achieve efficient movement • Evaluate proposed and traditional movement protocols Transactional Mobility in Pub/Sub

5. Publisher Subscriber Review of distributed publish/subscribe routing 1. Advertise 3. Publish 2. Subscribe Advertisement: item = computer brand = ibm price < 2000 Subscription: item = computer brand = ibm price < 1500 Publication: item = computer brand = ibm price = 1400 Transactional Mobility in Pub/Sub

6. A movement operation relocates a client (including its state) Client Container 1 Client Container 7 MOVE (to Broker 7) Client A Client A B1 B7 ∙∙∙ B2 B8 Client B • Movement may fail because target broker rejects it, etc. • During movement, there may be multiple copies of a client, but only one should be “running” at any time • Client container helps coordinate the movement Transactional Mobility in Pub/Sub

7. Transactional Movement Properties Transactional Mobility in Pub/Sub

8. Modular transactional properties simplify reasoning and implementation Client Application Pub/Sub Stub Notifications Ads/Subs/Pubs Routing (Considered in this work) (Out of scope) Messaging Broker Transactional Mobility in Pub/Sub

9. ACID-like transactional properties for various layers Atomicity Consistency Isolation Client layer A moving client must be exclusively either at its source or target broker. There must be at most one running instance of each client. Only the initial or final states of a moving client should be observable. Notifications layer Notifications are delivered exactly once to either the source or target client. A moving client (whether successful or not) should receive the same set of notifications as one that did not move. The set of notifications delivered to stationary clients from a moving publisher should be independent of whether the publisher successfully moves. Routing layer Either all or none of the set of routing table updates required for an operation (adv, sub, etc.) should all be applied. Each broker should have the minimal set of routing table entries for a set of advs and subs. A movement should only modify those routing table entries associated with the moving client. • Durability is omitted but can be achieved by persisting all state to stable storage • Refer to the paper for formal definitions Transactional Mobility in Pub/Sub

10. Movement Protocol Transactional Mobility in Pub/Sub

11. The source and target brokers negotiate the movement of a client • The intermediate broker routing tables are reconfigured along with message (2) Transactional Mobility in Pub/Sub

12. The client and coordinator at the source and target are modelled with state diagrams • Coordinators are based on the three-phase commit protocol • Global reachable state graph is used to prove some properties • E.g., in the commit state, the source client is clean, and the target client is started

13. The broker routing tables must be reconfigured when the client moves • Routing tables should be correct whether movement succeeds or fails • Reconfiguration should be efficient • Network traffic • Broker computation • Isolated from operations Transactional Mobility in Pub/Sub

14. Publisher Subscriber Unadv Traditional end-to-end movement can be expensive Bi Bl Bj Transactional Mobility in Pub/Sub

15. Traditional end-to-end movement can be expensive Publisher Subscriber Adv Bi Bl Bj Transactional Mobility in Pub/Sub

16. The presence of a covering advertisement can avoid flooding Bi Bl Bj

17. Movement can still trigger burst of covered messages A B C B B1 Bi A Bl Bj A A A C B2

18. Proposed protocol reconfigures routing tables hop-by-hop Publisher Subscriber 1 Routing entry 1 2 Routing shadow copy 1 1 1 1 2 2 2 Bi Bj 1 (2) Approve message Transactional Mobility in Pub/Sub

19. Proposed protocol reconfigures routing tables hop-by-hop Publisher Subscriber • Only brokers between Bi and Bj are updated • Predictable number of messages • Advertisement entries are simply reversed • Subscription entries are more complicated 1 Routing entry 1 2 Routing shadow copy 1 1 1 1 1 2 2 1 2 1 Bi Bj 1 (3) Ack + State message Transactional Mobility in Pub/Sub

20. Evaluations Transactional Mobility in Pub/Sub

21. Experimental setup • Two movement algorithms: proposed reconfiguration, traditional covering-based • Deployments in dedicated LAN and shared WAN (PlanetLab) environments • 400 clients repeatedly move between brokers 1,13 and 2,14 • Pause for 10 seconds at each broker • Five subscription workloads: covered, chained, tree, distinct, random • Metrics: network message traffic, movement duration, and throughput Default topology Subscriptions Transactional Mobility in Pub/Sub

22. The reconfiguration protocol is much faster than the covering protocol • Movement of “root” subscriptions is more expensive in the covering protocol Transactional Mobility in Pub/Sub

23. The reconfiguration protocol scales with the number of moving clients • The reconfiguration protocol achieves better movement latency despite more total messages because it is less bursty Transactional Mobility in Pub/Sub

24. The reconfiguration protocol is more stable and efficient with respect to the workload • Covered workload experiences high latency despite relatively few messages due to bimodal behaviour of covering protocol Transactional Mobility in Pub/Sub

25. The covering protocol is very sensitive to the movement of “root” subscriptions • Only one client is moving, confirming the effect of the pathological case for the covering protocol Transactional Mobility in Pub/Sub

26. The presence of stationary clients has little effect on moving clients • Each additional set of moving clients: 10 roots from covered, tree, chained workloads, 10 leaves from previous workloads, 10 from distinct workload Transactional Mobility in Pub/Sub

27. Keeping the length between source and target brokers constant, the topology size has little effect • The covered workload is used here to exaggerate the effects Transactional Mobility in Pub/Sub

28. Experiments on PlanetLab WAN support the earlier conclusions • Longer latencies are due to more limited network and compute resources • Same relative performance and trends are seen Transactional Mobility in Pub/Sub

29. Conclusions • Distributed publish/subscribe systems lack well-defined guarantees for the movement of clients • ACID-like transactional properties were defined for this problem • Properties are modularized to simplify reasoning and implementation • Client layer movement and routing layer hop-by-hop reconfiguration protocols were developed • Evaluations show proposed protocol is more efficient and stable with respect to various parameters • End-to-end movement using covering negatively affects performance • Future work includes more efficient failure resilience in the publish/subscribe routing layer • Use cases for supporting transactions that span multiple operations would be interesting Transactional Mobility in Pub/Sub

30. Q&A Transactional Mobility in Distributed Content-Based Publish/Subscribe Systemshttp://padres.msrg.toronto.edu Transactional Mobility in Pub/Sub

31. Extra slides Transactional Mobility in Pub/Sub

32. Client and coordinator states at the source and target • Coordinators are based on the three-phase commit protocol • The failure and timeout transitions are omitted for brevity

33. The global reachable state graph can be used to prove some of the transactional properties • E.g., in the commit state, the source client is clean, and the target client is started • Refer to the paper for proofs

34. Proposed protocol reconfigures routing tables hop-by-hop Bi Bl Bj • Only brokers between Bi and Bj are updated • Advertisement entries are simply reversed • Subscription entries are more complicated