Towards Scalable Pub/Sub Systems

1. Towards Scalable Pub/Sub Systems Shuping Ji1, Chunyang Ye2, Jun Wei1 and Arno Jacobsen3 1Chinese Academy of Sciences 2Hainan University 3Middleware Systems Research Group, University of Toronto

2. Existing routing algorithms for Pub/Sub Event Flooding: Events are flooded to brokers, heavy bandwidth use. Multicast-based Routing (MBR): Event space is partitioned into disjoint multicast groups (the right graph), hard to partition, bandwidth waste. Filter-based Routing (FBR): Widely used method, events are only routed to interested brokers, redundant matching, forwarding by non-interested brokers.

3. Limitations of the FBR algorithm Difficulty in supporting general overlay topologies. Subscription duplication: As shown in right graph, subscriptions are duplicated at B1, B2, B3, and B4. Redundant and repeated event matching. Lack of flexibility in supporting overlay reconfiguration: hard to change the topology in left graph to the one in right graph.

4. D-DBR and MERC algorithm Dynamic Destination-based Routing algorithm • Event matching and event routing are decoupled. • Subscriptions from local and remote brokers are separated. • Topology can be dynamically configured for optimization. • Events are attached with destinations of interested brokers. Match at Edge and Routing inter-Cluster algorithm • Overlay is divided into interconnected clusters. • The destination list size is limited. • Better scalability than D-DBR algorithm. • A reference model to construct large-scale pub/sub systems.

5. Two layers of D-DBR algorithm Matching Layer: • Only interested events are inserted into Match Queue. • Routing information divided into four tables. Multicasting Layer: • In charge of transferring events to interested brokers. • Topology changes do not affect upper matching layer.

6. Matching and routing layer Local/remote events/subs processing at the matching layer Event routing at the multicasting layer

7. MERC routing example

8. Evaluation Experiments on Computing Facility • We implemented both D-DBR and MERC algorithm in PADRES , a representative, open-source, content-based pub/sub system based on the FBR algorithm. • Experiments are executed on both acyclic linear topology and general topology. • Evaluation metrics: System Throughput, Event Routing Latency, CPU & RAM Utilization, and Subscription Duplication. Experiments Based on Simulations • Evaluation metrics: Destination List Overhead, Routing Accuracy, System Robustness, Topology Maintenance Overhead. • Routing accuracy is defined as the ratio between the number of brokers interested in an event over the number of brokers receiving that event when it is routed towards the interested brokers.

9. Performance Latency in acyclic linear topology CPU utilization in acyclic linear topology Latency in general topology Sub duplication in general topology

10. Routing accuracy and robustness Robustness under connection failures Routing accuracy improvement for D-DBR Robustness under broker failures

11. Destination list overhead Average destination list size Average destination list size Destination list size distribution Destination list size at source brokers