150 likes | 313 Views
Cognitive Publish/Subscribe for Heterogeneous Clouds. Šarūnas Girdzijauskas , Swedish Institute of Computer Science (SICS) sarunas@sics.se Joint work with: Fatemeh Rahimian (SICS). Future Clouds?. Based on decentralized architecture
E N D
Cognitive Publish/Subscribe for Heterogeneous Clouds ŠarūnasGirdzijauskas, Swedish Institute of Computer Science (SICS) sarunas@sics.se Joint work with: FatemehRahimian (SICS)
Future Clouds? • Based on decentralized architecture • Abundance of networked collection of connected devices forming micro-clouds • Decentralized Publish/Subscribe service • Content distribution • IP TV Streaming • Online gaming • Collaborative editing • Etc.. • Adapting to the topology and network dynamics of microclouds • Adapting to different usage patterns ŠarūnasGirdzijauskas, Cloud Futures, Redmond, April 2010
Pub/Sub Systems: Our Focus Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010 • Scalable pub/sub service • Very large number of nodes • Very large number of “topics” • Heterogeneous environments • Arbitrary geographical distribution • Arbitrary subscription and dissemination patterns • Central solutions will not scale
Pub/Sub Systems: Our Focus (2) Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010 • Tradeoffs: • Node degree • Number of uninterested (relay) nodes involved • Dissemination delay • Dissemination cost • Cognitive pub/sub: • Fixed node degree • Account for the underlying topology (bandwidth & cost) • Minimize the number of relay nodes by exploiting user subscription correlation & event publication rates
Conceptual Architecture Pub/sub Dissemination structures Cognitive Overlay Physical Network Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010
Gossip based pub/sub • Gossip (epidemic) overlays • A lot of research (e.g., Cyclon, T-man) • Lightweight, scalable and robust mechanism • Cyclic/Periodic, pair-wise interaction between peers (bounded amount of information) Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010
Towards Cognitive Structure • Gossiping enables us to find and cluster peers with similar interests connected by cheap and fast links • A node starts with a local fixed size view in a random network • Performs a bidirectional exchange of the view with a random node 2 views • Keeps the only the preferred (ranking function) nodes in the view 1 view • Repeat ŠarūnasGirdzijauskas, Cloud Futures, Redmond, April 2010
Building Cognitive Structure Gossiping $ Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010 • Making clusters by utilizing ranking function which prefers neighbors with similar interests • Peer interest similarity metric • Node subscriptions s1, s2 ⊆ T • sim(s1, s2) =|s1∩ s2|/|s1∪ s2| • Weighted by link cost (bandwidth and $) • Weighted by Topic publication rates • Number of neighbors is limited! • Decided locally on each peer
Problem: How to publish? • Clustering peers of similar interests into bandwidth and cost effective clusters • Clusters might (will) be disjoint • Event publishing requires connected components for each topic Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010
14 12 24 7 32 13 10 34 Navigable Small-World Network 9 6 20 5 29 16 18 2 31 21 8 15 22 28 3 4 1 23 27 19 35 17 26 30 11 Inter-Cluster Connectivity • Structure is added: Navigable Small-World topology • Purely by using gossiping Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010
Building Navigable Structure 2 5 8 1 4 3 7 6 9 10 Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010 • Every peer decides on random ID • Updating ranking function for choice of neighbors: • Ring Link(s) • Long-Range link (Small-World style) for polylogarithmic routing performance
14 12 24 7 32 13 10 34 Navigable Small-World Network 9 6 20 5 29 16 18 2 31 21 8 15 22 28 3 4 1 23 27 19 35 17 26 30 11 Inter-Cluster Connectivity • Structure is added (Navigable Small-World made by gossiping) • Ring Links • Long-Range (finger) link(s) • Clustering (friend) links • Clusters are connected by greedy routes • Rendezvous node for each topic • All links are used! • All topics become connected • For publishing “flood the topic”, or • Choose a rendezvous node to publish Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010
Ongoing work • Synthetic data sets for user subscription correlation • Twitter data set • Skype churn data • Our experiments show: • Up to 10 fold reduction of relay traffic as compared to existing approaches (e.g., Scribe, Bayeux) Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010
Gossip based pub/sub (recap) • Large scale pub/sub for heterogeneous environments • Dissemination structures are self-organizing • Forming clusters of similar nodes • Converging into least expensive dissemination paths on the underlying physical network • Continuously adapting to the environment conditions • Fast convergence, robustness to churn and failures. Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010
Thank you!Questions? Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010