Mathematics-Inspired Protocols for Distributed Systems

1. Mathematics-Inspired Protocols for Distributed Systems Yookyung Jo** Dept. of Computer Science Cornell University {ykjo@cs.cornell.edu} Steven Y. Ko*, Indranil Gupta Dept. of Computer Science University of Illinois at Urbana-Champaign {sko@cs.uiuc.edu; indy@cs.uiuc.edu} * Currently PhD student at UIUC. In the audience here. ** Work done during M.S. at UIUC.

2. An Open Design Challenge Phenomena and Results from Biological, Physical, Social Worlds, etc. Self-adaptive Protocols for Distributed Computing Problems

3. Lost in Translation Two popular ways today for this translation: • Sit down with social scientists, biologists etc.  Time consuming, terminology different • But good to talk! • Read textbooks written by them and derive protocols that “somewhat” model the phenomenon Both above approaches often lead to: • Hand-wavy design: non-rigorous translation leads to unpredictable protocols • Difficulty of analysis of derived protocol • Lack of generality of translation

4. The Third Way Phenomena and Results from Biological, Physical, Social Worlds, etc. Self-adaptive Protocols for Distributed Computing Problems Mathematical Models

5. Why Mathematical Models? For long, a popular language for representing phenomena and ideas • Scientists from fields of Biology, Sociology, Physics, etc. have used these models to represent phenomena, results, and ideas • Many decades (or centuries) of equations available in these fields • E.g., Sequence Equations, e.g., Translation is Systematic and Rigorous • Derive Protocol from Mathematical model (equation) • Translation is not hand-wavy Derived Protocol is easy to understand • Rigorous analysis and provable properties of derived protocol • Generality of translation • Amenable to augmentation with topology-awareness, etc.

6. Story of this Paper • Consider a popular class of mathematical models • E.g., Sequence equations, e.g., • Develop techniques for translating any mathematical equation belonging to this class into  a distributed protocol • Key idea: Emergent behaviorof the protocol across the distributed system = the mathematical equation • We are not simulating the mathematical equation at each process • Challenge: going from global (equation)  local (protocol) • Use these techniques to design adaptive protocols for P2P computing

7. Roadmap • Related Work and System Model • Translation of Sequence Equations into Sequence Protocols • Adaptive Protocols for P2P Computing • HoneyAdapt system for Grid computing

8. Related Work • Simulation of mathematical equation at each process [Uresin90] • Instead, our focus is on running a protocol that obtains equation as emergent behavior • Nature-inspired research, e.g., [Mute,AntNet], etc., and population protocols [Merritt00,Angluin04] • But not derived from mathematical models • [Gupta04] considered translation of continuous differential equations into equivalent distributed protocols • Current SASO 07 paper considers translation of sequence equations. Main differences from [Gupta04]: • Sequence equations discrete and not continuous • Require completely different translation techniques • Adaptive and phase-change behavior more pronounced

9. System Model • Static group of N non-faultyprocesses (N large)  Can be relaxed for most sequence protocols • Reliable unicast communication • TCP • Sequence protocols resilient to message losses • Coarse-grained time synchronization (O(minutes)) • allows processes to move synchronously • allows notion of “rounds” • Provided by NTP, TIME, or DAYTIME (e.g., NIST servers) • Many sequence protocols have asynchronous variants • Any process can randomly sample another process  Use CYCLON [Voulgaris05], Peer sampling service [Jelasity04], etc.

10. Creating Sequence Protocols • Canonical Sequence Equation: • is a variable in [0,1] • is its value at time t • k is constant [All our discussion is extendible to multi-variable sequence equations] • Challenge: global  local • Assign each process p a binary state variable Xp, representing whether it is in state X or not. Xp=1 means the process is in state X. Xp=0 means process is not in state X. • Let x = fraction of processes (system-wide) with Xp=1. So, • Derive a distributed protocol so that the time-variation of x is predicted by the sequence equation. That is:

11. Each round at process p: Flip a coin with heads probability r if heads Xp=1 else Xp=0 => Value of x, the fraction of processes in state X, is predicted by: Case Study I (Constant Term)

12. Each round at process p: remember last round’s Xp value // Token Generation if last round’s Xp was 1 generate an expected r tokens relay token to random process // Token Relay hold at most one token at any time if receive any additional tokens relay it to a random process // Token Apply at end of round if have > 0 tokens set Xp=1 else Xp=0 Number of tokens generated is Value of x, the fraction of processes in state X, is predicted by: This Protocol can be extended to: Arbitrary memory (k in sequence equation) Multi-variable equations Case Study II (Linear Term) Multiplicative Protocol

13. Stepping Back – General Methodology For the sequence equation: • Take each term on the right hand side • Term is minimal unit separated by + and – signs • Translate each term according to appropriate case studies • Generate positive tokens for + terms, and negative tokens for – terms • A positive token destroys a negative token • Relay and Apply tokens as usual Theorem: If for each term T, number of tokens generated is T X N, then Term Translation =Case Study

14. I. Polynomial Terms: Constant – Case Study I Linear – Case Study II (multi-variable equations) Multiplicative Polynomial - II. Non-polynomial Terms: Division Terms - Fractional Terms – next III. Recursive Translation – next What other Terms can we Translate? see paper see paper

15. Each round at process p: remember last k round’s Xp values divide round into two equi-long subrounds // Subround 1 // Token Generation for each j =1 to L if Xj(p)=1 generate aj tokens tagged with j // Token Relay and Apply multicast tokens to all other processes // Subround 2 // Token Generation select random token among those received suppose tag is j’ if (bj’=1) generate a token for subround 2 // Token Relay and Apply apply as usual Subround 1: E[Number of tokens generated at p] is= Subround 2: E[Number of tokens generated at p] is= Value of x, the fraction of processes in state X, is predicted by: Subround 1 Round Subround 2 Translation of Fractional Term

16. Subround 1 Round Subround 2 Recursive Translation • Any term that consists of sub-terms that are translatable, can itself be translated • Split a round into two subrounds • In subround 1, run the derived protocols for the subterms • In subround 2, run the derived protocol for the overall term • Subround division is also recursive STEP1 STEP2 EXAMPLE

17. Roadmap • Related Work and System Model • Translation of Sequence Equations into Sequence Protocols • Adaptive Protocols for P2P Computing • Multiplicative Protocol (based on ) • For detecting global thresholds in a distributed fashion • see paper for details • HoneyAdapt system for Grid computing • next

18. Challenge: how do clients choose “best” algorithm (A,B,…L) adaptively at run time in a black-box manner? e.g., for parallel sorting problem, A=quicksort, B=insertion sort,… HoneyAdapt - Motivation • Grid Server (master) • Partitions large data set • into chunks • Serves out chunks • on-demand to clients • Collates results in the end • E.g., parallel sorting • problem, graphics • rendering, etc. 1. Fetch next data chunk 3. Send back results to server Typical Client Grid Clients (workers) Connected in an overlay 2. Process data chunk using one of algorithms A,B,C,D,…L

19. HoneyAdapt – Inspiration Nectar Source A Nectar Source B • Honyebees (apis mellifera) • need to decide which is the • “better” nectar source • -in a distributed fashion

20. HoneyAdapt – Inspiration Nectar Source A Nectar Source B • (time t) • Forage a • nectar source 2. With probability (1-pf), use the same nectar source for time (t+1) pf=following probability 4. After dance, if did not follow (so with probability pf), decide next source to forage by picking a dancing bee at random 3. Execute honeybee dance of 8’s -Duration of dance proportional to quality of advertised nectar source -Direction of dance points towards source

21. Linear Term Fractional Term (+Recursive) HoneyAdapt – Mathematical Model Source A = Algorithm A Source B = Algorithm B = Bees converge quickly towards better source (proof in paper) Fraction of nodes (bees/clients) foraging source (algorithm) i at time (t+1) Following probability Quality of source (algorithm) j (See paper for general model. From [Seeley96].)

22. (Recall) HoneyAdapt - Motivation • Grid Server (master) • Partitions large data set • into chunks • Serves out chunks • on-demand to clients • Collates results in the end • E.g., parallel sorting • problem, graphics • rendering, etc. 1. Fetch next data chunk 3. Send back results to server Typical Client Grid Clients (workers) Connected in an overlay 2. Process data chunk using one of algorithms A,B,C,D,…L

23. Algorithm’s emergent behavior = Sequence equation (proof in paper) Fraction of nodes (bees/clients) foraging source (algorithm) i at time (t+1) Following probability Quality of algorithm j HoneyAdapt –Model and Derived Protocol = 2A. Choose algorithm i (initially, random) for this chunk 2B. With probability (1-pf), use same algorithm for next chunk 2B. Dance: create a number of advertisement messages for algo i. Number of adv. msgs. proportional to the quality of sorting (inversely proportional to running time of chunk with algorithm i) 2C. Send advertisement messages to immediate neighbors in overlay 2D. If follow (prob. pf), decide algorithm i for next chunk by picking an advertisement message at random

24. HoneyAdapt - Simulation Setup: * Random graph overlay of ~1000 clients * Dataset consists of 100K chunks of 10 different types * Each type has 10 algorithms assigned randomly in terms of quality * “Cluster”=consecutive chunks of same type (with same “best” algo.) * pf=0.9 Adaptivity: HoneyAdapt takes only 2x time compared to optimal, and beats non-adaptive strategies • Scalability up to and beyond 4000 nodes: • Running time: Only 85% worse than optimal • Bandwidth: 0.04 messages/node/chunk

25. Honeysort beats both quicksort and insertion sort! HoneySort – Deployment Setup: * Up to 30 COTS PC clients (Linux) * Complete graph overlay with TCP links * Clients choose between quicksort and insertion sort * Sort 1 million database of 8 B entries * Server pre-partitions data into 333 chunks • Results: • Sorted Arrays: HoneySort as good as insertion sort  Randomized arrays: HoneySort as good as quicksort  Part-sorted part-randomized arrays:

26. Summary • This paper: • Model=Sequence Equations • Translation techniques for polynomial/non terms • Derived Sequence Protocol so its emergent behavior = Sequence equation • HoneyAdapt for adaptive Grid computing • HoneySort beats traditional parallel sorting algorithms Phenomena and Results from Biological, Physical, Social Worlds, etc. Self-adaptive Protocols for Distributed Computing Problems Mathematical Models Distributed Protocols Research Group (DPRG): http://kepler.cs.uiuc.edu

27. Backup Slides

28. Each round at process p: remember last k round’s Xp values divide round into two equi-long subrounds // Token Generation [subround 1] integer i=0 do select a random process q query the value of Xq k rounds ago i=i+1 until (Xq=1) generate i token messages // Token Relay [subround 2] relay token to random process hold at most one token at any time if receive any additional tokens relay it to a random process // Token Apply [subround 2] at end of round use tokens for next subround E[Number of tokens generated at p] is Total number of tokens generated is Value of x, the fraction of processes in state X, is predicted by: This Protocol can be extended to: Arbitrary memory (k in sequence equation) Multi-variable equations Translation of Division Term (usually a sub-term in a larger term) Subround 1 Round Subround 2

29. Big Picture • Self-adaptive and self-organizing distributed protocols • Protocol design • Biological, Physical, Social phenomena as a source of ideas for protocol design • Need: Systematic Translation of phenomena into distributed protocols • Use mathematical models as a conduit