DISTRIBUTED ALGORITHMS

Several sets of slides by Prof. Jennifer Welch will be used in this course. The slides are mostly identical to her slides, with some minor changes. Set 1: Introduction

Set 1: Introduction DISTRIBUTED ALGORITHMS Spring 2014 Prof. Jennifer Welch

Distributed Systems • Distributed systems have become ubiquitous: • share resources • communicate • increase performance • speed • fault tolerance • Characterized by • independent activities (concurrency) • loosely coupled parallelism (heterogeneity) • inherent uncertainty Set 1: Introduction

Uncertainty in Distributed Systems • Uncertainty comes from • differing processor speeds • varying communication delays • (partial) failures • multiple input streams and interactive behavior Set 1: Introduction

Reasoning about Distributed Systems • Uncertainty makes it hard to be confident that system is correct • To address this difficulty: • identify and abstract fundamental problems • state problems precisely • design algorithms to solve problems • prove correctness of algorithms • analyze complexity of algorithms (e.g., time, space, messages) • prove impossibility results and lower bounds Set 1: Introduction

Potential Payoff of Theoretical Paradigm • careful specifications clarify intent • increased confidence in correctness • if abstracted well then results are relevant in multiple situations • indicate inherent limitations • cf. NP-completeness Set 1: Introduction

Application Areas • These areas have provided classic problems in distributed/concurrent computing: • operating systems • (distributed) database systems • software fault-tolerance • communication networks • multiprocessor architectures • Newer application areas: • cloud computing • mobile computing, … Set 1: Introduction

Course Overview: Part I (Fundamentals) • Introduce two basic communication models: • message passing • shared memory • and two basic timing models: • synchronous • asynchronous Set 1: Introduction

Course Overview: Basic Models Message passing Shared memory synchronous Yes No asynchronous Yes Yes (Synchronous shared memory model is PRAM) Set 1: Introduction

Course Overview: Part I • Covers the canonical problems and issues: • graph algorithms (Ch 2) • leader election (Ch 3) • mutual exclusion (Ch 4) • fault-tolerant consensus (Ch 5) • causality and time (Ch 6) Set 1: Introduction

Course Overview: Part II (Simulations) • Here "simulations" means abstractions, or techniques for making it easier to program, by making one model appear to be an easier model. For example: • broadcast and multicast (Ch 8) • distributed shared memory (Ch 9) • stronger kinds of shared variables (Ch 10) • more synchrony (Chs 11, 13) • more benign faults (Ch 12) Set 1: Introduction

Course Overview: Part II • For each of the techniques: • describe algorithms for implementing it • analyze the cost of these algorithms • explore limitations • mention applications that use the techniques Set 1: Introduction

Course Overview: Part III (Advanced Topics) • Push further in some directions already introduced: • randomized algorithms (Ch 14) • stronger kinds of shared objects of arbitrary type (Ch 15) • what kinds of problems are solvable in asynchronous systems (Ch 16) • failure detectors (Ch 17) • self-stabilization Set 1: Introduction

Course overview: Part IV (Other topics) • Debugging of parallel programs • Distributed function computation • Distributed algorithms for wireless networks Set 1: Introduction

Relationship of Theory to Practice • time-shared operating systems: issues relating to (virtual) concurrency of processes such as • mutual exclusion • deadlock also arise in distributed systems • MIMD multiprocessors: • no common clock => asynchronous model • common clock => synchronous model • loosely coupled networks, such as Internet, => asynchronous model Set 1: Introduction

Relationship of Theory to Practice • Failure models: • crash: faulty processor just stops. Idealization of reality. • Byzantine (arbitrary): conservative assumption, fits when failure model is unknown or malicious • self-stabilization: algorithm automatically recovers from transient corruption of state; appropriate for long-running applications Set 1: Introduction

Message-Passing Model • processors are p0, p1, …, pn-1 (nodes of graph) • bidirectional point-to-point channels (undirected edges of graph) • each processor labels its incident channels 1, 2, 3,…; might not know who is at other end Set 1: Introduction

Message-Passing Model 1 1 p3 p0 3 2 2 2 p2 p1 1 1 Set 1: Introduction

Modeling Processors and Channels • Processor is a state machine including • local state of the processor • mechanisms for modeling channels • Channel directed from processor pi to processor pj is modeled in two pieces: • outbuf variable of pi and • inbuf variable of pj • Outbuf corresponds to physical channel, inbuf to incoming message queue Set 1: Introduction

inbuf[1] outbuf[2] p1's local variables p2's local variables outbuf[1] inbuf[2] Modeling Processors and Channels Pink area (local vars + inbuf) is accessible state for a processor. Set 1: Introduction

Configuration • Vector of processor states (including outbufs, i.e., channels), one per processor, is a configuration of the system • Captures current snapshot of entire system: accessible processor states (local vars + incoming msg queues) as well as communication channels. Set 1: Introduction

p1 m3 m2 m1 p2 Deliver Event • Moves a message from sender's outbuf to receiver's inbuf; message will be available next time receiver takes a step p1 m3 m2 m1 p2 Set 1: Introduction

Computation Event • Occurs at one processor • Start with old accessible state (local vars + incoming messages) • Apply transition function of processor's state machine; handles all incoming messages • End with new accessible state with empty inbufs, and new outgoing messages Set 1: Introduction

Computation Event b a old local state new local state c d e pink indicates accessible state: local vars and incoming msgs white indicates outgoing msg buffers Set 1: Introduction

Execution • Format is config, event, config, event, config, … • in first config: each processor is in initial state and all inbufs are empty • for each consecutive (config, event, config), new config is same as old config except: • if delivery event: specified msg is transferred from sender's outbuf to receiver's inbuf • if computation event: specified processor's state (including outbufs) change according to transition function Set 1: Introduction

Admissibility • Definition of execution gives some basic "syntactic" conditions. • usually safety conditions (true in every finite prefix) • Sometimes we want to impose additional constraints • usually liveness conditions (eventually something happens) • Executions satisfying the additional constraints are admissible. These are the executions that must solve the problem of interest. • Definition of “admissible” can change from context to context, depending on details of what we are modeling Set 1: Introduction

Asynchronous Executions • An execution is admissible for the asynchronous model if • every message in an outbuf is eventually delivered • every processor takes an infinite number of steps • No constraints on when these events take place: arbitrary message delays and relative processor speeds are not ruled out • Models reliable system (no message is lost and no processor stops working) Set 1: Introduction

Example: Flooding • Describe a simple flooding algorithm as a collection of interacting state machines. • Each processor's local state consists of variable color, either red or green • Initially: • p0: color = green, all outbufs contain M • others: color = red, all outbufs empty • Transition: If M is in an inbuf and color = red, then change color to green and send M on all outbufs Set 1: Introduction

p0 p0 M M M M p2 p1 deliver event at p1from p0 p2 p1 computation event by p1 p0 p0 M M M M computation event by p2 deliver event at p2from p1 p2 p1 p2 p1 M M Example: Flooding Set 1: Introduction

p0 p0 M M M M M M p2 p1 deliver event at p1from p2 p2 p1 computation event by p1 M M p0 p0 M M M M M M etc. to deliver rest of msgs p2 p1 deliver event at p0from p1 p2 p1 Example: Flooding (cont'd) Set 1: Introduction

Nondeterminism • The previous execution is not the only admissible execution of the Flooding algorithm on that triangle. • There are several, depending on the order in which messages are delivered. • For instance, the message from p0 could arrive at p2 before the message from p1 does. Set 1: Introduction

Termination • For technical reasons, admissible executions are defined as infinite. • But often algorithms terminate. • To model algorithm termination, identify terminated states of processors: states which, once entered, are never left • Execution has terminated when all processors are terminated and no messages are in transit (in inbufs or outbufs) Set 1: Introduction

Termination of Flooding Algorithm • Define terminated processor states as those in which color = green. Set 1: Introduction

Message Complexity Measure • Message complexity: maximum number of messages sent in any admissible execution • This is a worst-case measure. • Later we will mention average-case measures. Set 1: Introduction

Message Complexity of Flooding Algorithm • Message complexity: one message is sent over each edge in each direction. So number is 2m, where m = number of edges. Set 1: Introduction

Time Complexity Measure • How can we measure time in asynchronous executions? • Produce a timed execution from an execution by assigning non-decreasing real times to events such that time between sending and receiving any message is at most 1. • Essentially normalizes the greatest message delay in an execution to be one time unit; still allows arbitrary interleavings of events. • Time complexity: maximum time until termination in any timed admissible execution. Set 1: Introduction

Time Complexity of Flooding Algorithm • Recall that terminated processor states are those in which color = green. • Time complexity: diameter + 1 time units. (A node turns green once a "chain" of messages has reached it from p0.) • Diameter of a graph is the maximum, over all nodes v and w in the graph of the shortest path from v to w Set 1: Introduction

Synchronous Message PassingSystems • An execution is admissible for the synchronous model if it is an infinite sequence of "rounds" • What is a "round"? • It is a sequence of deliver events that move all messages in transit into inbuf's, followed by a sequence of computation events, one for each processor. Set 1: Introduction

Synchronous Message Passing Systems • The new definition of admissible captures lockstep unison feature of synchronous model. • This definition also implies • every message sent is delivered • every processor takes an infinite number of steps. • Time is measured as number of rounds until termination. Set 1: Introduction

Example of Synchronous Model • Suppose flooding algorithm is executed in synchronous model on the triangle. • Round 1: • deliver M to p1 from p0 • deliver M to p2 from p0 • p0 does nothing (as it has no incoming messages) • p1 receives M, turns green and sends M to p0 and p1 • p2 receives M, turns green and sends M to p0 and p1 Set 1: Introduction

Example of Synchronous Model • Round 2: • deliver M to p0 from p1 • deliver M to p0 from p2 • deliver M to p1 from p2 • deliver M to p2 from p1 • p0 does nothing since its color variable is already green • p1 does nothing since its color variable is already green • p2 does nothing since its color variable is already green Set 1: Introduction

p0 p0 M M p2 p1 round 1 events p2 p1 round 2 events p0 M M p2 p1 M M Example of Synchronous Model Set 1: Introduction

Complexity of Synchronous Flooding Algorithm • Just consider executions that are admissible w.r.t. synchronous model (i.e., that satisfy the definition of synchronous model) • Time complexity is diameter + 1 • Message complexity is 2m • Same as for asynchronous case. • Not true for all algorithms though… Set 1: Introduction

DISTRIBUTED ALGORITHMS