Algorithm Design and Analysis (ADA)

1. Algorithm Design and Analysis (ADA) 242-535, Semester 1 2014-2015 • Objective • we describe three algorithms for finding the shortest path tree through a graph: • Dijkstra, Bellman-Ford, and topological sort 11. Shortest Paths

2. Overview • Problem Overview • Dijkstra’s Algorithm • Bellman-Ford Algorithm • DAGs and Topological Sort • Summary of Algorithms

3. 1. Problem Overview • Edge weights may represent distances, costs, etc. • Example: • the distances between airports 849 PVD 1843 ORD 142 SFO 802 LGA 1205 1743 337 1387 HNL 2555 1099 1233 LAX 1120 DFW MIA

4. Shortest Paths • The length of a path = the sum of the weights of the edges in the path. • The shortest path between two verticies = the path having the minimum length. • Example: • shortest path between Providence and Honolulu 849 PVD 1843 ORD 142 SFO 802 LGA 1205 1743 337 1387 HNL 2555 1099 1233 LAX 1120 DFW MIA Graphs

5. Typical Applications the fast buying and selling of financial items (e.g. foreign exchange) in different markets to profit from price differences

6. Arbitrage Graph A chance to make money.

7. Shortest Path Properties Property 1 A subpath of a shortest path is itself a shortest path This means that shortest path has an optimal structure. Property 2 There is a shortest pathstree (SPT) from a start vertex to all the other vertices (e.g. see below with Providence as the start). 849 PVD 1843 ORD 142 SFO 802 LGA 1205 1743 337 1387 HNL 2555 1099 1233 LAX 1120 DFW MIA Graphs

8. Another SPT • A shortest path tree is a tree that spans all the nodes in the graph, using edges that give the shortest paths from the starting node to all the other vertices. start node is 0 SPT

9. 2. Dijkstra’s Algorithm • Build a shortest path tree (SPT) • No negative edge weights allowed. • Due to Edsger W. Dijkstra (1959, when 29). • Similar to Prim's algorithm • Grow a tree gradually, advancing from vertices taken from a queue • Dijkstra is a greedy algorithm like Prim's • Instead of a MST, we are building a SPT

10. Algorithm in Words • Each vertex has a entry in a distTo[] array. • Initialize dist[start] to 0 and all other distTo[] entries to positive infinity () • During the execution, the distTo[] values will get smaller. This is called (edge) relaxing. • At each iteration, the SPT is grown by adding a vertex with the smallest distTo[] value.

11. Algorithm Graphically • The growing SPT is drawn with black edges • The crossing edge with the shortest path is chosen by looking at the distTo[] values • e.g. distTo[w] will be the smallest which is not already in the SPT

12. What is Relaxing? b b 5 5 distTo[b] = 13 distTo[b] = 8 a a 4 4 c c distTo[a] = 3 distTo[a] = 3 relax 'edges' linked to a distTo[c] = 13 distTo[c] = 7 a is in the SPT 8 8 d d distTo[d] = 10 distTo[d] = 10 c will be the next vertex to join SPT if (distTo[b] > distTo[a] + b.weight()) distTo[b] = distTo[a] + b.weight();

13. SPT Data Structures • distTo[v] is the length of the shortest known path from the start node to vertex v. • distTo[] values will get smaller (be relaxed) • edgeTo[v]contains the edge that connects v to its parent in the SPT • this is used to reconstruct the shortest path at the end of the algorithm • a priority queuekeeps track of vertices that are candidates for being added to the SPT

14. Data Structures in Use SPT from 0

15. Note that we dont need to maintain a tree data structure for the SPT. • Instead the edgeTo[] array is used to reconstruct the paths through the SPT.

16. Building the Path 0 --> 6

17. Code DirectedEdge[] edgeTo; double[] distTo; MinPriQueue<Double, Double> pq; /* this is a priority queue where each element's position is based on a separate value, which is not its vertex number */ 3 0 4 2 3 17 23 61 add() remove() change() value

18. void dijkstra(Digraph graph, int start) { for (int v : graph.allVerts()) distTo[v] = Double.POSITIVE_INFINITY; distTo[start] = 0; pq.add(start, distTo[start]); // add start node to queue using distTo[] as value while (!pq.isEmpty()) { int small = pq.remove(); // small now 'in' SPT relax(G, small); } }

19. relax all edges adjacent to u void relax(Digraph graph, int u) { for(DirectedEdge e : graph.adj(u)) { int v = e.to(); // e is u --> v edge if (distTo[v] > distTo[u] + e.weight()) { distTo[v] = distTo[u] + e.weight(); edgeTo[v] = e; // e is used in SPT to goto v if (pq.contains(v)) // v already in queue? pq.change(v, distTo[v]); // change v's position in the queue // based on the changed distTo[] value else pq.add(v, distTo[v]); } } }

20. Example in Detail Start node is 0 removed(so 'in' SPT)

22. queue is empty Finished

23. Notes • On each iteration of the while-loop: • one vertex is removed from the queue, which means that it becomes part of the SPT • the selected vertex changes its adjacent verticies’ distTo[]if they can be made smaller • thr adjacent edges are relaxed • edgeTo[] changes as well • The algorithm eventually considers every vertex.

24. Proof of the Algorithm • Inductive statement S(i) • on the ith iteration of the while-loop, the v node is removed from the queue (and so added to the SPT) • distTo[v], is the shortest distance from the start node to v Is this statement true? continued

25. Basis (i = 1). • initially distTo[start] == 0, and all other distTo[]s are Infinity • so the while-loop selection will be the start node • distTo(start) is the shortest path from the start node to the start node; it is 0 • S(1) is true. continued

26. Induction Case. • Prove that S(k)  S(k+1) • Assume that S(k) • On the k+1 iteration, the while-loop removes node v from the queue • Is distTo[v] the shortest path from start node to v?

27. Diagram of Induction Case Is distTo[v] the shortestpath fromstart node to v ? still in the queue removed from queue (so now in SPT) start node choose v because of dist[v] x v w u

28. Proof by Contradiction • Assume that distTo[v] is not the shortest path from the start node to v. • That means there must be another path to v:start -> ... -> w -> u -> ... -> v which is shorter. something not already selected from the queue continued

29. This means that: path_length(a -> ... -> w -> u -> ... -> v) < path_length(a -> ... -> v) • which means that: path_length(a -> ... -> w -> u) < path_length(a -> ... -> v) • which means that:distTo[u] < distTo[v] continued

30. But, if distTo[u] < distTo[v] then u should have been selected instead of v on this iteration. • Since v was chosen, it means that:distTo[u] ≥ distTo[v], for all u • So distTo[v]is the shortest path. continued

31. The induction case was: • On the k+1 iteration, select node v. Is distTo[v] the shortest path from start node to v? • The answer is yes. • So, S(k+1) is true. • Since the basis and induction cases are true, S(i) is true for all i.

32. Running Time • Dijkstra's original algorithm did not use a minimum priority queue and so ran in O(|V|2) • it used an array to store the vertices • The implementation based on a minimum priority queue coded as a minimum heap runs in O(|E| * log |V|)

33. Why O(E log V)? in a dense graph E ≈ V2, so is bigger than V O(E*log V) dijkstra look at every vertex O(V) for while O(E*log V) look at all edges = E iters relax() for look at every edge from a vertex queue ops O(log V) due to heap impl of the queue the queue contains at most V vertices

34. Why O(log V)? PQ Operation Array Binary heap d-ary Heap Fib heap † remove() V log V log V log V change() 1 log V log V 1 Total V2 E log V E log E/V V V log V + E † Individual ops are amortized analysis minimum binary heap Fibonacci Heap (multiple trees)

35. Why Doesn’t Dijkstra Work for Negative Weight Edges? • Adding a negative edge can change the distTo[] values for vertices already in the SPT. • This breaks the algorithm since distTo[] is assumed to be the smallest when its node is selected for SPT 0 A 4 SPT vertices 5 1 4 C D -8 5 F 9 distTo[c] will change from 5 to 4+5-8 = 1

36. 3. Bellman-Ford Algorithm • Unlike Dijkstra's algorithm, the Bellman-Ford algorithm can be used on graphs with negative edge weights, as long as the graph contains no negative cycles reachable from the source vertex s. • The presence of such cycles means there is no shortest path, since the total weight becomes lower each time the cycle is traversed. • Drawback: slower than Dijkstra's algorithm for the same problem • O(|V| * |E|)

37. Negative Cycles Problem • In graphs with negative weight cycles, some shortest paths will not exist: -4 What is distTo[x] starting from s? w v 2 1 s t u x 2 1 1

38. Example This example can be processed by Bellman-Ford since there are no negative cycles. 29 32 28 39 52 34 28 35 35 26 37 -120 38 -140 -125

39. SPT Using Bellman-Ford 32 39 34 52 35 26 -125 distTo[4] = 26 + 34 + 39 + 52 – 125 = 26

40. Algorithm in Words • Bellman-Ford is similar to Dijkstra's Algorithm, but instead of greedily selecting the minimum weight vertex not yet in the SPT, it relaxes all the edges|V | − 1 times. • The repetitions reduce the distTo[] values to their smallest throughout the graph. Why? • The SPT visits every node in the graph at most once, so the biggest path in the SPT is at most |V|-1 edges long. So |V|-1 loops is certain to process it all. • assuming there are no negative cycles

41. Prove: after |V|-1 passes, all distTo[] values correct • Consider shortest path from s to v:s  v1  v2  v3  v4  v • Initially, distTo[s] = 0 is correct, and doesn’t change. • After 1 pass through edges, distTo[v1] is correct and doesn’t change • After 2 passes, distTo[v2] is correct and doesn’t change • … • Terminates in |V| - 1 passes (if no negative cycles).

42. Code DirectedEdge[] edgeTo; double[] distTo; void bellmanFord(Digraph graph, int start) { for (int v : graph.allVerts()) distTo[v] = Double.POSITIVE_INFINITY; distTo[start] = 0; for (int i=1; i < graph.numV(); i++) relax(graph); for (DirectedEdge e : graph.allEdges()) { int u = e.from(); // edge e is u --> v int v = e.to(); if (distTo[v] > distTo[u] + e.weight()) println("There's no solution"); } } Initialize distTo[], which will reduce to shortest-path value Relaxation: Make |V|-1 passes, relaxing every edge Test for solutionWhendo we not get a solution?

43. void relax(Digraph graph) // a simpler version than Dijkstra { for (DirectedEdge e : graph.allEdges()) { int u = e.from(); // edge e is u --> v int v = e.to(); if (distTo[v] > distTo[u] + e.weight()) { distTo[v] = distTo[u] + e.weight(); edgeTo[v] = e; } } }

44. An Example in Detail The start node is s 5 distTo[] edgeTo[] t x s t x y z 6 - - - - - 0     -2 -3 8 -4 s 7 2 7 y z 9 after each relax for-loop iteration 5 distTo[] edgeTo[] t x s t x y z 6 - s->t - s->y - 0 6  7  -2 -3 8 -4 s 7 2 7 y z 9

45. 5 distTo[] edgeTo[] t x s t x y z 6 - s->t y->x s->y t->z 0 6 4 7 2 -2 -3 8 -4 s 7 2 7 y z 9 5 t x s t x y z 6 - x->t y->x s->y t->z 0 2 4 7 2 -2 -3 8 -4 s 7 2 7 y z 9 t x s t x y z 6 - x->t y->x s->y t->z 0 2 4 7 -2 -2 -3 8 -4 s 7 2 7 y z 9 Finished

46. Running Time • The second for-loop executes |V|-1 times, and processes every edge (in the worst case) inside the body. • Worse case running time is O(|V|*|E|) • Note that order in which edges are processed affects how quickly the algorithm finishes

47. 4. DAGs and Topological Sort • In many applications, edge-weighted digraphs have no directed cycles: they are DAGs • Directed Acyclic Graphs • We can modify a topological sort to find the SPT in a DAG • simpler and faster than Dijskstra • linear running time: O(E + V) • handles negative edge weights • can be easily extended to solve related problems, such as finding all the longest paths

48. DAG Shortest Paths • A topological sort puts the vertices in an order where all the vertices which "affect / go to" a vertex are before it. • This is the ordering uses by edge relaxing to calculate the smallest distTo[] values. • This means that relaxing need only pass through the sorted graph once to reduce all the distTo[] values to their smallest values • the running time is linear

49. Example of a DAG 29 32 39 28 34 52 35 37 26 38 40 58 93 Negative weights are allowed, but there aren't any in this example.

50. Topological Sort of the Example 5 1 3 6 4 7 0 2 • A topological sort of a DAG is a linear ordering of its vertices such that for every directed edge u  v,u comes before v in the ordering. • Any DAG has at least one topological sort (and often many).