ADVANCED ALGORITHM ANALYSIS LECTURE # 1 INTRODUCTION-H euristic Algorithms

1. ADVANCED ALGORITHM ANALYSISLECTURE # 1INTRODUCTION-Heuristic Algorithms

2. ADVANCED ALGORITHM ANALYSIS • 1.Introduction • 2.Basic concept of Heuristic algorithms. • 2.1.Example of Heuristic algorithms. • 2.2The Traveling Salesman Problem. • 2.3Traveling Salesman Problem algorithm with example.

3. WhereWe'reGoing • Learn general approaches to algorithm design • Divide and conquer • Greedy method • Dynamic Programming • Basic Search and Traversal Technique • Graph Theory • Linear Programming • Approximation Algorithm • NP Problem

4. WhereWe'reGoing • Examine methods of analyzing algorithm correctness and efficiency • Recursion equations • Lower bound techniques • O,Omega and Theta notations for best/worst/average case analysis • Decide whether some problems have no solution in reasonable time • List all permutations of n objects (takes n! steps) • Travelling salesman problem • Investigate memory usage as a different measure of efficiency

5. Algorithm. (webster.com) • A procedure for solving a mathematical problem (as of finding the greatest common divisor) in a finite number of steps that frequently involves repetition of an operation. • Broadly: a step-by-step procedure for solving a problem or accomplishing some end especially by a computer. "Great algorithms are the poetry of computation." Etymology: • "algos" = Greek word for pain. • "algor" = Latin word for to be cold. • Abu Ja’far al-Khwarizmi’s = 9th century Arab scholar. • his book "Al-Jabr wa-al-Muqabilah" evolved into today’s high school algebra text

6. Example • How to make a coffee with a coffee machine:1. Open the coffee machine2. Put water in it.3. Close it4. Open the coffee container (hope it's called like that)5. Fill it with finely ground coffee.6. Close it7. Turn the machine on8. Wait 3-4 minutes9. Open it 10. Pour the coffee.11. etc.

7. Imagine: A WorldWithNoAlgorithms • Fast arithmetic. • Cryptography. • Quick sort. • Databases. • Signal processing. • Huffman codes. • Data compression. • Network flow. • Routing Internet packets. • Linear programming. • Planning, decision-making.

8. What is an Algorithm?- Anany Levitin • An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

9. What is a ComputerAlgorithm? • A computer algorithm is • a detailed step-by-step method for • solving a problem • by using a computer.

10. Whichalgorithmisbetter? The algorithms are correct, but which is the best? Measure the running time (number of operations needed). Measure the amount of memory used. Note that the running time of the algorithms increase as the size of the input increases.

11. ImportanceofAnalyzeAlgorithm • Need to recognize limitations of various algorithms for solving a problem. • Need to understand relationship between problem size and running time. • When is a running program not good enough? • Need to learn how to analyze an algorithm's running time without coding it. • Need to learn techniques for writing more efficient code. • Need to recognize bottlenecks in code as well as which parts of code are easiest to optimize.

12. Whydoweanalyzeaboutthem? • understand their behavior, and (Job -- Selection, performance, modify)improve them. (Research) • Correctness • Does the input/output relation match algorithm requirement? • Amount of work done ( complexity) • Basic operations to do task • Amount of space used • Memory used • Simplicity, clarity • Verification and implementation. • Optimality • Is it impossible to do better?

13. Algorithm Classification • There are various ways to classify algorithms: • 1. Classification by implementation : • Recursion or iteration: • Logical: • Serial or parallel or distributed: • Deterministic or non-deterministic: • Exact or approximate: • 2.Classification by Design Paradigm : • Divide and conquer. • Dynamic programming. • The greedy method. • Linear programming. • Reduction. • Search and enumeration. • The probabilistic and heuristic paradigm.

14. Solution Methods • Try every possibility (n-1)! possibilities – • grows faster than exponentially • If it took 1 microsecond to calculate each possibility takes 10140 centuriesto calculate all possibilities when n = 100 • Optimising Methodsobtain guaranteed optimal solution, but can take a very, very, long time III. Heuristic Methodsobtain ‘good’ solutions ‘quickly’ by intuitive methods. No guarantee of optimality.

15. HeuristicAlgorithms • The term heuristic is used for algorithms which find solutions among all possible ones ,but they do not guarantee that the best will be found, therefore they may be considered as approximately and not accurate algorithms. • These algorithms, usually find a solution close to the best one and they find it fast and easily. • Sometimes these algorithms can be accurate, that is they actually find the best solution, but the algorithm is still called heuristic until this best solution is proven to be the best . • EXAMPLE: Heuristic algorithm for the Traveling Salesman Problem (T.S.P) .

16. Traveling Salesman Problem A Salesman wishes to travel around a given set of cities, and return to the beginning, covering the smallest total distance. Easy to State Difficult to Solve

17. Traveling Salesman Problem (T.S.P)  William Rowan Hamilton • This is one of the most known problems, and is often called as a difficult problem. • A salesman must visit n cities, passing through each city only once, beginning from one of them which is considered as his base, and returning to it. • The cost of the transportation among the cities (whichever combination possible) is given. • The program of the journey is requested, that is the order of visiting the cities in such a way that the cost is the minimum.

18. Traveling Salesman Problem • Let's number the cities from 1 to n, and • let city 1 be the city-base of the salesman. • Also let's assume that c(i, j) is the visiting cost from i to j. • There can be c(i, j)<>c(j, i). • Apparently all the possible solutions are (n-1)!. • Someone could probably determine them systematically, find the cost for each and every one of these solutions and finally keep the one with the minimum cost. • These require at least (n-1)! steps.

19. Traveling Salesman Problem • If for example there were 21 cities the steps required are (n-1)!=(21-1)!=20! steps. • If every step required a msecwe would need about 770 centuries of calculations. • Apparently, the exhausting examination of all possible solutions is out of the question. Since we are not aware of any other quick algorithm that finds a best solution we will use a heuristic algorithm.

20. Traveling Salesman Problem • According to this algorithm whenever the salesman is in town i he chooses as his next city, the city j for which the c(i,j) cost, is the minimum among all c(i,k) costs, where k are the pointers of the city the salesman has not visited yet. • There is also a simple rule just in case more than one cities give the minimum cost, for example in such a case the city with the smaller k will be chosen. • This is a greedy algorithm which selects in every step the cheapest visit and does not care whether this will lead to a wrong result or not.

21. Traveling Salesman Problem Algorithm T.S.P. algorithm: Input : Number of cities n and array of costs c(i,j) i, j=1,..n (We begin from city number 1)Output : Vector of cities and total cost. • (* starting values *) • C=0 • cost=0 • visits=0 • e=1 (*e=pointer of the visited city) • (* determination of round and cost) • for r=1 to n-1 do • choose of pointer j with • minimum=c(e,j)=min{c(e,k);visits(k)=0 and k=1,..,n} • cost=cost+ minimum • e=j • C(r)=j • end r-loop • C(n)=1 • cost=cost+ c(e,1)

22. Traveling Salesman Problem Algorithm • We can find situations in which the TSP algorithm don't give the best solution. • We can also succeed on improving the algorithm. • For example we can apply the algorithm t times for t different cities and keep the best round every time. • We can also unbend the greeding in such a way to reduce the algorithm problem ,that is there is no room for choosing cheep sides at the end of algorithm because the cheapest sides have been exhausted.

23. If there is no condition to return to the beginning. It can still be regarded as a TSP. Suppose we wish to go from A to B visiting all cities. A B

24. If there is no condition to return to the beginning. It can still beregarded as a TSP. Connect A and B to a ‘dummy’ city at zero distance (If no stipulation of start and finish cities connect all to dummy at zero distance) A B

25. If there is no condition to return to the beginning. It can still be regarded as a TSP. Create a TSP Tour around all cities A B

26. A route returning to the beginning is known as a Hamiltonian Circuit A route not returning to the beginning is known as a Hamiltonian Path Essentially the same class of problem

27. Hamilton Circuits • Euler circuit/path => Visit each edge once and only once • Hamilton circuit => Visit each vertex once and only once (except at the end, where it returns to the starting vertex) • Hamilton path => Visit each vertex once and only once • Difference: Edge (Euler)  Vertex (Hamilton)

28. A B E D C Examples of Hamilton circuits • Has many Hamilton circuits: • A, B, C, D, E, A • A, D, C, E, B, A • Has many Hamilton paths: • A, B, C, D, E • A, D, C, E, B • Has no Euler circuit, no Euler path => 4 vertices of odd degree Graph 1 Hamilton circuits can be shortened into a Hamilton path by removal of the last edge

29. Examples of Hamilton circuits • Has no Hamilton circuits: • What ever the starting point, we are going to have to pass through vertex E more than once to close the circuit. • Has many Hamilton paths: • A, B, E, C, D • C, D, E, A, B • Has Euler circuit => each vertex has even degree A B E D C Graph 2

30. Examples of Hamilton circuits F A B • Has many Hamilton circuits: • A, F, B, E, C, G, D, A • A, F, B, C, G, D, E, A • Has many Hamilton paths: • A, F, B, E, C, G, D • A, F, B, C, G, D, E • Has Euler circuit => Every vertex has even degree E D C G Graph 3

31. Examples of Hamilton circuits G F Has no Hamilton circuits: Has no Hamilton paths: Has no Euler circuit Has no Euler path => more than 2 vertices of odd degree A B E D C I H Graph 4

32. A B D C Complete graph • A graph with N vertices in which every pair of vertices is joined by exactly one edge is called the complete graph. • Total no. of edges = N(N-1)/2 In K4, each vertex has degree 3 and the number of edges = 4 (3)/2 = 6

33. The six Hamilton circuits of K4 A B D C Rows => 6 Hamilton circuits Cols=> same Hamilton circuit with different reference points Graph Reference point A Reference point B Reference point C Reference point D

34. Complete graph • The number of Hamilton circuits in a complete graph can be computed by using factorials. • The complete graph with N vertices has • N! (factorial of N) = 1x 2x3x4x … x(N-1)x N (N-1)! Hamilton circuits. • Example: The complete graph with 5 vertices has 4! = 1x2x3x4 = 24 Hamilton circuits

35. Factorial Which of the following is true? n! = n! x (n-1)! n! = n! + (n-1)! n! = n x (n-1)! n! = n + (n-1)!

36. Traveling Salesman Problem • Traveling Salesman Problem (TSP): Given a complete graph with nonnegative edge costs, Find a minimum cost cycle visiting every vertex exactly once. • Application (Example): Given a number of cities and the costs of traveling from any city to any other city, what is the cheapest round-trip route that visits each city exactly once and then returns to the starting city.

37. Applications of the TSP Routing around Cities Computer Wiring - connecting together computer components using minimum wire length Archaeological Seriation - ordering sites in time Genome Sequencing - arranging DNA fragments in sequence Job Sequencing - sequencing jobs in order to minimise total set-up time between jobs Wallpapering to Minimise Waste NB: First three applications generally symmetric Last three asymmetric

38. 3am-5am 10am-1pm 7am-8am 4pm-7pm 6pm-7pm 8am-10am 6am-9am 2pm-3pm Major Practical Extension of the TSP Vehicle Routing - Meet customers demands within given time windows using lorries of limited capacity Depot Much more difficult than TSP

39. History of TSP 1800’s Irish Mathematician, Sir William Rowan Hamilton 1930’s Studied by Mathematicians Menger, Whitney, Flood etc. 1954 Dantzig, Fulkerson, Johnson, 49 cities (capitals of USA states) problem solved • 64 Cities 1975 100 Cities 1977 120 Cities 1980 318 Cities 1987 666 Cities 1987 2392 Cities (Electronic Wiring Example) 1994 7397 Cities • 13509 Cities (all towns in the USA with population > 500) • 15112 Cities (towns in Germany) • 24978 Cities (places in Sweden) • But many smaller instances not yet solved (to proven optimality) But there are still many smaller instances which have not been solved.

40. Finding an Approximate Solution • Cheapest Link Algorithm • Edge with smallest weight is drawn first • Edge with second smallest weight is drawn in • Continue unless it closes a smaller circuit or three edges come out of one vertex • Finished once a complete Hamilton Circuit is drawn

41. Cheapest Link Algorithm

42. Cheapest Link Algorithm

43. Cheapest Link Algorithm

44. Cheapest Link Algorithm

45. Cheapest Link Algorithm

46. Finding an Approximate Solution • Nearest Neighbor Algorithm • Start at any given vertex • Travel to edge that yields smallest weight and has not been traveled through yet • Continue until we have a complete Hamilton circuit

47. Nearest Neighbor Algorithm

48. Nearest Neighbor Algorithm

49. Nearest Neighbor Algorithm

50. Nearest Neighbor Algorithm