Analysis & Design of Algorithms (CSCE 321)

1. Analysis & Design of Algorithms(CSCE 321) Prof. Amr Goneid Department of Computer Science, AUC Part 4. Brute Force Algorithms Prof. Amr Goneid, AUC

2. Brute Force Algorithms Prof. Amr Goneid, AUC

3. Brute Force Algorithms • Brute Force Algorithms • Elementary Sorting Algorithms • Selection Sort • Bubble Sort • Insertion Sort • Brute Force Sequential Search • Brute Force String Matching • Closest Pair Problem • Exhaustive Search • Traveling Salesman Problem • 0/1 Knapsack Problem • Assignment problem Prof. Amr Goneid, AUC

4. 1. Brute Force Algorithms • Brute Force is a straightforward approach to solving a problem, usually directly based on the problem’s statement and definitions of the concepts involved • In many cases, Brute Force does not provide you a very efficient solution • Brute Force may be enough for moderate size problems with current computers…. Prof. Amr Goneid, AUC

5. 2. Elementary Sorting Algorithms • Sorting • Selection Sort • Bubble Sort • Insertion Sort Prof. Amr Goneid, AUC

6. Sorting • The Sorting Problem: Input: A sequence of keys {a1 , a2 , …, an} output: A permutation (re-ordering) of the input, {a’1 , a’2 , …, a’n} such that a’1 ≤ a’2 ≤ …≤ a’n • Sorting is the most fundamental algorithmic problem in computer science • Sorting can efficiently solve many problems • Different sorting algorithms have been developed, each of which rests on a particular idea. Prof. Amr Goneid, AUC

7. Some Applications of Sorting • Efficient Searching Ordering a set so as to permit efficient binary search. • Uniqueness Testing Test if the elements of a given collection of items are all distinct. • Deleting Duplicates Remove all but one copy of any repeated elements in a collection. Prof. Amr Goneid, AUC

8. Some Applications of Sorting • Closest Pair Given a set of n numbers, find the pair of numbers that have the smallest difference between them • Median/Selection Find the kth largest item in set S. Can be used to find the median of a collection. • Frequency Counting Find the most frequently occurring element in S, i.e., the mode. • Prioritizing Events Sorting the items according to the deadline date. Prof. Amr Goneid, AUC

9. Types of Sorts • In-Place Sort: • An in-place sort of an array occurs within the array and uses no external storage or other arrays • In-place sorts are more efficientin space utilization • Stable Sorts: • A sort is stable if it preserves the ordering of elements with the same key. • i.e. If elements e1 and e2 have the same key, and e1 appears earlier than e2 before sorting, then e1 is located before e2 after sorting. Prof. Amr Goneid, AUC

10. Sorting • What will count towards T(n): • Comparisons of array elements • Swaps or shifts of array elements • We compare sorting methods on the bases of: • The time complexity of the algorithm • If the algorithm is In-Place and Stable or not Prof. Amr Goneid, AUC

11. Sorting • We examine here 3 elementary sorting algorithms: • Selection Sort • Bubble Sort • Insertion Sort These have a complexity of O(n2) Prof. Amr Goneid, AUC

12. (a) Selection Sort • The general idea of the selection sort is that for each slot, find the element that belongs there • Assume elements to be in locations 0..n-1 • Let (i) be the start of a sub-array of at least 2 elements, i.e. i = 0 .. n-2 • for each i = 0 .. n-2 Find smallest element in sub-array a[i..n-1] Swap that element with that at the start of the sub-array (i.e. with a[i]). Prof. Amr Goneid, AUC

13. How it works 4 3 1 6 2 5 1 3 4 6 2 5 1 2 4 6 3 5 1 2 3 6 4 5 1 2 3 4 6 5 1 2 3 4 5 6 Prof. Amr Goneid, AUC

14. Demos http://www.cosc.canterbury.ac.nz/people/mukundan/dsal/SSort.html http://coderaptors.com/?SelectionSort Prof. Amr Goneid, AUC

15. Selection Sort Algorithm First, let us compute the cost of finding the location of the minimum element in a subarray a[s..e ] minimum (a,s,e) { m = s ; for j = s+1 to e if (a[j]  a[m]) m = j ; return m; } Hence, T(s,e) = e – s For an array of size (n) it will cost T(n) = n-1 comparisons Prof. Amr Goneid, AUC

16. Selection Sort Algorithm SelectSort (a[0..n-1 ]) { for i = 0 to n-2 { m = minimum (a , i , n-1) ; swap (a[i] , a[m]); } } Prof. Amr Goneid, AUC

17. Analysis of Selection Sort Exact algorithm Prof. Amr Goneid, AUC

18. Performance of Selection Sort • The complexity of the selection sort is  (n2), independent of the initial data ordering. • In-Place Sort Yes • Stable Algorithm No • This technique is satisfactory for small jobs, not efficient enough for large amounts of data. Prof. Amr Goneid, AUC

19. (b) Bubble Sort • The general idea is to compare adjacent elements and swap if necessary • Assume elements to be in locations 0..n-1 • Let (i) be the index of the last element in a sub-array of at least 2 elements, i.e. i = n-1 .. 1 Prof. Amr Goneid, AUC

20. Bubble Sort Algorithm • for each i = n-1…1 Compare adjacent elements aj and aj+1 , j = 0..i-1 and swap them if aj > aj+1 This will bubble the largest element in the sub-array a[0..i] to location (i). Prof. Amr Goneid, AUC

21. How it works 4 3 1 2 3 4 1 2 3 1 4 2 3 1 2 4 1 3 2 4 1 2 3 4 Prof. Amr Goneid, AUC

22. Demos http://www.cosc.canterbury.ac.nz/people/mukundan/dsal/BSort.html http://coderaptors.com/?BubbleSort Prof. Amr Goneid, AUC

23. Bubble Sort Algorithm BubbleSort (a[ 0..n-1]) { for i = n-1 down to 1 for j = 0 to i-1 if (a[j] > a[j+1] ) swap (a[j] , a[j+1]); } Prof. Amr Goneid, AUC

24. Analysis of Bubble Sort Worst case when the array is inversely sorted: Prof. Amr Goneid, AUC

25. Bubble Sort (a variant) void bubbleSort (itemType a[ ], int n) { int i , j; bool swapped; for (i = n-1; i >= 1; i--) { swapped = false; for (j = 0; j < i; j++ ) { if (a[j] > a[j+1] ) { swap(a[j] , a[j+1]); swapped = true; } } if (!swapped) return; } } Prof. Amr Goneid, AUC

26. Performance of Bubble Sort • The worst case complexity of the bubble sort is when the array is inversely sorted: O(n2). • The variant has a best case when the array is already sorted in ascending order:Ω(n). Outer loop works for only i = n-1 and there will be no swaps. • In-Place Sort Yes • Stable Algorithm Yes • For small jobs, not efficient enough for large amounts of data. Prof. Amr Goneid, AUC

27. (c) Insertion Sort • The general idea of the insertion sort is that for each element, find the slot where it belongs. • Performs successive scans through the data. When an element is out of sequence (less than its predecessor), it is pulled out and then inserted where it should belong. • Array elements have to be shifted to the right to make space for the insertion. Prof. Amr Goneid, AUC

28. How it works 2 7 1 5 8 6 3 4 1 2 7 5 8 6 3 4 1 2 5 7 8 6 3 4 1 2 5 6 7 8 3 4 1 2 3 5 6 7 8 4 1 2 3 4 5 6 7 8 Prof. Amr Goneid, AUC

29. Demos http://www.cosc.canterbury.ac.nz/people/mukundan/dsal/ISort.html http://coderaptors.com/?InsertionSort Prof. Amr Goneid, AUC

30. Insertion Sort Algorithm InsertSort (a[0..n-1]) {for i =1 to n-1 { j = pointer to element ai; v = copy of ai; while( j > 0 && aj-1 > v) { move data right:aj ← aj-1 move pointer left: j-- } Insert v at last (j) location: aj ← v; } } Prof. Amr Goneid, AUC

31. Analysis of Insertion Sort Worst case when the array is inversely sorted, while loop iterates (i) times: Prof. Amr Goneid, AUC

32. Analysis of Insertion Sort Best case when the array is already sorted in ascending order,while loop works zero times Prof. Amr Goneid, AUC

33. Performance of Insertion Sort • The worst case complexity of the insertion sort is when the array is inversely sorted: O(n2). • Best case when the array is already sorted in ascending order:Ω(n). While loop will not execute and there will be no data movement. • In-Place Sort Yes • Stable Algorithm Yes • For modest jobs, not efficient enough for large amounts of data. Prof. Amr Goneid, AUC

34. 3. Brute Force Sequential Search Find whether a search key is present in an array Worst case: Key not found. T(n) = O(n) comparisons Prof. Amr Goneid, AUC

35. 4. Brute Force String Matching • Pattern: a string of m characters to search for • Text: a (longer) string of n characters to search in • Problem: find a substring in the text that matches the pattern • Example: Pattern – ‘NOT’, text – ‘NOBODY_NOTICED_HIM’ • Typical Applications ‘find’ function in the text editor, e.g., MS-Word, Google search Prof. Amr Goneid, AUC

36. Brute Force String Matching Brute-force algorithm Step 1 Align pattern at beginning of text Step 2 Moving from left to right, compare each character of pattern to the corresponding character in text until • all characters are found to match (successful search); or • a mismatch is detected Step 3 While pattern is not found and the text is not yet exhausted, realign pattern one position to the right and repeat Step 2 Prof. Amr Goneid, AUC

37. Brute Force String Matching T[0..n-1]: Text string P[0..m-1]: Pattern string Show that the worst case number of comparisons is O(mn) Prof. Amr Goneid, AUC

38. 5. Closest Pair Problem Problem: Find the two closest points in a set of n points (in the two dimensional Cartesian plane). Brute-force algorithm • Compute the distance between every pair of distinct points • Return the indexes of the points for which the distance is the smallest. Prof. Amr Goneid, AUC

39. Closest Pair Problem Prof. Amr Goneid, AUC

40. Closest Pair Problem • The Hamming distance between two strings of equal length is defined as the number of positions at which the corresponding symbols are different. It is named after Richard Hamming (1915–1998), a prominent American scientist. • Suppose that the points in the problem of closest pairs are strings and hence the distance between two of such strings is now measured by the Hamming Distance. Write an algorithm to find the closest pair of strings in an array of strings and find the number of comparisons done by this algorithm. Prof. Amr Goneid, AUC

41. 6. Exhaustive Search • A brute-force approach to combinatorial problem • Generate each and every element of the problem’s domain • Then compare and select the desirable element that satisfies the set constraints • Involve combinatorial objects such as permutations, combinations, and subsets of a given set • The time efficiency is usually bad – usually the complexity grows exponentially with the input size • Three examples • Traveling salesman problem • Knapsack problem • Assignment problem Prof. Amr Goneid, AUC

42. 2 a b 5 3 4 8 c d 7 (a)Traveling Salesman Problem (TSP) • Given n cities with known distances between each pair, find the shortest tour that passes through all the cities exactly once before returning to the starting city • Alternatively: Find shortest Hamiltonian circuitin a weighted connected graph (The circuit problem is named after Sir William Rowan Hamilton) • Example: Prof. Amr Goneid, AUC

43. Traveling Salesman Problem (TSP) Hamiltonian Circuit Cost a→b→c→d→a 2+3+7+5 = 17 Optimal a→b→d→c→a 2+4+7+8 = 21 a→c→b→d→a 8+3+4+5 = 20 a→c→d→b→a 8+7+4+2 = 21 a→d→b→c→a 5+4+3+8 = 20 a→d→c→b→a 5+7+3+2 = 17 Optimal Number of candidate circuits = (n-1)! Very High complexity O(n!) Prof. Amr Goneid, AUC

44. (b) 0/1 Knapsack Problem Given n items: • weights: w1 w2 … wn • values: v1 v2 … vn • a knapsack of capacity W Find most valuable subset of the items that fit into the knapsack Example: Knapsack capacity W=16 item weight value ($) • 2 20 • 5 30 • 10 50 • 5 10 Prof. Amr Goneid, AUC

45. 0/1 Knapsack Problem SubsetTotal weightTotal value { } 0 $00 {1} 2 $20 {2} 5 $30 {3} 10 $50 {4} 5 $10 {1,2} 7 $50 {1,3} 12 $70 {1,4} 7 $30 {2,3} 15 $80 Optimal {2,4} 10 $40 {3,4} 15 $60 {1,2,3} 17 not feasible {1,2,4} 12 $60 {1,3,4} 17 not feasible {2,3,4} 20 not feasible {1,2,3,4} 22 not feasible Generates 2n possible subsets, T(n) = O(2n) Prof. Amr Goneid, AUC

46. (c) Assignment Problem by Exhaustive Search There are n people who need to be assigned to n jobs, one person per job. The cost of assigning person ito job j is C[i,j]. Find an assignment that minimizes the total cost. Job 1 Job 2 Job 3 Job 4 Person 1 9 2 7 8 Person 2 6 4 3 7 Person 3 5 8 1 8 Person 4 7 6 9 4 Algorithmic Plan: Generate all legitimate assignments, compute their costs, and select the cheapest one. How many assignments are there? n! Pose the problem as one about a cost matrix: Prof. Amr Goneid, AUC

47. Assignment Problem by Exhaustive Search Assignment (col.#s) Total Cost 1, 2, 3, 4 9+4+1+4=18 1, 2, 4, 3 9+4+8+9=30 1, 3, 2, 4 9+3+8+4=24 1, 3, 4, 2 9+3+8+6=26 1, 4, 2, 3 9+7+8+9=33 1, 4, 3, 2 9+7+1+6=23 etc. T(n) = # of assignments = O(n!) Prof. Amr Goneid, AUC