ICS 353: Design and Analysis of Algorithms

King Fahd University of Petroleum & Minerals Information & Computer Science Department ICS 353: Design and Analysis of Algorithms Summer Semester 2011 - 2012 (113) 01: Dr. Wasfi G. Al-Khatib SUMT: 10:30-11:45am - 24:180

Basic Concepts in Algorithmic Analysis • Outline • Introduction • Time Complexity • Space Complexity • Optimal Algorithms • How to estimate the running time of an algorithm • Worst Case Analysis and Average Case Analysis • Amortized Analysis • Input Size and Problem Instance

M. Alsuwaiyel, Introduction to Algorithms: Design Techniques and Analysis, World Scientific Publishing Co., Inc. 1999. • Chapter 1 (All) • In particular, sections 1-3,6,8-14 will be discussed in class. Reading Assignment

What is an algorithm? • An algorithm is defined as a finite set of steps, each of which may require one or more operations and if carried out on a set of inputs, will produce one or more outputs after a finite amount of time. • Examples of Algorithms • Examples of computations that are not algorithms

Properties of Algorithms • Definiteness: It must be clear what should be done. • Effectiveness: Each step must be such that it can, at least in principle, be carried out by a person using pencil and paper in a finite amount of time. E.g. integer arithmetic. • An algorithm produces one or more outputs and may have zero or more externally supplied inputs. • Finiteness: Algorithms should terminate after a finite number of operations.

Our Objective • Find the most efficient algorithm for solving a particular problem. • In order to achieve the objective, we need to determine: • How can we find such algorithm? • What does it mean to be an efficient algorithm? • How can one tell that it is more efficient than other algorithms?

In the First Chapter • We will answer the following two questions • What does an efficient algorithm mean? • How can one tell that it is more efficient than other algorithms? based on some easy-to-understand searching and sorting algorithms that we may have seen earlier.

Searching Problem • Assume A is an array with n elements A[1], A[2], … A[n]. For a given element x, we must determine whether there is an index j; 1 ≤ j ≤ n, such that x = A[j] • Two algorithms, among others, address this problem • Linear Search • Binary Search

Linear Search Algorithm Algorithm: LINEARSEARCH Input: array A[1..n] of n elements and an element x. Output: j if x = A[j], 1 ≤ j ≤ n, and 0 otherwise. 1. for j 1 to ndo 2. if (x = A[j]) 3. return j; 4. return 0

Analyzing Linear Search • One way to measure efficiency is to count how many statements get executed before the algorithm terminates • One should keep an eye, though, on statements that are executed “repeatedly”. • What will be the number of “element” comparisons if x • First appears in the first element of A • First appears in the middle element of A • First appears in the last element of A • Doesn’t appear in A.

Analyzing Linear Search • One way to measure efficiency is to count how many statements get executed before the algorithm terminates • One should keep an eye, though, on statements that are executed “repeatedly”. • What will be the number of “element” comparisons if x • First appears in the first element of A 1 comparison • First appears in the middle element of A n/2 comparisons • First appears in the last element of A n comparisons • Doesn’t appear in A. n comparisons

Binary Search • We can do “better” than linear search if we knew that the elements of A are sorted, say in non-decreasing order. • The idea is that you can compare x to the middle element of A, say A[middle]. • If x < A[middle] then you know that x cannot be an element from A[middle+1], A[middle+2], …A[n]. Why? • If x > A[middle] then you know that x cannot be an element from A[1], A[2], …A[middle-1]. Why?

Binary Search Algorithm Algorithm: BINARYSEARCH Input: An array A[1..n] of n elements sorted in nondecreasing order and an element x. Output: j if x = A[j], 1 ≤ j ≤ n, and 0 otherwise. 1. low  1; high n; j  0 2. while (low ≤high) and (j = 0) 3. mid (low + high)/2 4. if x = A[mid] then j mid 5. else if x < A[mid] then high mid - 1 6. else low mid + 1 7. end while 8. return j

Worst Case Analysis of Binary Search • What to do: Find the maximum number of element comparisons • How to do: Assuming x is greater than the largest element in the array, the number of “element” comparisons is equal to the number of iterations of the while loop in steps 2-7. HOW? • How many elements of the input do we have in the • First iteration • Second iteration • Third iteration • … • ith iteration • The last iteration occurs when the size of input we have =

Worst Case Analysis of Binary Search • What to do: Find the maximum number of element comparisons • How to do: Assuming x is greater than the largest element in the array, the number of “element” comparisons is equal to the number of iterations of the while loop in steps 2-7. HOW? • How many elements of the input do we have in the • First iteration n elements • Second iteration n/2 [n is even] or (n-1)/2 elements [n is odd] • In both cases, n/2 and (n-1)/2 is equal to n/2 • Third iteration n/22 • … • ith iteration n/2i-1 • The last iteration occurs when the size of input we have = 1

Theorem • The number of comparisons performed by Algorithm BINARYSEARCH on a sorted array of size n is at most

Insertion Sort Algorithm: INSERTIONSORT Input: An array A[1..n] of n elements. Output: A[1..n] sorted in nondecreasing order. 1. for i  2 to n 2. x A[i] 3. j i - 1 4. while (j > 0) and (A[j] > x) 5. A[j + 1] A[j] 6. j j - 1 7. end while 8. A[j + 1] x 9. end for

Insertion Sort Example 5 2 9 8 4

Insertion Sort Example x=2 5 2 9 8 4 x=9 2 5 9 8 4 x=8 2 5 9 8 4 x=4 2 5 8 9 4 2 4 5 8 9

Analyzing Insertion Sort • The minimum number of element comparisons iswhich occurs when • The maximum number of element comparisons iswhich occurs when • The number of element assignments is

Analyzing Insertion Sort • The minimum number of element comparisons is n-1 which occurs when the array is sorted • The maximum number of element comparisons is(n-1)n/2 which occurs whenthe array is sorted in reverse order • The number of element assignments isthe number of element comparisons + n – 1.

Time Complexity • One way of measuring the performance of an algorithm is how fast it executes. The question is how to measure this “time”? • Is having a digital stop watch suitable?

Order of Growth • As measuring time is subjective to many factors, we look for a more “objective” measure, i.e. the number of operations • Since counting the exact number of operations is cumbersome, sometimes impossible, we can always focus our attention to asymptotic analysis, where constants and lower-order terms are ignored. • E.g. n3, 1000n3, and 10n3+10000n2+5n-1 are all “the same” • The reason we can do this is that we are always interested in comparing different algorithms for arbitrary large number of inputs.

Example Growth rate for some function

Example Growth rate for same previous functions showing larger input sizes

Running Times for Different Sizes of Inputs of Different Functions

Asymptotic Analysis: Big-oh (O()) • Definition: For T(n) a non-negatively valued function, T(n) is in the set O(f(n)) if there exist two positive constants c and n0 such that T(n) cf(n) for all nn0. • Usage: The algorithm is in O(n2) in [best, average, worst] case. • Meaning: For all data sets big enough (i.e., nn0), the algorithm always executes in less than or equal to cf(n) steps in [best, average, worst] case.

Big O() • O() notation indicates an upper bound. • Usually, we look for the tightest upper bound: • while T(n) = 3n2 is in O(n3), we prefer O(n2).

Big O() Examples • Example 1: Find c and n0 to show that T(n) = (n+2)/2 is in O(n) • Example 2: Find c and n0 to show that T(n)=c1n2+c2n is in O(n2), where c1 and c2 are greater than zero. • Example 3: T(n) = c. We say this is in O(1).

Asymptotic Analysis: Big-Omega (()) • Definition: For T(n) a non-negatively valued function, T(n) is in the set (g(n)) if there exist two positive constants c and n0 such that T(n) >= cg(n) for all nn0. • Meaning: For all data sets big enough (i.e., nn0), the algorithm always executes in more than or equal to cg(n) steps. • () notation indicates a lower bound.

() Example • Find c and n0 to show that T(n) = c1n2 + c2n is in (n2), where c1 and c2are greater than zero.

Asymptotic Analysis: Big Theta (()) • When O() and () meet, we indicate this by using () (big-Theta) notation. • Definition: An algorithm is said to be (h(n)) if it is in O(h(n)) and it is in (h(n)).

Example • Show that log(n!) is in (n log n).

Complexity Classes and small-oh (o()) • Using ()notation, one can divide the functions into different equivalence classes, where f(n) and g(n) belong to the same equivalence class if f(n) = (g(n)) • To show that two functions belong to different equivalence classes, the small-oh notation has been introduced • Definition: Let f(n) and g(n) be two functions from the set of natural numbers to the set of non-negative real numbers. f(n) is said to be in o(g(n)) if for every constant c > 0, there is a positive integer n0 such that f(n) < cg(n) for all n  n0.

Simplifying Rules • If f(n) is in O(g(n)) and g(n) is in O(h(n)), then f(n) is in O(h(n)) • If f(n) is in O(kg(n)) for any constant k > 0, then f(n) is in ……… • If f1(n) is in O(g1(n)) and f2(n) is in O(g2(n)), then (f1 + f2)(n) is in ……… • If f1(n) is in O(g1(n)) and f2(n) is in O(g2(n)) then f1(n)f2(n) is in ……… • You can safely “globally” replace O with  or  in the above, where the above rules will still hold.

Very Useful Simplifying Rule • Let f(n) and g(n) be be two functions from the set of natural numbers to the set of non-negative real numbers such that: • Then • if L <  then f(n) is in • if L > 0 then f(n) is in • if 0 < L <  then f(n) is in • if L = 0 then f(n) is in

Space Complexity • Space complexity refers to the number of memory cells needed to carry out the computational steps required in an algorithm excluding memory cells needed to hold the input. • Compare additional space needed to carry out SELECTIONSORT to that of BOTTOMUPSORT if we have an array with 2 million elements!

Examples • What is the space complexity for • Linear search • Binary search • Selection sort • Insertion sort • Merge (that merges two sorted lists) • Bottom up merge sort

Optimal Algorithms • If one can show that there is no algorithm that solves a certain problem in asymptotically less than that of a certain algorithm A, we call A an optimal algorithm.

Optimal Algorithms: Example A decision tree for sorting three elements Figure 12.1 page 338 from the textbook

Optimal Algorithms: Example • Consider the sorting problem of n distinct elements using element comparison-based sorting • Using the decision tree model, the number of possible solutions (leaf nodes) in the binary tree is equal to ................ • You have learnt earlier that a binary tree with n nodes has height of at least  log n  (Observation 3.3 page 111 of the textbook) • Hence, the length of the longest path in a decision tree for sorting n distinct elements is at least ............. • Therefore, • Insertion sort is • Selection sort is • Merge sort is • Quick sort is

Estimating the Running Time of an Algorithm • As mentioned earlier, we need to focus on counting those operations which represent, in general, the behavior of the algorithm • This is achieved by • Counting the frequency of basic operations. • Basic operation is an operation with highest frequency to within a constant factor among all other elementary operations • Recurrence Relations

Counting the Frequency of Basic Operations • Sometimes, it is easier to compute the frequency of an operation that is a good representative of the overall time complexity of the algorithm • For example, Algorithm MERGE. • Counting the number of iterations • The number of iterations in a while loop and/or a for loop is a good indication of the total number of operations

Represent the cost of the for loop in summation form. • The main idea is to make sure that we find an iterator that increase/decrease its values by 1. • For example, consider finding the number of times statements 1, 2 and 3 get executed below: Simple Complexity Analysis: Complex Loops for (int i = 1; i < n; i++) statement1; for (int i = 1; i <= n; i++) for (int j = 1; j <= n; j++) statement2; } for (int i = 1; i <= n; i++) for (int j = 1; j <= i; j++) statement3; }

Useful Summation Formulas

Example 1 sum = 0; for (j=1; j<=n; j++) for (i=1; i<=j; i++) sum++; for (k=0; k<n; k++) A[k] = k;

Example 2 for j := 1 to do sum[j] := 0; for i := 1 to j2 do sum[j] := sum[j] + i; end for; end for; return sum[1.. ];

Example 3 count := 0; for i := 1 to n do m :=  n/i  for j := 1 to m do count := count + 1 ; end for; end for; return count;

Examples 4 & 5 sum1 = 0; for (k=1; k<=n; k*=2) for (j=1; j<=n; j++) sum1++; sum2 = 0; for (k=1; k<=n; k*=2) for (j=1; j<=k; j++) sum2++;

Example 6 count := 0; while n >= 1 do for j := 1 to n do execute_algorithm_x; count := count + 1; end for n := n / 2; end while return count;

ICS 353: Design and Analysis of Algorithms