Algorithms and data structures: basic definitions

1. Algorithms and data structures: basic definitions An algorithm is a precise set of instructions for solving a particular task. A data structure is any data type (or representation) with its associated operations. Example data structures: • primitive data types (such as int, double, char) are data structures, because they have built-in algorithms for comparison, arithmetic, etc. • More typical data structures are meant to organize and structure collections of data items. A sorted list of integers stored in an array is an example of such a data structure. Classes in JAVA, where data items are defined by means of instance variables, and associated operations are implemented by class methods, is another example. In many cases, the same operation can be carried out in different ways, by means of different algorithms. One of the most important tasks for the program designer at the initial stage of software development is to identify the most appropriate algorithm for any operation associated with the DS.

2. Introduction to algorithm analysis Algorithms can be compared and evaluated based on different criteria depending on the purposes of the analysis. Among them are: • Execution (or running) time. • Space (or memory) needed. • Correctness. • Clarity. • Etc. In the majority of cases, the execution time and correctness are the most important criteria upon which a decision is made about how good or bad (with respect to that particular case) an algorithm is. This is why, we must know how to analyze and classify the execution time of an algorithm, and how to demonstrate its correctness.

3. How algorithm correctness relates to the quality of software implementation? Fundamental implementation goals for any software are: • Robustness. The program must generate the correct output for any possible input including those not explicitly defined. • Adaptability. The program needs to be able to evolve over time to reflect changes in hardware and software environments. • Reusability. The same code should be usable in different applications. In object-oriented programming there are three more goals in addition to these: • Abstraction, i.e. identifying certain properties (both, procedural and declarative) of an object and then using them to specify a new object which represents a more detailed embodiment of the original one. Example: a JAVA interface specifies what an object does, but not how it does it, while the class implementing it handles the “how” part. • Encapsulation, i.e. software components should implement an abstraction without revealing its internal details. • Modularity, i.e. dividing the program into separate functional components (classes).

4. What affects the execution time of an algorithm? Execution time depends upon: • the size of the input (the number of steps performed for different inputs is different); • computer characteristics (mostly processor speed); • implementation details (programming language, compiler, etc.). Taking these characteristics into account makes it very hard to define how efficient a given algorithm is in general. Therefore, we want to ignore all machine- and problem-dependent considerations in our analysis, and focus on the analysis of the algorithm’s structure. The first step in this analysis is to identify a a small number of operations that are executed most often and thus affect the execution time the most.

5. Example 1: In the following program, which operation affects the run time of the program the most? class lec1ex1 { public static void main (String [] args) { int[][] table = {{1, 2, 3, 4}, {5, 6, 7, 8}, {9, 10, 11, 12}}; int[] sum = new int[3]; int total_sum = 0; for (int i = 0; i < table.length; i++) { sum[i] = 0; //compute the sum of all entries of a given row as well as the total sum of all entries for (int j = 0; j < table[i].length; j++) { sum[i] = sum[i] + table[i][j]; total_sum = total_sum + table[i][j]; } System.out.println ("The sum of the entries of row " + i + " is " + sum[i]); } System.out.println ("The total sum of entries is " + total_sum); } }

6. Example 1 (cont.): Consider the following change in the above algorithm .... for (int i = 0; i < table.length; i++) { sum[i] = 0; //compute the sum of all entries of a given row as well as the total sum of all entries for (int j = 0; j < table[i].length; j++) { sum[i] = sum[i] + table[i][j]; } System.out.println ("The sum of the entries of row " + i + " is " + sum[i]); total_sum = total_sum + sum[i]; .... Question 1: How this change affects the run time of the algorithm? Question 2: How significant is the difference?

7. public static boolean linearSearch (int[] list, int target) { boolean result = false; for (int i = 0; i < list.length; i++) if (list[i] == target) result = true; return result; } public static boolean binarySearch (int[] list, int target) { boolean result = false; int low = 0, high = list.length - 1, middle; while (low <= high) { middle = (low + high) / 2; if (list[middle] == target) { result = true; return result; } else if (list[middle] < target) low = middle + 1; else high = middle - 1; } return result; } Example 2: Compare the run times of the following two algorithms (performing linear and binary search in an array of integers)

8. That is, the run time of an algorithm can be determined by analyzing its structure and counting the number of operations affecting its performance. Mathematically, this can be expressed by the polynomial C0 + C1*f1(N) + C2*f2(N) + ... + Cn*fn(N) Typically, one of the terms of this polynomial is much bigger than the other terms. This term is called the leading term, and it defines the run time; Ci is called a constant of proportionality, and in most cases it can be ignored. In general, the run time behavior of an algorithm is dominated by its behavior in the loops. Therefore, by analyzing the loop structure we can define the number and the type of operations that affect algorithm performance the most.

9. A majority of algorithms have a run time proportional to one of the following functions (defined by the leading term with the constant of proportionality ignored): 1 All instructions are executed only once or at most several times. In this case, we say that the algorithm has a constant execution time. logN If the algorithm solves the original problem by transforming it into a smaller problem by cutting the size of the input by some constant fraction, then the program gets slightly slower if N grows. In this case we say that the algorithm has a logarithmic execution time. N If a small amount of processing is done on each input element, we say that the algorithm has a linear execution time. NlogN If the algorithm solves the original problem by breaking it into sub-problems which can be solved independently, and then combines those solutions to get the solution of the original problem, its execution time is said to be NlogN. N^2 If the algorithm processes all input data in a double nested loop, it is said to have a quadratic execution time. N^3 If the algorithm processes all input data in a triple nested loop, it is said to have a cubic execution time. 2^N If the execution time squares when the input size doubles, we say that the algorithm has an exponential execution time.

10. version 1 for (int i = 0; i < table.length; i++) { sum[i] = 0; for (int j = 0; j < table[i].length; j++) { sum[i] = sum[i] + table[i][j]; total_sum = total_sum + table[i][j]; } Number of additions: 2 * (i ^ 2) Algorithm efficiency: N^2 version 2 for (int i = 0; i < table.length; i++) { sum[i] = 0; for (int j = 0; j < table[i].length; j++) sum[i] = sum[i] + table[i][j]; total_sum = total_sum + sum[i]; } Number of additions: (i ^ 2) + i Algorithm efficiency: N^2 Example 1 (cont.) Define and compare the run times of the two versions of the “sum problem”:

11. version 1 for (int i = 0; i < list.length; i++) if (list[i] == target) result = true; Number of comparisons: i Algorithm efficiency: N version 2 while (low <= high) { middle = (low + high) / 2; if (list[middle] == target) { result = true; return result; } else if (list[middle] < target) low = middle + 1; else high = middle - 1; } Number of comparisons: log list.length Algorithm efficiency: log N Example 2 (cont.) Define and compare the run times of the two versions of the “search problem”:

12. Average case and worst case analysis In the search problem, it will take at mostN or log N (for linear and binary search, respectively) steps to find the target or to show that the target is not on the list. These cases are the worst cases and most often we want to know algorithm efficiency in exactly this case; this is called the worst case run time efficiency. In most cases, it will take less than N (or log N) steps for the algorithm to find the solution (it may even take just one step in the best case). How much “less”, however, is often difficult to determine. The average run time of an algorithm can only be an estimate, because it depends on the input. This is why it is a less important characteristic of algorithm efficiency.

13. The big-O notation To more precisely express the run time efficiency of an algorithm, we use the so-called big-O notation which is defined as follows: Definition A function g(N) is said to be O(f(N)) is there exist constants C0 and N0 such that g(N) < C0f(N0) for all N > N0. Consider the summing problem. It takes (N^2 + N) steps for version 2 to find the two sums. Here, g(N) = N^2 + N < N^2 + N^2 = 2 * N^2. Let C0 = 2. Therefore, for both versions the run time of an algorithm is O(f(N^2)). The goal of the efficiency analysis is to show that the running time of an algorithm under consideration is O(f(N)) for some f.

14. Notes on big-O notation 1. The statement that the running time of an algorithm is O(f(N)) does not mean that the algorithm ever takes that long. 2. The input that causes the worst case may be unlikely to occur in practice. 3. Almost always the constants C0 and N0 are unknown and need not be small. These constants may hide implementation details which are important in practice. 4. For small N, there is usually a little difference in the performance of different algorithms. 5. The constant of proportionality, C0, makes a difference only for comparing algorithms with the same O(f(N)).

15. To illustrate these notes, consider the following actual algorithms and their efficiencies: Algorithm # 1 2 3 4 5 Run time efficiency 33N 46Nlog N 13N^2 3N^3 2^n Actual run time for the following input sizes (in sec., stated otherwise) N = 10 0.00033 0.0015 0.0013 0.0034 0.001 N = 100 0.003 0.03 0.13 3.4 4*10^14 centuries N = 1000 0.033 0.45 13 0.94 hours N = 10000 0.33 6.1 22 min 39 days N = 100000 3.3 1.3 min 1.5 days 108 years

16. Efficiency of recursive algorithms Example 3: Consider the following recursive version on the binary search algorithm. public static boolean binarySearchR (int[] list, int target, int low, int high) { int middle = (low + high) / 2; if (list[middle] == target) return true; else if (low > high) return false; else if (list[middle] < target) return binarySearchR (list, target, middle+1, high); else return binarySearchR (list, target, low, middle-1); }

17. Efficiency of recursive algorithms (contd.) Two factors define the efficiency of a recursive algorithm: • The number of levels to which recursive calls are made before reaching the condition which triggers the return. • The amount of space and time consumed at any given recursive level. The number of levels can be explicated by a tree of recursive calls. For the binary search example, we have the following tree of recursive calls (assume a list with 15 elements): low = 0 LEVEL high = 14 0 low = 0 OR low = 8 1 high = 6 high = 14 low = 0 O R low = 4 OR low = 8 OR low = 12 2 high = 2 high = 6 high = 10 high = 14 OR OR OR OR low = 0 low = 2 low = 4 low = 6 low = 8 low = 10 low = 12 low = 14 3 hight = 0 high = 2 high = 4 high = 6 high = 8 high = 10 high = 12 high = 14

18. Efficiency of recursive binary search (contd.) The efficiency of recursive binary search is: - At each level, the work done is O(1); - The overall efficiency is proportional to the number of levels, i.e. O(log n + 1). Assume that always middle = low. The tree of recursive calls becomes: low = 0 high = 14 low = 0 high = 14 N levels, i.e. O(N) eff. low = 0 high = 14 . . . low = 0 high = 14

19. Efficiency of recursive binary search (contd.) An alternative way to define the efficiency of a recursive algorithm is by means of the so-called recurrence relations. A recurrence relation is an equation that expresses the time or space efficiency of an algorithm for data set of size N in terms of the efficiency of the algorithm on a smaller data set. For recursive binary search, the recurrence relation is: CN = C(N/2) + 1 for N >= 2 with C1 = 0 To define the efficiency, we have to solve this relation. Assume N = 2n. Then, C(2^n) = C(2^(n-1)) + 1 = C(2^(n-2)) + 1 + 1 = C(2^(n -3)) + 1 + 1 + 1 = ... ... = C(2^1) + (n - 1) = C(2^0) + n = 0 + n = log N

20. Efficiency of recursive binary search (contd.) The recurrence relation for binary search with middle = low is: CN = C(N-1) + 1 for N >= 2 with C1 = 1 To define the efficiency, we have to solve this relation. CN = CN-1 + 1 = CN-2 + 1 + 1 = CN-3 + 1 + 1 + 1 = ... ... = C1+ (N - 1) = 1 + N - 1 = N

21. Efficiency of recursive algorithms (contd.) Consider the “tower of Hanoi” algorithm: public static void towerOfHanoi (int numberOfDisks, char from, char temp, char to) { if (numberOfDisks == 1) System.out.println ("Disk 1 moved from " + from + " to " + to); else { towerOfHanoi (numberOfDisks-1, from, to, temp); System.out.println ("Disk " + numberOfDisks + " moved from " + from + " to " + to); towerOfHanoi (numberOfDisks-1, temp, from, to); } }

22. Efficiency of the “tower of Hanoi” algorithm (contd.) Notice that two new recursive calls are initiated at each step. This suggests an exponential efficiency, i.e. O(2N). This result is obvious from the tree of recursive calls, which for four disks is the following: N = 4 N = 3 AND N = 3 N = 2 AND N = 2 N = 2 AND N = 2 N = 1 N = 1 N = 1 N = 1 N = 1 N = 1 N = 1 N =1

23. Efficiency of the “tower of Hanoi” algorithm (contd.) The recurrence relation describing this algorithm is the following: CN = 2 * C(N-1) + 1 for N >= 1 with C1 = 1 The solution of this relation gives the efficiency of the “tower of Hanoi” algorithm. CN = 2 * CN-1 + 1 = 2 * (2 * CN-2 + 1) + 1 = 22 * CN-2 + 21 + 20 = = 22 * (2 * CN-3 + 1) + 21 + 20 = 23 * CN-3 + 22 + 21 + 20 = = 23 * (2 * CN-4 + 1) + 22 + 21 + 20 = 24 * CN-4 + 23 + 22 + 21 + 20 = ... = 2(N-1) * C1+ 2(N-2) + 2(N-3) + 22 + 21 + 20 = = 2(N-1) + 2(N-2) + 2(N-3) + 22 + 21 + 20 = 2N- 1

24. A note on space efficiency The amount of space used by a program, like the number of seconds, depends on a particular implementation. However, some general analysis of space needed for a given program can be made by examining the algorithm. A program requires storage space for instructions, input data, constants, variables and objects. If input data have one natural form (for example, an array) we can analyze the amount of extra space used, aside from the space needed for the program and its data. If the amount of extra space is constant w.r.t. the input size, the algorithm is said to work in place. If the input can be represented in different forms, then we must consider the space required for the input itself plus the extra space.