Theory of Algorithms: Introduction

Theory of Algorithms:Introduction James Gain and Edwin Blake {jgain | edwin} @cs.uct.ac.za Department of Computer Science University of Cape Town August - October 2004

Objectives • To define an algorithm • To introduce: • Problem types • The Process of Algorithm Design • Solution strategies • Ways of Analysing Algorithms • To cover the structure of the course, including practicals

Definitions • An algorithm is a sequence of unambiguous instructions for solving a problem • For obtaining a required output for any legitimate input in a finite amount of time • Does not require implementation in software • Not an answer but a method for deriving an answer • Historical Perspective: • Named after Muhammad ibn Musa al-Khwarizmi – 9th century mathematician • www.lib.virginia.edu/science/parshall/khwariz.html

Notion of algorithm problem algorithm “computer” input output • Each step of the algorithm must be unambiguous • The range of inputs must be specified carefully • The same algorithm can be represented in different ways • Several algorithms for solving the same problem may exist - with different properties

What is an algorithm? • Recipe, process, method, technique, procedure, routine,… with following requirements: • Finiteness • Terminates after a finite number of steps • Definiteness • Rigorously and unambiguously specified • Input • Valid inputs are clearly specified • Output • Can be proved to produce the correct output given a valid input • Effectiveness • Steps are sufficiently simple and basic

Example: Sorting • Statement of problem: • Input: A sequence of n numbers <a1, a2, …, an> • Output: A reordering of the input sequence <a´1, a´2, …, a´n> so that a´i≤ a´j whenever i < j • Instance: The sequence <5, 3, 2, 8, 3> • Algorithms: • Selection sort • Insertion sort • Merge sort • (many others)

for i=1 to n swap a[i] with smallest of a[i],…,a[n] Selection Sort • Input: array a[1],..,a[n] • Output: array a sorted in non-decreasing order • Algorithm: • for i1 to ndo • min  i • for j  i+1 to n do • if a[j] < a[min] min  j • swap a[i] and a[min]

Exercise: Bridge Puzzle • Problem: • 4 People want to cross a bridge. You have 17 minutes to get them across • Constraints: • It is night and you have 1 flashlight. Max of 2 on the bridge at one time. All start on the same side • Those crossing must have the flashlight with them. The flashlight must be walked back and forth (no throwing) • People walk at different speeds: person A = 1 minute to cross, person B = 2 minutes, person C = 5 minutes, person D = 10 minutes • A pair walks at the speed of the slower person’s pace • Rumour: this problem is given to Microsoft interviewees

Solution: Bridge Puzzle • Start (0 min): A B C D • AB Across (2 min): A B C D • A Back (1 min): B A C D • CD Across (10 min): B C D A • B Back (2 min): C D A B • AB Across (2 min): A B C D • Total Time = 17 minutes

Extension Exercise • This is an instance of a problem. How would you generalise it? • Can you derive an algorithm to solve this generalised problem? • Must show the sequence of moves • Must output the minimum time required for crossing • Are there any special cases to watch out for? • Are there any constraints on the input?

Extension Solution • Input: a list a of crossing times for n people, numbered 1, …, n • Output: total time to cross • Strategy: use 1 & 2 as shuttles and send the others across in pairs • for i 2 to n/2do • t  a[2] // 1 & 2 across • t  t + a[1] // 1 back • t  t + a[i*2] // i*2 & (i*2)-1 across • t  t + a[2] // 2 back • t  a[2] // 1 & 2 across • return t

Extension Problems • This is an inadequate solution • It falsely assumes certain inputs • List may not be sorted in ascending order • Sort a • n may not be even numbered • Alter final iteration of loop • n > 3 not guaranteed • Special case for n = 1, 2, 3 • Is not optimal for all inputs, e.g. 1, 20, 21, 22 • Can you quantify the nature of these inputs? Suggest an alternative. • Final solution is left as an exercise. Attempt to make your solution elegant

Fundamentals of Algorithmic Problem Solving • Understanding the Problem • Make sure you are solving the correct problem and for all legitimate inputs • Ascertaining the Capabilities of a Computational Device • Sequential vs. Parallel. • What are the speed and memory limits? • Choosing between exact and approximate Problem Solving • Is absolute precision required? Sometimes this may not be possible • Deciding on Appropriate Data Structures • Algorithms often rely on carefully structuring the data • Fundamental Data Structures: array, linked list, stacks, queues, heaps, graphs, trees, sets

Fundamentals of Algorithm Design • Applying an Algorithm Design Technique • Using a general approach to problem solving that is applicable to a variety of problems • Specifying the Algorithm • Pseudocode is a mixture of natural language and programming constructs that has replaced flowcharts • Proving an Algorithms Correctness • Prove that an algorithm yields a required result for legitimate inputs in finite time • Analyzing an Algorithm • Consider time efficiency, space efficiency, simplicity, generality, optimality • Analysis can be empirical or theoretical • Coding an Algorithm

Well known Computational Problems • Sorting • Searching • String Processing • String Matching • Graph Problems • Graph Traversal, Shortest Path, Graph Colouring • Combinatorial Problems • Find a combinatorial object - permutation, combination, subset - subject to constraints • Geometric Problems • Closest-Pair, Convex-Hull • Numerical Problems • Solving systems of equations, computing definite integrals, evaluating functions, etc.

Algorithm Design Strategies • Brute force • A straightforward approach to solving a problem, usually directly based on the problem’s statement • Divide and conquer • Divide a problem into smaller instances, solve smaller instances (perhaps recursively), combine • Decrease and conquer • Exploit relationship between the problem and a smaller instance reduced by some factor (often 1) • Transform and conquer • Transform the problem to a simpler instance, another representation or an instance with a known solution

More Algorithm Design Strategies • Greedy approach • Make locally optimal steps which (hopefully) lead to a globally optimal solution for an optimization problem • Dynamic programming • Technique for solving problems with overlapping sub-domains • Backtracking and Branch and bound • A way of tackling difficult optimization and combinatorial problems without exploring all state-space • Space and time tradeoffs • Preprocess the input and store additional information to accelerate solving the problem

How to Solve It: Understanding the Problem • Taken from G. Polya, “How to Solve It”, 2nd edition. A classic textbook on problem solving for mathematics • You have to understand the problem. • What is the unknown? What are the data? Is the problem statement sufficient, redundant, contradictory • Draw a figure. Introduce suitable notation • Separate the various parts of the problem. Can you write them down?

Devising a Plan • Find the connection between the data and the unknown. You may be obliged to consider auxiliary problems if an immediate connection cannot be found. You should obtain eventually a plan of the solution. • Have you seen it before? Or have you seen the same problem in a slightly different form? • Do you know a related problem? Do you know a theorem that could be useful? • Look at the unknown! And try to think of a familiar problem having the same or a similar unknown. • Could you restate the problem? Could you restate it still differently? Go back to definitions. • If you cannot solve the proposed problem try to solve first some related problem. Are the unknown and the new data nearer to each other? • Did you use all the data? Did you use the whole condition? Have you taken into account all essential notions involved in the problem?

Carrying it Through • Carry out the Plan • Carrying out your plan of the solution, check each step. Can you see clearly that the step is correct? Can you prove that it is correct? • Looking Back • Can you check the result? Can you check the argument? • Can you derive the solution differently? Can you see it at a glance? • Can you use the result, or the method, for some other problem?

Analysis of Algorithms • How good is the algorithm? • Correctness • Time efficiency • Space efficiency • Simplicity • Does there exist a better algorithm? • Lower bounds • Optimality

Why Study Algorithms? • Theoretical importance • The core of computer science • Practical importance • A practitioner’s toolkit of known algorithms • Framework for designing and analyzing algorithms for new problems • Useful mindset

Course Structure • Fundamentals of the Analysis of Algorithms (Ch. 2) • Asymptotic notations, analysis of recursive and non-recursive algorithms, empirical analysis • Algorithmic Strategies (Ch. 3-9) • Brute force, Divide-and-Conquer, Decrease-and-Conquer, Transform-and-Conquer, Space and Time Tradeoffs, Greedy Techniques, Biologically-inspired techniques, Dynamic Programming • Limitations of Algorithms (Ch. 10 + handouts) • Turing Machines, Computability, Problem Classification • Coping with Limitations on Algorithms (Ch. 11) • Backtracking and Branch and Bound • Anany Levitin, “Introduction to the Design and Analysis of Algorithms”, International Edition, Addison-Wesley, 2003

Practicals • Weekly mini prac exams • Given a problem specification that is solvable using the algorithm design strategies presented in the course • Design Algorithm • Code it in C++ • Submit it for automatic marking • After the 3-hour lab session will be asked to do a short analysis of the solution

Theory of Algorithms: Introduction