Tonga Institute of Higher Education Design and Analysis of Algorithms

1. Tonga Institute of Higher EducationDesign and Analysis of Algorithms IT 254 Lecture 8: Complexity Theory

2. Complexity Introduction • Some problems are intractable: as they grow large, we are unable to solve them in reasonable time • What does reasonable time mean? Most of the time this means polynomial time • On an input of size n the worst-case running time is O(nk) for some constant k • Polynomial time: O(n2), O(n3), O(1), O(n lg n) • Not in polynomial time: O(2n), O(nn), O(n!)

3. Polynomial Algorithms • Are some problems solvable in polynomial time? • Most algorithms we have looked at provide polynomial-time solutions to some problem • In computer science, we say P is the class of problems solvable in polynomial time • Are all problems solvable in polynomial time? • No: Turing’s “Halting Problem” is not solvable by any computer, no matter how much time is given • Turing’s problem tries to decide if a computer will stop on a given input or keep going. (Explained in class.) • There are also problems that are clearly intractable (not in P), and there are also some that might be in P, but no one has figured out a good algorithm yet

4. Exponential Algorithms • There are some algorithms that can only be solved in exponential time [O(n!),O(2n)] • This is called Exp for exponential • We need to realize that all algorithms in P are also in Exp, because the exponential class contains all polynomial time solutions as well. • Think of this like upper bounds on algorithms Exp time Problems That can’t Be solved P time

5. NP Complete Problems • The NP-Complete problems are an interesting class of problems whose status is unknown • No polynomial-time algorithm has been discovered for an NP-Complete problem • But, also no lower bound has been proved for any NP-Complete problem that says it must be bigger than polynomial • We call this the P = NP question • The biggest open problem in CS and the person who solves it will win a lot of prizes and a lot of money

6. NP Complete Example • An example of an NP-Complete problem: • A hamiltonian cycle in an undirected graph is a simple cycle that contains every vertex • The hamiltonian-cycle problem: given a graph G, does it have a hamiltonian cycle? • A graph that has a hamiltonian cycle is said to be "Hamiltonian" Is there a Hamiltonian Cycle here? Can we think of an algorithm to find it?

7. Hamiltonian Cycles • One way to check is if you list all the different permutations of the vertices of a graph and then check each permutation to see if it is a hamiltonian path. • A permutation here means a configuration of a path that tries to go to each vertex in a cycle • This idea unfortunately runs in O(n!) where n is the number of vertices. • But we can check it easily. All we do is go through a permutation and see if it is Hamiltonian.

8. P and NP • As mentioned, P is set of problems that can be solved in polynomial time • NP (nondeterministic polynomial time) is the set of problems that can be solved in polynomial time by a nondeterministic computer. • What does that mean?

9. Non-deterministic Computers • Think of a non-deterministic computer as a computer that magically “guesses” a solution, then has to make sure that it is correct • If a solution exists, the computer always guesses it • One way to imagine it: a computer with an infinite number of CPUs that can use all of them • Have one processor work on each possible solution • All processors try to make sure that their solution works • If a processor finds it has a working solution, then it’s solved • So: NP = problems that can be checked for correctness in polynomial time, • or in other words, if you had a magic computer, you could guess all the solutions and then just check them all to find the right one

10. N and NP • Summary so far: • P = problems that can be solved in polynomial time • NP = problems for which a solution can be verified in polynomial time • Exp = problems that can only be solved in Exponential time • Unknown whether P = NP (most suspect not) • Hamiltonian-cycle problem is in NP: • Cannot be solved in polynomial time (yet) • Easy to verify solution in polynomial time because we just need to run through a path to see if it touches all vertexes

11. … And NP Complete • There is also a special class of NP problems, called NP-Complete problems. It says: • If any NP-Complete problem can be solved in polynomial time… • …then every NP-Complete problem can be solved in polynomial time… • …and, in fact, every problem in NP can be solved in polynomial time (which would show P = NP) • Thus: If you can solve the Hamiltonian-cycle in O(n100) time for any n, you’ve proved that P = NP. • Retire rich & famous.

12. Reducibility • The most important part of NP-Completeness is called reducibility • We can say that a problem P can be reduced to another problem Q if all the types of P can be “easily changed” into a type of Q • And the solution to Q will provide a solution to the instance of P • Thus, if P reduces to Q, then P is “no harder to solve” than Q. • The key is “easily changed” – by this, we mean can change the problem from P to Q in polynomial time • Example: The problem of solving linear equations for an unknown "x" can reduce to solving quadratic equations: • Instance: ax + b = 0. We can change this into 0x2 + ax + b = 0 and then we can use the quadratic formula to solve • Changing it was easy, just add a 0x2

13. Reducibility • An example of reducibility (that is not NP): • P: Given a set of Booleans, is at least one TRUE? • Example { x1, x2, x3, x4, x5 } where x1..5 can be true or false • Q: Given a set of integers, is their sum positive? • Can we change from P  Q? • Transformation: (x1, x2, …, xn) = (y1, y2, …, yn) where yi = 1 if xi = TRUE, yi = 0 if xi = FALSE • Then add the y’s together, if it is greater than 0, then we also know at least one x is true.

14. Using Reductions • If S is polynomial-time reducible to Q, we write this like, S p Q • Definition of NP-Hard and NP-Complete: • If all problems R  NP are reducible to S, then S is NP-Hard • We say S is NP-Complete if S is NP-Hard and S  NP • Formally, NP Complete means: R p S  R  NP • If S p Q and S is NP-Complete, Q is also NP-Complete • This is the important idea you should take away today NP P NPC

15. Who cares about NP-Complete? • Though nobody has proven that P != NP, if you prove a problem NP-Complete, most people accept that it is probably intractable • Therefore it can be important to prove that a problem is NP-Complete • Don’t need to come up with an efficient algorithm, because it can't be solved • Can instead work on approximation algorithms (algorithms that come close to the solution)

16. Proving NP-Completeness • What steps do we have to take to prove a problem Pis NP-Complete? • Pick a known NP-Complete problem Q • Reduce Q to P, which means: • Describe an algorithm that changes an instance of Q to an instance of P • Prove the transformation works • Prove it runs in polynomial time • And don’t forget, we need to prove P NP • Meaning, we have to prove that the solution can be checked in polynomial time

17. NP Complete problems • Given one NP-Complete problem, we can prove many interesting problems NP-Complete • Hamiltonian cycle • Graph coloring • If you have a graph, can you color the graph with at most k colors so no two vertices with the same color are next to each other? • Hamiltonian path • Does there exist a path between two vertices of a graph that visits each vertex only once? • Knapsack problem • If you have a bag that is a certain size, and there are different things with different sizes you can put in your bag, what is the best way to maximize what you put in • Traveling salesman • If a salesman has to visit every city, how can he minimize the distance that he travels totally.

18. Hamiltonian means Traveling Salesman • The Hamiltonian path problem is similar to a very important problem. • The well-known traveling salesman problem: • It’s goal is Optimization : a salesman must travel to n cities, visiting each city exactly once and finishing where he begins. How to minimize travel time or travel distance? • Model as complete graph with cost c(i,j) to go from city i to city j • How would we turn this into a decision problem that can be verified in polynomial time? • Answer: ask if there exists a TSP with cost < k

19. Traveling Salesman • The steps to prove TSP is NP-Complete: • Prove that TSP  NP (we will not do) • Reduce the undirected hamiltonian cycle problem to the TSP • So if we had a TSP-solver, we could use it to solve the hamilitonian cycle problem in polynomial time • How can we transform an instance of the hamiltonian cycle problem to an instance of the TSP? • Can we do this in polynomial time?

20. Traveling Salesmen • To transform the hamiltonian cycle problem on graph G = (V,E) into TSP, • create graph G1 = (V,E1): • G1 is a complete graph • Edges in E1 also in E have weight 0 • All other edges in E1 have weight 1 • TSP: is there a TSP on G1 with weight 0? • If G has a hamiltonian cycle, G1 has a cycle w/ weight 0 • If G1 has cycle w/ weight 0, every edge of that cycle has weight 0 and is thus in G. Thus G has a ham. cycle

21. The SAT Problem • One of the first problems to be proved NP-Complete was satisfiability (SAT): • Given a Boolean expression on n variables, can we assign values such that the expression is TRUE? • Ex: ((x1x2)  ((x1 x3)  x4)) x2 • Cook’s Theorem: The satisfiability problem is NP-Complete. • It is quite complicated to prove that it is NP-Complete, • Thus most problems today are reduced with this SAT problem and that makes them NP-Complete

22. Conjuctive Normal Form • Even if the form of the Boolean expression is simplified, the problem is still NP-Complete • Literal: an occurrence of a Boolean or its negation • A Boolean formula is in conjunctive normal form, or CNF, if it is an AND of clauses, each of which is an OR of literals • Ex: (x1  x2)  (x1  x3  x4)  (x5) • 3-CNF: each clause has exactly 3 distinct literals • Ex: (x1  x2  x3)  (x1  x3  x4)  (x5  x3  x4) • Notice: true if at least one literal in each clause is true • It is possible to reduce all Boolean equations to 3-CNF form in polynomial time

23. Satisifiability • How can we transform a Boolean function into 3-CNF form? • Make a truth table and change it into normal form. • Then we can use DeMorgan's law to change to conjuctive normal form (product of sums). • Then we need to make sure there are only 3 literals per statement, which is not difficult, but there are extra rules to follow. • For example, if we have (p + q), to make this into 3-CNF format we write: (p + q + r) + (p + q + ~r) • Does this still give you the same value? • Even though it might seem easy to pick values to make a statement true, no one has been able to do so in Polynomial time.

24. Summary • What is important to know about NP completeness and complexity? • There will be times that you have problems and you might like to use a computer to solve them. It’s good to know that you can’t solve these problems in a reasonable amount of time • One example – scheduling classes for students is a NP complete problem. You might remember waiting weeks for the teachers to finally make up all the student’s schedules. • If you wanted a computer to do this, you would need to realize it is a NP-hard problem (there does not exist a way to do it in polynomial time… yet)