1 / 67

Understanding Complexity: O, Ω, and Θ Notations in Computer Science

Learn about the O, Ω, and Θ notations commonly used in computer science to analyze algorithmic complexity. This lecture provides examples and explanations of these notations and how they can help measure the efficiency of programs.

maryhorton
Download Presentation

Understanding Complexity: O, Ω, and Θ Notations in Computer Science

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lecture 15: Complexity Notes David Evans http://www.cs.virginia.edu/evans This is selected from CS200 lectures 16, 30, 37, 38 from http://www.cs.virginia.edu/cs200/lectures/ CS588: Security and Privacy University of Virginia Computer Science

  2. What does  really mean? • O(x)– it is no more thanxwork (upper bound) • (x) – work scales as x (tight bound) • (x) – it is at leastxwork (lower bound) If O(x) and (x) are true, then (x) is true. CS588 Spring 2005

  3. Meaning of O (“big Oh”) f(x) is O (g (x)) means: There is a positive constant c such that c * f(x) <g(x) for all but a finite number of x values. CS588 Spring 2005

  4. O Examples f(x) is O (g (x)) means: There is a positive constant c such that c * f(x) <g(x) for all but a finite number of x values. x is O (x2)? Yes, c = 1 works fine. 10x is O (x)? Yes, c = .09 works fine. No, no matter what c we pick, cx2 > x for big enough x x2 is O (x)? CS588 Spring 2005

  5. Lower Bound:  (Omega) f(x) is  (g (x)) means: There is a positive constant c such that c * f(x) >g(x) for all but a finite number of x values. Difference from O – this was < CS588 Spring 2005

  6. xis  (x) Yes, pick c = 2 10xis  (x) Yes, pick c = 1 Is x2 (x)? Yes! x is O(x) Yes, pick c = .5 10x is O(x) Yes, pick c = .09 x2 is notO(x) f(x) is  (g (x)) means: There is a positive constant c such that c * f(x) >g(x) for all but a finite number of x values. f(x) is O (g (x)) means: There is a positive constant c such that c * f(x) <g(x) for all but a finite number of x values. Examples CS588 Spring 2005

  7. Tight Bound:  (Theta) f(x) is (g (x)) iff: f(x) is O (g (x)) and f(x) is  (g (x)) CS588 Spring 2005

  8.  Examples • 10xis (x) • Yes, since 10xis  (x) and 10x is O(x) • Doesn’t matter that you choose different c values for each part; they are independent • x2is/is not  (x)? • No, since x2is not O(x) • x is/is not (x2)? • No, since x2is not (x) CS588 Spring 2005

  9. Measuring Work • When we say a procedure or a problem is O(f(n)) what are we counting? • Need a model computer to count steps CS588 Spring 2005

  10. How should we model a Computer? Colossus (1944) Cray-1 (1976) Apollo Guidance Computer (1969) CS588 Spring 2005 IBM 5100 (1975)

  11. Modeling Computers • Input • Without it, we can’t describe a problem • Output • Without it, we can’t get an answer • Processing • Need some way of getting from the input to the output • Memory • Need to keep track of what we are doing CS588 Spring 2005

  12. Modeling Input Punch Cards Altair BASIC Paper Tape, 1976 Engelbart’s mouse and keypad CS588 Spring 2005

  13. Simplest Input • Non-interactive: like punch cards and paper tape • One-dimensional: just a single tape of values, pointer to one square on tape 0 0 1 1 0 0 1 0 0 0 How long should the tape be? Infinitely long! We are modeling a computer, not building one. Our model should not have silly practical limitations (like a real computer does). CS588 Spring 2005

  14. Modeling Output • Blinking lights are cool, but hard to model • Output is what is written on the tape at the end of a computation Connection Machine CM-5, 1993 CS588 Spring 2005

  15. Modeling Processing • Evaluation Rules • Given an input on our tape, how do we evaluate to produce the output • What do we need: • Read what is on the tape at the current square • Move the tape one square in either direction • Write into the current square 0 0 1 1 0 0 1 0 0 0 Is that enough to model a computer? CS588 Spring 2005

  16. Modeling Processing • Read, write and move is not enough • We also need to keep track of what we are doing: • How do we know whether to read, write or move at each step? • How do we know when we’re done? • What do we need for this? CS588 Spring 2005

  17. Finite State Machines 1 0 0 2 1 Start 1 # HALT CS588 Spring 2005

  18. Hmmm…maybe we don’t need those infinite tapes after all? not a paren not a paren ( 2 1 Start ) ) What if the next input symbol is ( in state 2? # HALT ERROR CS588 Spring 2005

  19. How many states do we need? not a paren not a paren ( 2 not a paren 1 Start ( ) 3 ) not a paren ) # ( 4 HALT ERROR ) ( CS588 Spring 2005

  20. Finite State Machine • There are lots of things we can’t compute with only a finite number of states • Solutions: • Infinite State Machine • Hard to describe and draw • Add a tape to the Finite State Machine CS588 Spring 2005

  21. FSM + Infinite Tape • Start: • FSM in Start State • Input on Infinite Tape • Pointer to start of input • Move: • Read one input symbol from tape • Follow transition rule from current state • To next state • Write symbol on tape, and move L or R one square • Finish: • Transition to halt state CS588 Spring 2005

  22. Matching Parentheses • Repeat until halt: • Find the leftmost ) • If you don’t find one, the parentheses match, write a 1 at the tape head and halt. • Replace it with an X • Look left for the first ( • If you find it, replace it with an X (they matched) • If you don’t find it, the parentheses didn’t match – end write a 0 at the tape head and halt CS588 Spring 2005

  23. Matching Parentheses Input: ) Write: X Move: L ), X, L (, (, R X, X, L X, X, R 2: look for ( 1 Start (, X, R #, 0, # #, 1, # HALT Will this report the correct result for (()? CS588 Spring 2005

  24. Matching Parentheses (, (, R X, X, L ), X, L X, X, R 1 2: look for ( (, X, R Start #, #, L #, 1, # #, 0, # 3: look for ( X, X, L HALT #, 1, # (, 0, # CS588 Spring 2005

  25. Turing Machine • Alan Turing, On computable numbers: With an application to the Entscheidungsproblem, 1936 • Turing developed the machine abstraction to show the halting problem really leads to a contradiction • Our informal argument, depended on assuming we could do if and everything else except halts? CS588 Spring 2005

  26. Describing Turing Machines z z z z z z z z z z z z z z z z z z z z z z TuringMachine ::= < Alphabet, Tape, FSM > Alphabet ::= { Symbol* } Tape ::= < LeftSide, Current, RightSide > OneSquare ::= Symbol | # Current ::= OneSquare LeftSide ::= [ Square* ] RightSide ::= [ Square* ] Everything to left of LeftSide is #. Everything to right of RightSide is #. ), X, L ), #, R (, #, L 2: look for ( 1 Start (, X, R HALT #, 0, - #, 1, - Finite State Machine CS588 Spring 2005

  27. ), X, L ), #, R (, #, L Describing Finite State Machines 2: look for ( 1 Start (, X, R #, 0, # #, 1, # HALT TuringMachine ::= < Alphabet, Tape, FSM > FSM ::= < States, TransitionRules, InitialState, HaltingStates > States ::= { StateName* } InitialState ::= StateName must be element of States HaltingStates ::= { StateName* } all must be elements of States TransitionRules ::= { TransitionRule* } TransitionRule ::= < StateName, ;; Current State OneSquare, ;; Current square StateName, ;; Next State OneSquare, ;; Write on tape Direction > ;; Move tape Direction ::= L, R, # Transition Rule is a procedure: StateName X OneSquare  StateName X OneSquare X Direction CS588 Spring 2005

  28. ), X, L ), #, R (, #, L Example Turing Machine 2: look for ( 1 Start (, X, R #, 0, # #, 1, # HALT TuringMachine ::= < Alphabet, Tape, FSM > FSM ::= < States, TransitionRules, InitialState, HaltingStates > Alphabet ::= { (, ), X } States ::= { 1, 2, HALT } InitialState ::= 1 HaltingStates ::= { HALT} TransitionRules ::= { <1, ), 2, X, L >, < 1, #, HALT, 1, # >, < 1, ), #, R >, < 2, (, 1, X, R >, < 2, #, HALT, 0, # >, < 2, ), #, L >,} CS588 Spring 2005

  29. Enumerating Turing Machines • Now that we’ve decided how to describe Turing Machines, we can number them • TM-5023582376 = balancing parens • TM-57239683 = even number of 1s • TM-3523796834721038296738259873 = Photomosaic Program • TM-3672349872381692309875823987609823712347823 = WindowsXP Not the real numbers – they would be much bigger! CS588 Spring 2005

  30. Complexity Class P Class P: problems that can be solved in polynomial time by a deterministic TM. O (nk) for some constant k. Easy problems like sorting, making a photomosaic using duplicate tiles, simulating the universe are all in P. CS588 Spring 2005

  31. Deterministic Computing • All our computing models so far are deterministic: you know exactly what to do every step • TM: only one transition rule can match any machine configuration, only one way to follow that transition rule • Lambda Calculus: always -reduce the outermost redex CS588 Spring 2005

  32. Nondeterministic Computing • Allows computer to try different options at each step, and then use whichever one works best • Make a Finite State Machine non-deterministic: doesn’t change computing time • Make a Turing Machine non-deterministic: might change computing time • No one knows whether it does or not • This is the biggest open problem in Computer Science CS588 Spring 2005

  33. Making FSM’s Nondeterministic 0 0 0 1 3 1 2 1 # HALT CS588 Spring 2005

  34. Nondeterministic FSM 0, 1 0 1 3 1 2 # HALT CS588 Spring 2005

  35. Power of NFSM • Can NFSMs computing anything FSMs cannot compute? • Can NFSMs compute faster than FSMs? No – you can always convert an NFSM to a FSM by making each possible set of states in the NFSM into one state in the new FSM. (Number of states may increase as 2s) No – both read input one letter at a time, and transition automatically to the next state. (Both are always (n) where n is the number of symbols in the input.) CS588 Spring 2005

  36. Nondeterministic TMs • Multiple transitions possible at every step • Follow all possible steps: • Each possible execution maintains its own state and its own infinite tape • If any possible execution halts (in an accepting state), the NTM halts, and the tape output of that execution is the NTM output CS588 Spring 2005

  37. Complexity Classes Class P: problems that can be solved in polynomial time by a deterministic TM. O (nk) for some constant k. Easy problems like simulating the universe are all in P. Class NP: problems that can be solved in polynomial time by a nondeterministic TM Hard problems like the pegboard puzzle sorting are in NP (as well as all problems in P). CS588 Spring 2005

  38. Problem Classes Fill tape with 2n *s: (2n) Simulating Universe: O(n3) Undecidable NP Decidable P Halting Problem: () Sorting: (n log n) Cracker Barrel: O(2n) and (n) CS588 Spring 2005

  39. P = NP? • Is there a polynomial-time solution to the “hardest” problems in NP? • No one knows the answer! • The most famous unsolved problem in computer science and math • Listed first on Millennium Prize Problems • win $1M if you can solve it • (also an automatic A+ in this course) CS588 Spring 2005

  40. If P  NP: Fill tape with 2n *s: (2n) Simulating Universe: O(n3) Undecidable NP Decidable P Halting Problem: () Sorting: (n log n) Cracker Barrel: O(2n) and (n) CS588 Spring 2005

  41. If P = NP: Fill tape with 2n *s: (2n) Simulating Universe: O(n3) Undecidable NP Decidable P Halting Problem: () Sorting: (n log n) Cracker Barrel: O(2n) and (n) CS588 Spring 2005

  42. Smileys Problem Input: n square tiles Output: Arrangement of the tiles in a square, where the colors and shapes match up, or “no, its impossible”. CS588 Spring 2005

  43. Thanks to Peggy Reed for making the Smiley Puzzles!

  44. How much work is the Smiley’s Problem? • Upper bound: (O) O (n!) Try all possible permutations • Lower bound: ()  (n) Must at least look at every tile • Tight bound: () No one knows! CS588 Spring 2005

  45. NP Problems • Can be solved by just trying all possible answers until we find one that is right • Easy to quickly check if an answer is right • Checking an answer is in P • The smileys problem is in NP We can easily try n! different answers We can quickly check if a guess is correct (check all n tiles) CS588 Spring 2005

  46. Is the Smiley’s Problem in P? No one knows! We can’t find a O(nk) solution. We can’t prove one doesn’t exist. CS588 Spring 2005

  47. This makes a huge difference! n! 2n time since “Big Bang” Solving a large smileys problem either takes a few seconds, or more time than the universe has been in existence. But, no one knows which for sure! 2032 today n2 n log n log-log scale CS588 Spring 2005

  48. Who cares about Smiley puzzles? If we had a fast (polynomial time) procedure to solve the smiley puzzle, we would also have a fast procedure to solve the 3/stone/apple/tower puzzle: 3 CS588 Spring 2005

  49. 3SAT  Smiley Step 1: Transform into smileys Step 2: Solve (using our fast smiley puzzle solving procedure) Step 3: Invert transform (back into 3SAT problem     CS588 Spring 2005

  50. The Real 3SAT Problem(also can be quickly transformed into the Smileys Puzzle) CS588 Spring 2005

More Related