Regular Expressions and Automata

1. Regular Expressions and Automata Lecture #2 August 30 2012

2. Rule-based vs Statistical Approaches • Rule-based = linguistic • Statistical – “learn” from a large corpus that has been marked up with the phenomina you are studying • For what problems is rule-based better suited and when is statistical better? • Identifying proper names • Distinguishing a biography from a dictionary entry • Answering Questions • May depend on how much good training data is available

3. Rule-based vs Statistical Approaches • Even when using statistical – many tasks easily done with rules: tokenization, sentence breaking, morphology • Some PARTS of a task may be done with rules: e.g., rules may be used to extract features, and then statistical learning methods might combine those features to perform a task. • How much knowledge of language do our algorithms need to do useful NLP? 80/20 rule: • Claim: 80% of NLP can be done with simple methods (low hanging fruit • The last 20% can be more difficult to attain – when should we worry about that??

4. Today • Review some of the simple representations and ask ourselves how we might use them to do interesting and useful things • Regular Expressions • Finite State Automata • How much can you get out of a simple tool? • Should also think about the limits of these approaches: • How far can they take us? • When are they enough? • When do we need something more?

5. Regular Expressions • Can be viewed as a way to specify: • Search patterns over text string • Design of a particular kind of machine, called a Finite State Automaton (FSA) • These are really equivalent

6. Uses of Regular Expressions in NLP • As grep, perl: Simple but powerful tools for large corpus analysis and ‘shallow’ processing • What word is most likely to begin a sentence? • What word is most likely to begin a question? • In your own email, are you more or less polite than the people you correspond with? • With other unix tools, allow us to • Obtain word frequency and co-occurrence statistics • Build simple interactive applications (e.g., Eliza) • Regular expressions define regular languages or sets

7. Regular Expressions • Simple but powerful tools for “shallow” processing of a document of “corpus” • What word begins a sentence? • What words begin a question • Identify all NPs • These simple tools can enable us to • Build simple interactive applications (e.g., Eliza) • Recognize date, time, money… expressions • Recognize Named Entities (NE): people names, company names • Do morphological analysis

8. Regular Expressions • Regular Expression: Formula in algebraic notation for specifying a set of strings • String: Any sequence of alphanumeric characters • Letters, numbers, spaces, tabs, punctuation marks • Regular Expression Search • Pattern: specifying the set of strings we want to search for • Corpus: the texts we want to search through

9. Simple Example • Sequence of simple characters:

10. Some Examples

12. RE E.G. /pupp(y|ies)/ Morphological variants of ‘puppy’ / (.+)ier and \1ier / happier and happier, fuzzier and fuzzier

13. Optionality and Repetition • /[Ww]oodchucks?/ matches woodchucks, Woodchucks, woodchuck, Woodchuck • /colou?r/ matches color or colour • /he{3}/ matches heee • /(he){3}/ matches hehehe • /(he){3,} matches a sequence of at least 3 he’s

14. Operator Precedence Hierarchy 1. Parentheses () 2. Counters * + ? {} 3. Sequence of Anchors the ^my end$ 4. Disjunction | Examples /moo+/ /try|ies/ /and|or/

15. A Simple Exercise • Write a regular expression to find all instances of the determiner “the”: /the/ /[tT]he/ /\b[tT]he\b/ /(^|[^a-zA-Z][tT]he[^a-zA-Z]/ The recent attempt by the police to retain their current rates of pay has not gathered much favor with the southern factions.

16. A Simple Exercise • Write a regular expression to find all instances of the determiner “the”: /the/ /[tT]he/ /\b[tT]he\b/ /(^|[^a-zA-Z][tT]he[^a-zA-Z]/ The recent attempt by the police to retain their current rates of pay has not gathered much favor with southern factions.

17. A Simple Exercise • Write a regular expression to find all instances of the determiner “the”: /the/ /[tT]he/ /\b[tT]he\b/ /(^|[^a-zA-Z][tT]he[^a-zA-Z]/ The recent attempt by the police to retain their current rates of pay has not gathered much favor with the southern factions.

18. A Simple Exercise • Write a regular expression to find all instances of the determiner “the”: /the/ /[tT]he/ /\b[tT]he\b/ /(^|[^a-zA-Z][tT]he[^a-zA-Z]/ The recent attempt by the police to retain their current rates of pay has not gathered much favor with the southern factions.

19. A Simple Exercise • Write a regular expression to find all instances of the determiner “the”: /the/ /[tT]he/ /\b[tT]he\b/ /(^|[^a-zA-Z][tT]he[^a-zA-Z]/ The recent attempt by the police to retain their current rates of pay has not gathered much favor with the southern factions.

20. A Simple Exercise • Write a regular expression to find all instances of the determiner “the”: /the/ /[tT]he/ /\b[tT]he\b/ /(^|[^a-zA-Z])[tT]he[^a-zA-Z]/ The recent attempt by the police to retain their current rates of pay has not gathered much favor with the southern factions.

21. The Two Kinds of Errors • The process we just went through was based on fixing errors in the regular expression • Errors where the instances were included (when they should not have been) – False positives • Errors where some of the instances were missed (judged to not be instances when they should have been) – False negatives • This is pretty much going to be the story of the rest of the course!

22. Errors • We’ll be telling the same story for many tasks, all semester. Reducing the error rate for an application often involves two antagonistic efforts: • Increasing accuracy, or precision, (minimizing false positives) • Increasing coverage, or recall, (minimizing false negatives). Speech and Language Processing - Jurafsky and Martin

23. Substitutions (Transductions) • Sed or ‘s’ operator in Perl • s/regexp1/pattern/ • s/I am feeling (.++)/You are feeling \1?/ • s/I gave (.+) to (.+)/Why would you give \2 \1?/

24. Next Several Slides from McKeown 2009

27. ELIZA: Substitutions Using Memory User: Men are all alike. ELIZA: IN WHAT WAY s/.* all .*/IN WHAT WAY/ They’re always bugging us about something or other. ELIZA: CAN YOU THINK OF A SPECIFIC EXAMPLE s/.* always .*/CAN YOU THINK OF A SPECIFIC EXAMPLE/ User: My boyfriend says I’m depressed. ELIZA: I AM SORRY TO HEAR YOU ARE DEPRESSED s/.* I’m (depressed|sad) .*/I AM SORRY TO HEAR YOU ARE \1/

29. Using RE’s: Examples • Predictions from a news corpus: • Which candidate for Governor of California is mentioned most often in the news? Is going to win? • What stock should you buy? • Which White House advisers have the most power? • Language use: • Which form of comparative is more frequent: ‘oftener’ or ‘moreoften’? • Which pronouns are conjoined most often? • How often do sentences end with infinitival ‘to’? • What words most often begin and end sentences? • What’s the most common word in your email? Is it different from your neighbor?

30. Personality profiling: • Are you more or less polite than the people you correspond with? • With labeled data, which words signal friendly messages vs. unfriendly ones?

31. Finite State Automata • Regular Expressions (REs) can be viewed as a way to describe machines called Finite State Automata (FSA, also known as automata, finite automata). • FSAs and their close variants are a theoretical foundation of much of the field of NLP.

32. a b a a ! q0 q1 q2 q3 q4 Finite State Automata • FSAs recognize the regular languages represented by regular expressions • SheepTalk: /baa+!/ • Directed graph with labeled nodes and arc transitions • Five states: q0 the start state, q4 the final state, 5 transitions

33. a b a a ! q0 q1 q2 q3 q4 Formally • FSA is a 5-tuple consisting of • Q: set of states {q0,q1,q2,q3,q4} • : an alphabet of symbols {a,b,!} • q0: a start state • F: a set of final states in Q {q4} • (q,i): a transition function mapping Q x  to Q

34. State Transition Table for SheepTalk

35. Recognition • Recognition (or acceptance) is the process of determining whether or not a given input should be accepted by a given machine. • Whether the string is in the language defined by the machine • In terms of REs, it’s the process of determining whether or not a given input matches a particular regular expression. • Whether the string is in the language defined by the expression • Traditionally, recognition is viewed as processing an input written on a tape consisting of cells containing elements from the alphabet.

36. Recognition • Recognition (or acceptance) is the process of determining whether or not a given input should be accepted by a given machine. • Or… it’s the process of determining if as string is in the language we’re defining with the machine • In terms of REs, it’s the process of determining whether or not a given input matches a particular regular expression. • Traditionally, recognition is viewed as processing an input written on a tape consisting of cells containing elements from the alphabet.

37. FSA recognizes (accepts) strings of a regular language • baa! • baaa! • baaa! • … • Tape metaphor: a rejected input q0 a b a ! b

38. Recognition • Simply a process of starting in the start state • Examining the current input • Consulting the table • Going to a new state and updating the tape pointer. • Until you run out of tape.

39. D-Recognize

40. q0 q3 q3 q4 q1 q2 q3

41. Key Points • Deterministic means that at each point in processing there is always one unique thing to do (no choices). • D-recognize is a simple table-driven interpreter • The algorithm is universal for all unambiguous languages. • To change the machine, you change the table.

42. Key Points • Crudely therefore… matching strings with regular expressions (ala Perl) is a matter of • translating the expression into a machine (table) and • passing the table to an interpreter

43. Recognition as Search • You can view this algorithm as a degenerate kind of state-space search. • States are pairings of tape positions and state numbers. • Operators are compiled into the table • Goal state is a pairing with the end of tape position and a final accept state • Its degenerate because?

44. Formal Languages • Formal Languages are sets of strings composed of symbols from a finite set of symbols. • Finite-state automate define formal languages (without having to enumerate all the strings in the language) • Given a machine m (such as a particular FSA) L(m) means the formal language characterized by m. • L(Sheeptalk FSA) = {baa!, baaa!, baaaa!, …} (an infinite set)

45. Generative Formalisms • The term Generative is based on the view that you can run the machine as a generator to get strings from the language. • FSAs can be viewed from two perspectives: • Acceptors that can tell you if a string is in the language • Generators to produce all and only the strings in the language

46. Three Views • Three equivalent formal ways to look at what we’re up to (not including tables – and we’ll find more…) Regular Expressions Finite State Automata Regular Languages

47. Determinism • Let’s take another look at what is going on with d-recognize. • In particular, let’s look at what it means to be deterministic here and see if we can relax that notion. • How would our recognition algorithm change? • What would it mean for the accepted language?

48. Determinism and Non-Determinism • Deterministic: There is at most one transition that can be taken given a current state and input symbol. • Non-deterministic: There is a choice of several transitions that can be taken given a current state and input symbol. (The machine doesn’t specify how to make the choice.)

49. Non-Deterministic FSAs for SheepTalk a b a a ! q0 q1 q2 q3 q4 b a a ! q0 q1 q2 q3 q4 

50. FSAs as Grammars for Natural Language dr the rev mr pat l. robinson q0 q1 q2 q3 q4 q5 q6 ms hon  mrs  Can you use a regexpr to capture this too?