Artificial Intelligence The different levels of language analysis

1. Artificial Intelligence The different levels of language analysis Fall 2008 professor: Luigi Ceccaroni

2. The different levels of language analysis • A NL system must use considerable knowledge about the structure of the language itself, including: • What the words are • How words combine to form sentences • What the words mean • How word meanings contribute to sentence meaning • …

3. The different levels of language analysis • We cannot completely account for linguistic behavior without also taking into account: • humans’ general world knowledge • their reasoning abilities • For example, to participate in a conversation it is necessary: • Knowledge about the structure of the language, but also • Knowledge about the world in general • Knowledge about the conversational setting in particular

4. The different levels of language analysis • The following are some of the different forms of knowledge relevant for NL understanding: • Phonetic and phonological knowledge • Morphological knowledge • Syntactic knowledge • Semantic knowledge • Pragmatic knowledge • Discourse knowledge • World knowledge

5. Phonetic and phonological knowledge • It concerns how words are related to the sounds that realize them • Such knowledge is crucial for speech-based systems

6. Morphological knowledge Lexeme cant- Information cantar V / Infinitive -o/-es/-a/-em/-eu/-en • Use of lexicons • Reference books containing an alphabetical list of words with information about them

7. Lexema cant Morphological knowledge Gramema o es a em en • Versió simple: utilització de formaris (llista de formes amb informació morfològica i els lexemes corresponents) • Morfema = lexema (o arrel) o gramema

8. Morphological knowledge • Analitzadors morfològics: • Diccionaris de morfemes: • diccionari d’arrels (lexemes), de sufixes, prefixes, infixes • Morfotàctica: regles de combinació de morfemes • Variacions fonològiques: canvis al combinar els morfemes (ex., ploure, plovisquejar) • Tipus d’analitzadors • FSA (finite state automaton) • FST (finite state transducer) • cascada de FSTs

9. Syntactic knowledge • Detection of tractable units: paragraphs and sentences: • Location of marks of punctuation: “.”, “?”, “!”, “…” • Problems: acronyms, initials • Machine learning and classification techniques • They take into account contextual information

10. Syntactic knowledge • Detection of units of meaning • Recognition and adequate grouping of words • “/Parlarà/ /sens dubte/ /de/ /les/ /reestructuracions/ /urbanes/ /a/ /Sant Cugat/” • Assignment of grammatical categories • Addition of semantic information to lexical units (use of ontologies and dictionaries) • Acknowledgment and classification of proper names and entities

11. Syntactic knowledge • Correspondence between orthographic and grammatical words • “dóna-m’ho”, “dímelo” (1 orthographic word, 3 grammatical words) • “sens dubte”, “sin embargo” (2 orthographic words, 1 p. grammatical word) • Homonymy • Same form, different grammatical categories: • “wheel” (rueda, ruleta): noun • “wheel” (empujar): verb

12. Syntactic knowledge • Polysemy • Same form and category, different meanings: • “bank” (grupo): data bank, electronic resource bank • “bank” (banco): bank account, bank balance • “bank” (ribera): river bank, river + burst its banks • Acronyms • “The cell’s DNA sample was identified by PRC, a process approved by the official UBI.”

13. Syntactic knowledge • Abbreviations • For example, the word "abbreviation" can itself be represented by the abbreviation "abbr." or "abbrev.“ • Formulae and units of measurements • “One of many famous formulae is Albert Einstein's E = mc².” • “In the US the inch is now defined as 0.0254 m, and the avoirdupois pound is now defined as 453.59237 g.”

14. Syntactic knowledge • Syntactic categories: • adjective (ADJ) • adjective phrase (ADJP) • adverb (ADV) • adverbial phrase (ADVP) • article (ART) • auxiliary verb (AUX) • determiner (DET) • noun (N) • noun phrase (NP) • preposition (P) • prepositional phrase (PP) • pronoun (PRO) • relative clause (REL) • relative pronoun (REL-PRO) • quantifying determiner (QDET) • sentence (S) • verb (V) • verb phrase (VP)

15. Syntactic knowledge • Problema de la granularitat (verb -> transitiu/intransitiu) • Propietats sintàctiques de concordança • gènere (masculí/femení) • nombre (singular/plural) • persona (primera, segona...) • cas (acusatiu,datiu..)

16. Representation • Altres propietats sintàctiques: • Tipus de complement del verb • Preposicions que accepta una paraula • Categoria semàntica • Informació morfològica • Derivació: prefixos/infixos/sufixos plov + -isquej- + ar re- + estructura + -cio + -ns arrel prefix sufix sufix

17. Representation re- + estructura + -cio + -ns • Informació lèxica repetició nom plural arrel sufix prefix sufix

18. Representation plov + -isquej- + ar • Informació lèxica diminutiu infinitiu arrel sufix infix

19. Syntax and semantics • Examples: • John sold the book to Mary. • The book was sold to Mary by John. • *After it fell in the river, John sold the book to Mary. • After it fell in the river, the book was sold to Mary by John. • *John are in the corner. • *John put the book. • Flying planes are dangerous. • Flying planes is dangerous.

20. Collaboration between parsers Interpretació Semàntica Anàlisi Sintàctica • sense sintaxi • sense semàntica • procés en cascada (1) • sintaxi | semàntica • procés en cascada (2) • {sintaxi + filtre semàntic} | semàntica • procés en paral·lel • {sintaxi, semàntica}

21. Pre-process • Segmentació • Localització d’unitats (paraules) • Lematització, anàlisi morfològica • Desambiguació morfosintàctica (POS-tagging) • Etiquetat semàntic • Desambiguació semàntica (WSD) • Detecció i classificació d‘entitats amb nom (named entity recognition, NER)

22. Quina es la capital de França? Example resultat de l'anàlisi morfològica quina quin DT0FS00 quina NCFS000 és ésser VMIP3S0 la el TDFS0 ell PP3FSO00 la I capital capital AQPCS00 capital NCFS000 capital NCMS000 de de SPS00 França frança NP00000-loc ? ? Fit resultat del POS-tagging quina quin DT0FS00 és ésser VMIP3S0 la el TDFS0 capital capital NCFS000 de de SPS00 França frança NP00000-loc ? ? Fit

23. Post-process • Anàlisi semàntica - pragmàtica • Anàlisi il·locutiva • Reconeixement d’intencions

24. The component steps of communication • A typical communication episode, in which speaker S wants to inform hearer H about proposition P using words W, is composed of 7 processes: • Intention. Speakers S decides that there is some proposition P that is worth saying to hearer H. • Generation. The speaker plans how to turn the proposition P into an utterance that makes it likely that the hearer can infer the meaning P (or something close to it). 24

25. The component steps of communication • Synthesis. The speaker produces the physical realization W’ of the words W. This can be via ink on paper, vibrations in air, or some other medium. • Perception. H perceives W’ as W2’ and decodes it as the words W2. When the medium is speech, the perception is called speech recognition; when it is printing, it is called optical character recognition (OCR). Both are now commonplace. 25

26. The component steps of communication • Analysis. We divide it into 3 main parts: • Syntactic interpretation (or parsing) is the process of building a parse tree for an input string. The interior nodes of the parse tree represent phrases and the leaf nodes represent words. • Semantic interpretation is the process of extracting the meaning of an utterance. Utterances with several possible interpretations are said to be ambiguous. • Pragmatic interpretation takes into account the fact that the same words can have different meanings in different situations. 26

27. The component steps of communication • Disambiguation. H infers that S intended to convey Pi (where ideally Pi = P). Communication works because H does the work of figuring out which interpretation is the one S probably meant to convey. Disambiguation is a process that depends heavily on uncertain reasoning. • Incorporation. H decides to believe Pi (or not). A totally naive agent might believe everything it hears. A sophisticated agent treats the speech act as evidence for Pi, not confirmation of it. 27

28. Syntactic analysis • Objectives: • Determining if a sentence is syntactically correct • Creating a structure with information which can be used during the semantic analysis

29. Syntactic analysis • Alphabet (vocabulary): Σ • Concatenation operations • Σ* (free monoid): set of all strings that can be formed with symbols of Σ • Language: L ⊆Σ* • Given a string w1n of Σ*: w1n = w1, …, wn wi∈Σ • We have to determine if w1n∈ L

30. Ways to define membership • Grammar • G ⇒ L(G) • w1n∈ L(G) ? • Language model • P(w1n) • si P(w1n) > 0 ⇒ w1n∈ L • Corpora (sentences, patterns), which define correct sentences: • Syntactic dictionaries • Composition rules

31. Most usual way: grammar <V, Σ, P, S> Initial variable Non-terminal vocabulary (set of variables) Production set Terminal vocabulary (alphabet) Σ∩ V = Ø Σ∪ V = vocabulary S ∈ V

32. Grammars and parsing • To examine how the syntactic structure of a sentence can be computed: • Grammar, a formal specification of the structures allowable in the language • Parsing technique, the method of analyzing a sentence to determine its structure according to the grammar • The most common way of representing how a sentence is broken into its major subparts (constituents), and how those subparts are broken up in turn, is a tree.

33. Grammars and sentence structure • Tree representation for the sentence Adrià menja el bacallà: S VP NP NAME NP V ART N menja bacallà Adrià el

34. Grammars and sentence structure • The sentence (S) consists of an initial noun phrase (NP) and a verb phrase (VP). • The initial noun phrase is made of the simple NAME Adrià. • The verb phrase is composed of a verb (V) menja and an NP, which consists of an article (ART) el and a common noun (N) bacallà. • In list notation this same structure could be represented as: (S (NP (NAME Adrià)) (VP (V menja) (NP (ART el) (N bacallà) )))

35. Grammars and sentence structure • To construct a tree structure for a sentence, you must know what structures are legal. • A set of rewrite rules describes what tree structures are allowable. • These rules say that a certain symbol may be expanded in the tree by a sequence of other symbols. • A set of rules constitutes a grammar: • S → NP VP • VP → V NP • NP → NAME • NP → ART N • NAME → Adrià • V → menja • ART → el • N → bacallà

36. Grammars and sentence structure • Rule 1 says that an S may consist of an NP followed by a VP. • Rule 2 says that a VP may consist of a V followed by an NP. • Rules 3 and 4 say that an NP may consist of a NAME or may consist of an ART followed by an N. • Rules 5 - 8 define possible words for the categories. • Grammars consisting entirely of rules with a single symbol on the left-hand side, called the mother, are called context-free grammars (CFGs).

37. Grammars and sentence structure • Context-free grammars (CFGs) are a very important class of grammars because: • the formalism is powerful enough to describe most of the structure in natural languages, • yet is restricted enough so that efficient parsers can be built to analyze sentences. • Symbols that cannot be further decomposed in a grammar (the words Adrià, menja…) are called terminal symbols. • The other symbols, such as NP and VP, are called nonterminal symbols. • The grammatical symbols such as N and V that describe word categories are called lexical symbols. • Many words will be listed under multiple categories. For example, poder would be listed under V (can) and N (power). • Grammars have a special symbol called the start symbol. Usually, the start symbol is S (also meaning sentence).

38. Grammars and sentence structure • A grammar is said to derive a sentence if there is a sequence of rules that allow you to rewrite the start symbol into the sentence, for instance, Adrià menja el bacallà. • This can be seen by showing the sequence of rewrites starting from the S symbol, as follows: S => NP VP (rewriting S)=> NAME VP (rewriting NP)=> Adrià VP (rewriting NAME)=> Adrià V NP (rewriting VP)=> Adrià menja NP (rewriting V)=> Adrià menja ART N (rewriting NP)=> Adrià menja el N (rewriting ART)=> Adrià menja el bacallà (rewriting N)

39. Grammars and sentence structure • Two important processes are based on derivations: • The first is sentence generation, which uses derivations to construct legal sentences. A simple generator could be implemented by randomly choosing rewrite rules, starting from the S symbol, until you have a sequence of words. The preceding example shows that the sentence Adrià menja el bacallà can be generated from the grammar. • The second process based on derivations is parsing, which identifies the structure of sentences given a grammar.

40. Parsing as a search procedure • In derivations, there are two basic methods of searching: • A top-down strategy starts with the S symbol and then searches through different ways to rewrite the symbols until the input sentence is generated, or until all possibilities have been explored. The preceding example demonstrates that Adrià menja el bacallà is a legal sentence by showing the derivation that could be found by this process.

41. Parsing as a search procedure • In a bottom-up strategy, you start with the words in the sentence and use the rewrite rules backward to reduce the sequence of symbols until it consists solely of S. The left-hand side of each rule is used to rewrite the symbol on the right-hand side. A possible bottom-up parse of the sentence Adrià menja el bacallà is: => NAME menja el bacallà (rewriting Adrià)=> NAME V el bacallà (rewriting menja)=> NAME V ART bacallà (rewriting el)=> NAME V ART N (rewriting bacallà)=> NP V ART N (rewriting NAME)=> NP V NP (rewriting ART N)=> NP VP (rewriting V NP)=> S (rewriting NP VP) • A tree representation can be viewed as a record of the CFG rules that account for the structure of the sentence.

42. What makes a good grammar • In constructing a grammar for a language, you are interested in: • generality, the range of sentences the grammar analyzes correctly; • selectivity, the range of non-sentences it identifies as problematic; • understandability, the simplicity of the grammar itself.

43. Generative capacity • Grammatical formalisms based on rewrite rules can be compared according to their generative capacity, which is the range of languages that each formalism can describe. • It turns out that no natural language can be characterized precisely enough to define the generative capacity. • Formal languages, however, allow a precise mathematical characterization.

44. Generative capacity • Consider a formal language consisting of the symbols a, b, c and d (think of these as words). • Then consider a language L1 that allows sequences of letters in alphabetical order. For example, abd, ad, bcd and abcd are all legal sentences. To describe this language, we can write a grammar in which the right-hand side of every rule consists of one terminal symbol possibly followed by one nonterminal. • Such a grammar is called a regular grammar. For L1 the grammar would be: S -> a S1S -> dS1 -> dS3 -> d S -> b S2S1 -> b S2S2 -> c S3 S -> c S3S1 -> c S3S2 -> d

45. Generative capacity • Consider another language, L2, that consists only of sentences that have a sequence of a’s followed by an equal number of b’s—that is, ab, aabb, aaabbb, and so on. You cannot write a regular grammar that can generate L2 exactly. • A context-free grammar to generate L2, however, is simple: S -> a bS -> a S b

46. Generative capacity • Some languages cannot be generated by a CFG. • One example is the language that consists of a sequence of a’s, followed by the same number of b’s, followed by the same number of c's - that is, abc, aabbcc, aaabbbccc, and so on. • Similarly, no context-free grammar can generate the language that consists of any sequence of letters repeated in the same order twice, such as abab, abcabc, acdabacdab, and so on. • There are more general grammatical systems that can generate such sequences, however. One important class is the context-sensitive grammar, which consists of rules of the form: α A β→αψβ where A is a symbol, α and β are (possibly empty) sequences of symbols, and ψ is a nonempty sequence of symbols.

47. Generative capacity • Even more general are the type 0 grammars, which allow arbitrary rewrite rules. • Work in formal language theory began with Chomsky (1956). Since the languages generated by regular grammars are a subset of those generated by context-free grammars, which in turn are a subset of those generated by context-sensitive grammars, which in turn are a subset of those generated by type 0 languages, they form a hierarchy of languages (called the Chomsky Hierarchy).

48. Languages associated to Chomsky-hierarchy grammars

49. Grammaticality condition A sentence w (a string of Σ*) pertains to the language generated by grammar G, if grammar G can derive w starting from S, using production rules.

50. Obtaining the grammar • Definition of the terminal tagset (Σ) • Definition of the non-terminal tagset (V) • Definition of grammar rules (P): • Manual construction • Automatic construction • Grammatical inference (induction) • Semiautomatic construction