1 / 45

Natural Language Processing winter / fall 2010/2011 41.4268

Natural Language Processing winter / fall 2010/2011 41.4268. Prof. Dr. Bettina Harriehausen-Mühlbauer Univ. of Applied Science, Darmstadt, Germany www.fbi.h-da.de/~harriehausen b.harriehausen@fbi.h-da.de Bettina.Harriehausen@h-da.de. the past - the present - the future.

nicole
Download Presentation

Natural Language Processing winter / fall 2010/2011 41.4268

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Natural Language Processingwinter / fall 2010/201141.4268 Prof. Dr. Bettina Harriehausen-Mühlbauer Univ. of Applied Science, Darmstadt, Germany www.fbi.h-da.de/~harriehausen b.harriehausen@fbi.h-da.de Bettina.Harriehausen@h-da.de

  2. the past - the present - the future What does Star Trek have to do with NLP ?

  3. What is NLP / Computational Linguistics ?

  4. What is NLP / Computational Linguistics ? • definition: • A system is called a natural language processing system when • a subset of the input or output of the system is coded / written in a natural languageand • the processing of the data is performed by algorithms for the morpho-syntactic, semantic, and pragmatic analysis or generation of natural language

  5. Natural Language Processing is ... • an interdisciplinary field / art / science • computer science (A.I.) • linguistics (language independent) • mathematics (logics, predicate logic, knowledge based systems, statistics, ...) • psychology (cognitive science) • physics (speech recognition, spoken language) • ...

  6. Natural Language Processing is ... • a broad field / art / science • phonetics / phonology (speech processing / speech recognition)phonemes = the smallest meaning-distinguishing items • morphology (segmentation , compounding,...) - tokenization morphemes = the smallest items carrying meaning • lexicology / electronic dictionaries – tagginglexemes , lemmas vs. full-forms (each entry needs a tag) idiomatic expressions , neologisms / „trendy words“ , homonyms , … • syntax (analysis and generation of phonemes, morphemes, lexemes, phrases, sentences, paragraphs) …grammar / - formalisms (from transformation to unification) • semantics (meaning, disambiguation, anaphora resolution,...) • pragmatics (discourse representation)

  7. We will focus on...(1) • intro • morphology • parsing / tokenization • compounds • lexicon / electronic dictionaries • lemmas / inflected forms • coding features / tagging • idiomatic expressions • neologisms / „trendy words“ • homonyms

  8. We will focus on...(2) • syntax -> semantics : from transformation to unification (RTN / ATN), case grammar (Fillmore) , CD-structures • machine translation • data mining / text mining • speech recognition

  9. We will focus on...(3) dictionary grammar parser

  10. We will focus on...(4) together, we want to: • get an overview of and understand the scope of NLP • get an overview of the state-of-the-art technologies (subset) • understand the parallels between CL and NLP and A.I. • reach the ability to use principles of linguistic theories in NLP programming

  11. reading material Latest edition: Prentice Hall, 2008 ISBN-10: 0131873210, ISBN-13: 978-0131873216 First chapter: http://www.cs.colorado.edu/~martin/SLP/Updates/1.pdf

  12. reading material http://cognet.mit.edu/library/books/view?isbn=0262133601 MIT Press, 1999, ISBN 0262133601 Reader link: http://www.amazon.de/gp/reader/0262133601/ref=sib_dp_pt/028-2523061-0018166#reader-page

  13. more...reading material (A.I.) • Bobrow, D.G., Winograd, T. „An Overview of KRL, a Knowledge Representation Language“ in: Cognitive Science, Vol.1, No.1, 3-46, 1977 • Charniak, E. „A common representation for problem solving and natural language comprehension information“. Artificial Intelligence, 1981, 225-255. • Friedman, J.A. Computer Model of Transformational Grammar. New York: Elsevier. 1971. • Christopher D. Manning (Author), Prabhakar Raghavan (Author), Hinrich Schütze (Author). Introduction to Information Retrieval. Cambridge University Press. 2008. ISBN-10: 0521865719 ISBN-13: 978-0521865715 • Norvig, Peter. Unified Theory of Inference for Text Understanding. Univ. of California, Berkeley, Computer Science Division. Report. No. UCB/CSD 87/339. 1987. • Quillian, M.R. „Sematic Memory“. In: M.Minsky, ed. Semantic Information Processing. MIT Press. Cambridge. 1968. more

  14. more...reading material (A.I.) • Stuart Russell (Author), Peter Norvig (Author) Artificial Intelligence: A Modern Approach (2nd Edition) (Prentice Hall Series in Artificial Intelligence). Prentice Hall, 2002. ISBN-10: 0137903952 • Schank, R.C. Conceptual Information Processing. Amsterdam: North Holland. 1975. • Schank, R.C., Abelson, R.P. Scripts, Goals and Understanding: An Inquiry into Human Knowledge Structures. Hillsdale: Lawrence Erlbaum Associates. 1977. • Wilensky, R., Arens, Y. PHRAN: A knowledge-based approach to natural language analysis. Electronics Research Laboratory, College of Engineering. University of California, Berkeley. Memorandum No. UCB/ERL M80/34. 1980. • Wilensky, Robert. „Some Problems for proposals for Knowledge Representation“. University of Berkeley, CS Dept. 1986. • Woods, W.A. „What‘s a link: Foundations for Semantic Networks“. In: Representation and Understanding: Studies in Cognitive Science. D.G. Bobrow, A. Collins, eds. New York: Academic Press, 1975. more

  15. more...reading material (NLP) • Bresnan, Joan, ed. The mental Representations of Language. London: MIT Press. 1982. • Bresnan, Joan. Lexical Functional Grammar. Stanford Linguistic Institute. 1987 • Chomsky, Noam. Aspects of the Theory of Syntax. Cambridge: MIT Press. 1965. • Ronen Feldman (Author), James Sanger (Author) The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data (Hardcover). Cambridge University Press. 2006. • Fillmore, Charles. The Case for Case. Ohio State University, 1968. • Fillmore, Charles. „The case for Case reopened“. In: P. Cole, J.M. Saddock, eds. Syntax and Semantics 8: Grammatical Relations. Academic Press, N.Y. 1977. • Harriehausen, B. „Why grammars need to expand their scope of parsable input“, Proceedings Second Conference on Arabic Computational Linguistics, Kuwait, 11/89. • Harriehausen, B. „The PLNLP Grammar checkers - CRITIQUE“, Proceedings ALLC-ACH 90 Conference „The New Medium“. Siegen. 6/1990. • Harriehausen-Mühlbauer, B. „PLNLP - a comprehensive natural language processing system for analysis and generation across languages“, Proceedings: The First International Seminar on Arabic Computational Linguistics, Egyptian Computer Society, Cairo, 6/92. more

  16. more...reading material (NLP) • Harriehausen-Mühlbauer, B,. Koop, A. „SCRIPT - a prototype for the recognition of continuous, cursive, handwritten input by means of a neural network simulator“, Proceedings 1993 IEEE International Conference on Neural Networks, San Francisco, 3/1993. • Jurafsky, Daniel, and James H. Martin. 2008. Speech and Language Processing: An Introduction to Natural Language Processing, Speech Recognition, and Computational Linguistics. 2nd edition. Prentice-Hall. • Manning, Christopher / Schütze, Hinrich. Foundations of Statistical Natural Language Processing. MIT Press. 1999. • Levin, L., Rappaport, M., Zaenen, A., eds. Papers in Lexical Functional Grammar. Bloomington: Indiana University Linguistics Club. 1983. • Ruslan Mitkov (Editor) The Oxford Handbook of Computational Linguistics (Oxford Handbooks in Linguistics). Oxford University Press. 2005 . • Radford, A. Transformational Syntax. Cambridge: Cambridge University Press. 1981. • Rieger, C.J. „Conceptual Memory and Inference“. In: R.C. Schank. Conceptual Information Processing. North Holland. 1975. • Shieber, S.M. An Introduction to Unification-based Approaches to Grammar. Stanford: CSLI. 1986. • Winograd, T. Phenomenological Foundations of AI in Language.Stanford University, Linguistic Institute, 1987.

  17. history of NLP / CL How did it all start ? • 1949-1960 beginning of electronic language processing: machine translation, linguistics data processing The spirit is strong but the flesh is weak. -> The vodka is strong but the meat is rotten.

  18. history of NLP / CL How did it all start ? • 1960-1970 first formal (transformation) grammars (Chomsky 1957), beginning of language oriented research in A.I.: first simple question-answering- systems; keyword (pattern-matching)- systems • 1963 Sad-Sam (Lindsay), BASEBALL (Green) • 1966 DEACON (Craig), ELIZA (Weizenbaum), SYNTHEX (Simmons et.al.) • 1968 TLC (Quillian), SIR (Raphael), STUDENT (Bobrow), CONVERSE (Kellog)

  19. ELIZA – pattern-matching(1) • ELIZA is a computer program devised by Joseph Weizenbaum (1966) that simulates the role of a Rogerian psychologist. • ELIZA was one of the first programs developed that explored the issues involved in using natural language as the mode of communication between humans and the machine.

  20. ELIZA – pattern-matching(2)Why Simulate a Rogerian Psychologist? Client-Centered Therapy (CCT), was developed by Carl Rogers in the 40's and 50's and is described as being a"non-directive" approach to counselling. That is, unlike most other forms of counselling, the therapist does not offer treatment,disagree, point outcontradictions, or make interpretations or diagnoses. Instead, CCT is founded on the belief that peoplehave the capacity to figure out their own solutions which can be facilitated by a psychologist who provides an accepting andunderstanding environment. As pointed out by Weizenbaum, "[this form of] psychiatric interview is one of the few examples ofcategorized dyadic natural language communication in which one of the participating pair is free to assume the pose of knowingalmost nothing of the real world." For example, an appropriate response to a client's comment of "I went for a long walk„could possibly be "Tell me about long walks." In this reply, the client would not assume that the therapist knew nothing aboutlong walks, but instead, had some motive for steering the conversation in this direction. Such assumptions make this anappealing domain to simulate, as a degree of realism can be obtained without the need for storing explicit information about thereal world.

  21. ELIZA – pattern-matching(3)How successful is ELIZA ?

  22. ELIZA – pattern-matching(4)How does ELIZA work? • identifying keywords or phrases that the user inputs • using patterns associated with these phrases to generate responses • the most basic of these output patterns respond identically to all sentences containing the keyword

  23. ELIZA – pattern-matching(5)How does ELIZA work? single keywords triggering a response: key: xnone 0 answer: I‘m not sure I understand you fully- answer: That is interesting. Please continue. key: sorry answer: Please don‘t apologise. answer: Apologies are not necessary. xnone = ELIZA responds to an input sentence that is not understood (xnone is the default used when no other keyword is found in the sentence) sorry = ELIZA responds to an input sentence that contains the word „sorry“

  24. ELIZA – pattern-matching(6)How does ELIZA work? keyphrases triggering a response with a conversion: key: I like xxx. (where xxx is an arbitrary string) answer: Why do you like xxx ? answer: Why do you say you like xxx ? Example: user: I like xxx. ELIZA: Why do you like xxx?

  25. ELIZA – pattern-matching(7)How does ELIZA work? keyphrases triggering a response with a conversion: key: I amxxx. (where xxx is an arbitrary string) answer: Tell me why you thinkyou arexxx . Example: user: I amvery unhappy at the moment. ELIZA: Tell me why you thinkyou arevery unhappy at the moment.

  26. ELIZA – pattern-matching(8)How does ELIZA work? keyphrases triggering a response with a conversion plus postprocessing of reference words: key: remember decomp: * I remember * answer: Do you often think of (2) ? answer: What else do you recollect ? Example: user: I remember my first boyfriend. Decomposition: the first * = empty string, the second * = my first boyfriend (= (2)) ELIZA: Do you often think of (* my )your first boyfriend.

  27. ELIZA – pattern-matching (9) Now it‘s your turn ! Try out ELIZA, make up your own mind as to ELIZA‘s realism. Get a first idea of man-machine communication.

  28. ELIZA – pattern-matching(10) to „play“ with ELIZA (see: the following links) ELIZA program:  http://www.manifestation.com/neurotoys/eliza.php3 http://www-ai.ijs.si/eliza-cgi-bin/eliza_script http://www-ai.ijs.si/eliza/eliza.html Reading:  http://i5.nyu.edu/~mm64/x52.9265/january1966.html

  29. but now back to the history of NLP / CL

  30. history of NLP / CL How did it all start ? • 1970-1980 knowledge-based expert systems and natural language database interfaces, development of formal grammars (esp. syntax analysis)dialogue systems1972 SHRDLU (Winograd)1977 GUS (Bobrow et.al.), PAL (Sidner et.al.)natural language interfaces1972 LUNAR (Woods et.al.)1972-1976 RENDEVOUZ (Codd), REL (Thompson), REQUEST (Plath)1977 LIFER (Henrix), INTELLECT (Harris), PLANES (Waltz et.al.), CO-OP (Kaplan)

  31. Natural language DB interface LanguageAccess (natural language interface to a relational database) Sentence xy: WHICH COUNTRY EXPORTS FISH SQL-query: SELECT DISTINCT X1 COUNTRY, X1.PRODUCT FROM EXPORTBASE X1 WHERE X1.PCLASS=„FMF“

  32. history of NLP / CL How did it all start ? text “understanding“ and text generating systems1975 MARGIE (Schank et.al.), SAM (Schank et.al.)1976-1979 TALE-SPIN (Meehan), PAM (Wilensky), FRUMP (DeJong) 1980 PHRAN (Wilensky) • 1980-1990 focus on semantic-pragmatic analysis, natural language applications, models of complex communication pattern • - robust dialogue systems • - integration of natural language components in expert systems • - knowledge acquisition via natural language (both man and machine learn)

  33. history of NLP / CL How did it all start ? • 1990-2000+ machine translation (revival), data mining / text mining, intelligent text processing systems (text critiquing), integration of computerlinguistic components in multimedia (CALL, CBT, TELL,...)... • boom (integration of NLP everywhere) • that‘s where we are today: • growing demand • growing size of applications • growing user expectations

  34. NLP / CL today • we have come very far, • but... ...there are still a lot of open questions: • what is knowledge ? • when do we have to consider knowledge in natural language processing ? • how can knowledge be formalized ? • how are the analysis of language and the understanding of language interrelated ? • what is communication ? • easy (?) natural language • technical language as a „dialect“ of natural language (e.g. medical language) • artificial language as „meta language“ (e.g. Esperanto) • logics (a special form of representation on an abstract level)

  35. Question : Natural language ... easy ? • …after all… we all use / speak / write it Does this mean natural language is easy and easy to formalize ?

  36. Natural language ... easy ? Little Red Ridinghood Rotkäppchen Do you remember the story of the little girl that wore a redcape and which met a wolf while going to her grandmother‘s house ? What‘s the problem ? a little girl -> in German, -chen is the diminutive Don -> Donny ; Kate -> Katie , Bill -> Billy

  37. Natural language ... easy ? other application: natural language database query LanguageAccess (natural language interface to a relational database) Sentence xy: WHICH COUNTRY EXPORTS FISH natural language paraphrase / disambiguation of the input: Which interpretation did you mean ? Which country exports the product fish (fish = object) Which country is exported by fish (fish=subject) in German: with zero-article, it‘s ambiguous (disambiguation by case marking of the article) SQL-query: SELECT DISTINCT X1 COUNTRY, X1.PRODUCT FROM EXPORTBASE X1 WHERE X1.PCLASS=„FMF“

  38. Natural language ... easy ? SENTENCE XY: Who placed as many software orders as Garzillo? SQL-query: SELECT DISTINCT X.1 NAME, X1.PURCHASENUMBER FROM PURCHASES X1, ORDERS X2 WHERE X1.PURCHASERNUMBER=X2.PURCHASERNUMBER GROUP BY X1.PURCHASERNUMBER, X1.NAME HAVING COUNT (*) => (SELECT COUNT (*) FROM ORDERS X3, PURCHASERS X4 WHERE X3.PURCHASERNUMBER = X4.PURCHASERNUMBER AND X4.NAME=‘Garzillo‘ GROUP BY X3.PURCHASERNUMBER)

  39. Natural language ... easy ? • language is extremely ambiguous • easy for humans ??? easy for machines ??? • lexical The pipe was brandnew. • structural I saw the man with the telescope. • deep structural She got ready for the picture. • semantic Mary wants to get married to an Italian. • pragmatic While walking from the gate to the house it collapsed.

  40. Natural language ... easy ? language is complex...you can say a lot with a few words Mary sold John a book. surface structure (obvious): transfer of book deep structure (implication of „to sell“): transfer of money

  41. Natural language ... easy ? language can do a lot....e.g. with conjunctions NP–NP I am eating a hamburger and a pizza. VP-VP I will eat the hamburger and throw away the pizza. S-S I eat a hamburger and Bill eats a pizza. PP-PP I eat a pizza with ham and with salami. ADJP-ADJP I eat a cold but delicious hamburger. ADVP-ADVP I eat the hamburger slowly and patiently. V-V I bake and eat a hamburger. AUX-AUX I can and will eat a hamburger. and even more.... ???-??? Mary is sitting on and Bill under the table.

  42. Natural language ... easy ? language is analyzed on different levels

  43. The 7 levels of language understanding Needed knowledge: features of the voice phonetic analysis sound combinations of language phonological analysis dictionary morphological / lexical analysis grammar rules (parser) syntactic analysis knowledge representation semantic analysis world knowledge pragmatic analysis acoustic signals æ ç Þ ð ţ ş sounds Bill ... letters Billy... words Billy is eating his lunch. sentences small (Billy) knowledge [Billy]is[mother] consequences child of

  44. Natural language ... easy ? Why then natural language ? Computers speak their own language. This language is efficient, economical, and exact. Why then would we want to „teach“ the computer a natural language with all its ambiguities and difficulties ? when you don‘t want to learn a database query language to get data (Startrek) (textanalysis, textgeneration, machine translation) when you don‘t want to learn a programming language to program your computer (machine translation) when busy with your hands and you still want to type (voice type) when travelling (machine translation) when you need to make a phonecall with someone in Japan, but you don‘t speak Japanese (voice recognition, machine translation) when you are a slow typer (voice type) Back to the boom! when you want to evaluate millions of lines of text (text/data mining)

  45. Applications applications of natural language systems spoken text written text Speech input Speech output dialogue understanding text analysis generation Dialogue Systems translation • spell aid • text critiquing • text summaries • knowledge acquisition (e.g. for expert systems) • speaker & voice recognition • spoken commands /command &control • automatic dictation • text-to-speech • telephony • IVR (interactive voice response) • help functions for translations • automatic translation • simultaneous translation • explanations for users • knowledge representation • text generation • writing support • information systems • DB query • expert systems • CALL • robot stearing • programming languages

More Related