1 / 21

Language Data Resources

About Corpora. Language Data Resources. P. Eisner: “ Znáte jej, ten svůj jazyk? Řekl by přec člověk, že mám-li něco milovat, musím to znát. Vy však češtinu neznáte, a říkám-li to, není to ani obžaloba, ani vůbec výtka. Nemůžete ji znát a obsáhnout, to se dokonale nepodařilo ještě nikomu… “.

sonora
Download Presentation

Language Data Resources

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. About Corpora Language Data Resources

  2. P. Eisner: “Znáte jej, ten svůj jazyk? Řekl by přec člověk, že mám-li něco milovat, musím to znát. Vy však češtinu neznáte, a říkám-li to, není to ani obžaloba, ani vůbec výtka. Nemůžete ji znát a obsáhnout, to se dokonale nepodařilo ještě nikomu…“ J. Sinclair: “Language looks rather different when you look at a lot of it at once.“

  3. Merriam-Webster OnLine:

  4. Corpus • F. Čermák: corpus – a structured, unified (and often also tagged) large collection of language data • T.McEnery: Corpus data – the raw fuel of NLP

  5. Corpus linguistics • A study of language that includes all processes related to processing, usage and analysis of written or spoken machine-readable corpora. Corpus linguistics is a relatively modern term used to refer to a methodology, which is based on examples of ‘real life’ language use • Corpus linguistics is not a language theory.

  6. Corpora classification • A. by medium: • printed, electronic text, digitized speech, video • B. by design method: • balanced vs.special • C. language variables: • monolingual vs. multilingual • original vs. translations • native speaker vs. Learner • D. language evolution: • synchronic vs. diachronic • E. Plain vs. annotated

  7. Balanced corpora (?)‏ • T.McEnery: “Sampling is inescapable.“ • Proportions corresponding to the real language usage • Is that possible? Criteria for choosing styles, genres, and eventually concrete texts? • reception (a few authors, large audience) vs. perception (produkce of a large community of language users)‏ • N. Chomsky: “Any natural corpus will be skewed. Some sentences won’t occur because they are obvious, others because they are false, still others because they are impolite. The corpus, if natural, will be so wildly skewed that the description would be no more than a mere list.“

  8. Corpus size • Brown Corpus – 1 MW (1964)‏ • British Natural Corpus – 100 MW (1994)‏ • http://www.natcorp.ox.ac.uk/ • Cosmas – 1.6 GW (2004)‏ • http://corpora.ids-mannheim.de/cosmas/

  9. Exercise • Could you estimate the amount of Czech texts (measured in running words) available on the Internet?

  10. Antecedents of corpora • Excerption tickets • For Czech systematically from 1911 • Electronic corpus of Czech tests • 1970s • around 500kW

  11. Corpus annotation • K.Pala: “Annotating consist of adding selected linguistic information to an existing corpus of written or spoken language. Typically, this is done by some kind of coding being attached (semi)automatically or manually to the electronic representation of the text.“ • Raw texts: difficult to exploit • solution: gradual „information adding“ (more exactly: adding information in an explicit, machine tractable form), • Annotation  ease of exploitation + reusability

  12. Criticism of corpus annotation • Corpus annotations produce impure corpora • forced interpretations • Consistency vs. Accuracy

  13. Czech National Corpus • http://ucnk.ff.cuni.cz • ÚČNK (Institute of Czech National Corpus) founded in 1994 • diachronous section 13-19th century - DIAKORP • synchronous section – from around 1900 • written language – 100MW v SYN 2000 • spoken language – Prague spoken corpus (PMK), Brno spoken corpus (BMK)‏ • dialects

  14. Czech National Corpus

  15. SYN2000

  16. Preprocessing • Collect textual material • electronic form • scanning+OCR • trend: WWW as a corpus • Conversion and cleaning • Unified format (problém: loosing some information)‏ • Unified encoding (problem: encoding detection)‏ • Document classification • Document segmentation • segmentation on sentence boundaries (problem: tables, direct speech…)‏ • Tokenization on word boundaries (problem: what is a word?)

  17. (Morphological) Tagging • (1) Morphological analysis • For each word form, list all possible lemma+tag pairs (or list of sequences of such pairs, if tokenization is not straightforward)‏ • (2) Disambiguation • choose one lemma+tag pair

  18. Parallel corpora • texts and their translations into another language (or into more languages)‏ • added value - alignment • explicit pairing of corresponding chunks of text • ideally diagonal • often just sentence-level alignment • automatized alignment? • anchor points, word-pairs, … • http://utkl.ff.cuni.cz/~rosen/public/parabrati.ppt

  19. MULTEXT-EAST • Multilingual Text Tools and Corpora for Central and Eastern European Languages • Lexical resources • Entry: word form + lemma + MSD • MSD – morphosyntactic descriptions (Ncms – Noun common masculine singular)‏ • Annotated multilingual corpus • Translations of George Orwell's "1984" , about 100kW • Bulgarian, Czech, Estonian, Hungarian, Romanian, and Slovene, as well as for English (hub language)‏ • (and recently also Croatian, Lithuanian, Resian, Romanian, Russian, Slovene)‏ • Hand-validated sentence alignment • http://nl.ijs.si/ME • version 3 released in 2004 (publically available)‏ • TEI P4 XML

  20. PCEDT • Prague Czech-English Dependency Treebank • http://ufal.ms.mff.cuni.cz/pcedt/ • Czech translation of 21,600 English sentences from the Wall Street Journal part of Penn Treebank 3 corpus • Czech-English corpus of plain text from Reader's Digest 1993-1996 consisting of 53,000 parallel sentences • automatically morphologically annotated and parsed into two levels (analytical and tectogrammatical) of dependency structures • Available via LDC

  21. E. Brill: “More data is more important than better algorithms“ • E. Charniak: “Future is in statistics.“

More Related