Laboratorio di analisi di dati linguistici

1. Laboratorio di analisi di dati linguistici Laurea specialistica in Linguistica Teorica e Applicata, Università di Pavia Andrea Sansò sanso@humnet.unipi.it A.A. 2005-2006 Corso progredito 10 CFU

2. Laboratorio di analisi di risorse linguistiche Parte quarta Lessici Risorse per la linguistica tipologica Strumenti e tecnologie per la creazione di risorse linguistiche

3. Lessici Una definizione: “A computational lexicon is a very complex – and expensive – component to be built adequately. It must contain, in an explicit and formalised way, all the information which a native speaker uses in everyday situations, from the simpler orthographic, phonetic, morphologic information, to the more complex syntactic, semantic, pragmatic, logical, ontological, multilingual information. A ‘complete’ lexicon should practically incorporate our ‘knowledge of the world’, and represent it in an explicit and formal way” N. Calzolari, “Computational lexicons and corpora. Complementary components in human language technology”, in P. van Sterkenburg (ed.), Linguistics Today – Facing a Greater Challenge, 89-107. Amsterdam-Philadelphia: J. Benjamins, 2004.

4. Lessici • Lessici   corpora • Questa conoscenza del mondo è un oggetto mutevole e continuamente in accrescimento, impossibile da “congelare” in un lessico statico. • The only way of reflecting and capturing all the potentialities of a language relies on trying to extract the linguistic and lexical information not only from ‘experts’, i.e. native speakers or linguists, but from the texts themselves in which the language is actually used, with a continuous process of enrichment. From these considerations the importance of corpora obviously emerges. N. Calzolari, ibidem

5. Lessici Lessici   corpora L  C POS tagging / lemmatisation C  L frequencies of different linguistic objects C  L proper nouns / named entity recognition L  C syntactic parsing C  L updating / tuning a lexicon C  L collocational data

6. Lessici Lessici   corpora C  L semantic clustering and ‘nuances’ of meaning L  C semantic mark-up C  L lexical knowledge acquisition L  C word sense disambiguation C  L validation of lexical models C  L corpus-based computational lexicography

7. Lessici • Lessici   corpora • Esempio: italiano chiedere vs domandare • Dal punto di vista teorico (introspettivo) sono sinonimi; i dizionari cartacei utilizzano la stessa definizione • Ma: • domandare è utilizzato quasi sempre in senso interrogativo (ask to know); chiedere è utilizzato spesso in senso imperativo (ask to have); • chiedere è molto più usato di domandare;

8. FrameNet FrameNet (FN) is a corpus-based lexicon-building project that documents the links between lexical items and the semantic frame(s) they evoke; it accomplishes this by annotating sets of sentences that exemplify the items being described, and performing various operations on the resulting annotations. The basic units in FN descriptions are the frame and the lexical unit (LU), the latter understood as the pairing of a “word” with just one of its meanings; thus, a word with four meanings is treated as four lexical units. In most cases, for a word to have more than one meaning implies that it belongs to more than one frame.  Charles J. Fillmore, Collin F. Baker, and Hiroaki Sato, “FrameNet as a ‘Net’”, in Proceedings of the 4th International Conference on Language Resources and Evaluation, Lisbon 2004, pp. 1091-1094.

9. FrameNet • Main components of the FrameNet database • the frame ontology, • the set of annotated sentences, and • the set of lexical entries. The basis of the ontology is the set of frames, each of which consists of an informal characterization of a situation type (the frame definition), together with a collection of frame elements (FEs). The FEs are the semantic roles of the entities involved in each frame. FE names are used as labels for the words or phrases that are in grammatical construction with the L(exical) U(nit)s that evoke that particular frame. For example, the frame that includes the English verb inform has as its core FEs SPEAKER, ADDRESSEE and MESSAGE.

10. FrameNet • The example sentences are selected by FrameNet annotators as representing the typical uses of the LUs belonging to individual frames. Each set of annotations is centered around a particular LU; the sentence’s constituents are labeled (with FE names) according to the ways in which they fill in information about the frame. For example, sentences (1) and (2) have SPEAKER appearing as subject, and ADDRESSEE as object; the MESSAGE FE appears as a that clause in sentence (1), and as an event-naming nominalization introduced by of in sentence (2). • [SPEAKER We] informed [ADDRESSEE the press] [MESSAGE that the prime minister has resigned] • [SPEAKER We] informed [ADDRESSEE the press] [MESSAGE of the prime minister’s resignation]

11. FrameNet • The lexical entry for each LU is a summary of what has been recorded in its annotations, presented as valence descriptions, showing all the ways in which its frame elements can be realized, such as the alternative syntactic realizations of the MESSAGE just shown for the verb inform. The collection of annotated sentences is made available in the database as evidence for the analysis. • The first and most obvious way in which LUs are related to each other is through membership in the same frame. Thus inform shares a frame with the verbs notify and announce, and also with the nouns notification and announcement, and the verb resign shares frame membership with its nominal partner resignation, and with verbal expressions like abdicate, step down and stand down. But LUs can also be related to each other in other ways, either because their frames are related to other frames, or through semantic properties (called semantic types in the FN database) assigned to LUs individually rather than through their frames.

12. FrameNet Semantic types: • The FrameNet database allows the assignment of semantic types to LUs, FEs and frames. The perception verbs hear vs. listen are distinguished as passive versus active perception verbs, and so, respectively, are see vs. look. Hearing and seeing are things that happen to you, listening and looking are things that you do, and this difference is considered important enough to merit entry into separate frames. In the FN database, hear and see and the passive perception uses of other sensory words, such as feel, taste and smell, belong to the Perception experience frame; the verbs look and listen belong to the Perception active frame, along with the corresponding active uses of feel, taste and smell.

13. FrameNet Subframes Subframes are used for representing subevents; frames that represent complex processes have subframes representing their subparts. To take a simple example, the Motion scenario frame has three subframes, Departing, Motion, and Arriving. In this case, the subframes are temporally ordered, but in general, subframes need not be completely ordered with respect to each other. For example, the Commercial transaction frame has two subframes Commerce goods-transfer and Commerce money-transfer, but these are not ordered with respect to each other. In some commercial transactions, you pay in advance, in others, only after receiving the goods or services.

14. FrameNet in azione… http://framenet.icsi.berkeley.edu/index.php Tutta la documentazione si trova in un manuale: http://framenet.icsi.berkeley.edu/index.php?option=com_wrapper&Itemid=126

15. FrameNet in altre lingue Salsa Project – FrameNet in German http://www.coli.uni-saarland.de/projects/salsa/ Spanish FrameNet http://gemini.uab.es/SFN/index.html

16. WordNet Sistema di riferimento lessicale disponibile online: http://wordnet.princeton.edu I significati delle parole sono rappresentati da gruppi di sinonimi (synsets). Sono rappresentate anche relazioni quali meronimia, iperonimia, antonimia, etc. Bibliografia aggiornata: http://mira.csci.unt.edu/~wordnet/

17. Altri lessici multilingui Mimida: http://www.gittens.nl/SemanticNetworks.html Un lessico multilingue basato su WordNet e su vocabolari liberamente disponibili sul web. MultiWordNet: http://multiwordnet.itc.it/english/home.php Un lessico multilingue (italiano, spagnolo, ebraico, rumeno) in cui i synsets sono allineati, laddove possibile, con i synsets del WordNet di Princeton. Sviluppato all’IRST-ITC di Povo (TN).

18. EuroWordNet http://www.illc.uva.nl/EuroWordNet Un progetto analogo per le lingue europee: è possibile scaricarne una demo I vari WordNets sono collegati ad un Interlingual index che è basato sul Wordnet americano e che permette di passare da una parola in una lingua a una parola analoga in un’altra. Questo index consente anche di accedere a un’ontologia condivisa di 63 distinzioni semantiche, che fornisce una base semantica comune per le varie lingue

19. Altre iniziative • Progetto EAGLES (Expert Advisory Group for Language Engineering Standards): http://www.ilc.cnr.it/EAGLES96/home.html • development of standards in morphosyntax, syntax and semantics • awareness of the interdependence between lexical specifications and corpus tagsets / syntactic annotations • gli standard sviluppati sono serviti nella creazione di risorse (sia corpora che lessici) creati all’interno dei progetti europei Parole e Simple

20. Altre iniziative • Progetto ISLE (International Standards for Language Engineering) – Computational Lexicon Working Group: • http://www.ilc.cnr.it/EAGLES96/isle/ISLE_Home_Page.htm • una continuazione del progetto EAGLES • sviluppo di uno schema generale per la codifica dell’informazione lessicale multilingue (MILE; Multilingual ISLE Lexical Entry) • impegno a raggiungere consenso su standard di fatto attraverso una procedura bottom-up • impegno a massimizzare l’interazione e le sinergie con chi lavora nell’ambito del semantic web

21. Altre iniziative • Progetto PAROLE: http://www.ub.es/gilcub/SIMPLE/simple.html • obiettivo: produrre in Europa un nucleo iniziale di corpora e lessici armonizzati (catalano, danese, olandese, inglese, finlandese, francese, tedesco, greco, italiano, portoghese, spagnolo, svedese) • Informazione codificata: • Morfologia: written forms, including stems and variants; morphosyntactic category; inflected forms; morphological features; derivation; abridged forms

22. Altre iniziative • Progetto PAROLE: • Informazione codificata: • Sintassi: subcategorization patterns; grammatical relations of subcategorised complements; control; diathesis and lexical alternations; pronominalization; linear order constraints; constraints on the syntactic context where the lexical entry is inserted; idioms and collocations

23. Altre iniziative • Progetto SIMPLE: http://www.ub.es/gilcub/SIMPLE/simple.html • Aggiunta di un livello semantico a PAROLE • “The first attempt to tackle harmonised encoding of semantic types and semantic (subcategorisation) frames on a large scale, i.e. for so many languages and with wide coverage”

24. Altre iniziative • Progetto SIMPLE: • Informazione semantica: semantic type; domain information; lexicographic gloss; argument structure for predicative semantic units; event type, to characterise the aspectual properties of verbal predicates; links of the arguments to the syntactic subcategorization frames; ‘qualia’ structure, represented by a very large and granular set of semantic relations and features; regular polysemous alternations (e.g. container for content); hyponymy, synonymy, etc.

25. Due tipi di database tipologici Databases that collect and document primary language data e.g. Agreement database Autotyp Reflexives and intensifiers database Stresstyp... Databases documenting secondary language data e.g. Noun Phrase Universals Database (Edinburgh) The Universals Archive (Konstanz) Das grammatikalische Raritätenkabinett (Konstanz)  http://ling.uni-konstanz.de/pages/proj/sprachbau.htm

26. Database tipologici http://www.lotschool.nl/Research/ltrc/databases/index.htm contiene un elenco dei database tipologici elaborati all’interno del progetto LTRC (Utrecht) Particolarmente user-friendly: Typological Database of Intensifiers and Reflexives (TDIR): http://noam.philologie.fu-berlin.de/~gast/tdir/index.htm Reduplication database: http://ling.uni-graz.at/redup/ The SMG databases: http://www.smg.surrey.ac.uk/

27. Database tipologici World Atlas of Language Structure The World Atlas of Language Structures consists of 142 maps with accompanying texts on diverse features (such as vowel inventory size, noun-genitive order, passive constructions, and 'hand'/'arm' polysemy), each of which is the responsibility of a single author (or team of authors). Each maps shows between 120 (35) and 1110 languages, each language being represented by a dot, and different dot colors showing different values of the features. Altogether 2,650 languages are shown on the maps, and more than 58,000 dots give information on features in particular languages Tools per la ricerca tipologica: http://lingweb.eva.mpg.de/fieldtools/tools.htm

28. Strumenti e tecnologie per la creazione di risorse Tools specializzati Fieldwork: Shoebox: http://www.sil.org/computing/shoebox Fieldworks Data Notebook: http://fieldworks.sil.org(open source) Speech analysis: Praat: http://fonsg3.hum.uva.nl/praat(gratuito) SpeechAnalyzer: http://www.sil.org/computing/speechtools/speechanalyzer.htm (versione 2.1 non gratuita; versione 1.5 gratuita) Annotation tools: CLAN: http://childes.psy.cmu.edu(gratuito) Altri strumenti si possono trovare sulla pagina del LARL, nei link (categorie: concordancing tools e altre risorse linguistiche)

29. Strumenti e tecnologie per la creazione di risorse Tools specializzati Tagger morfologici: Morph-it – tagger morfologico dell’italiano ; disponibile una demo in rete sul sito: http://sslmitdev-online.sslmit.unibo.it/linguistics/morph-it.php POS taggers: CLAWS: www.comp.lancs.ac.uk/computing/research/ucrel/claws/trial.html TREE tagger: www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/DecisionTreeTagger.html

30. Strumenti e tecnologie per la creazione di risorse Tools specializzati Codifica di testi DBT (DataBase Testuale): software di analisi testuale e interrogazione full-text sviluppato da E. Picchi (ILC, CNR, Pisa) http://www.ilc.cnr.it/pisystem/demo/index.html Il LARL possiede un corpus di italiano L2 e il corpus del LIP (Lessico di frequenza dell’italiano parlato) interrogabili attraverso il DBT