Steve Pepper 2013-09-28 LCR 2013

1. Steve Pepper2013-09-28LCR 2013 Data mining, transfer and learner corpora Using data mining to discover evidence ofcross-linguistic influence in learner corpora

2. Introduction • This presentation based on Pepper (2012) • Detect cross-linguistic influence using data mining methods • Data from ASK (Norwegian Second Language Corpus) • Learners of Norwegian from 10 L1 backgrounds • Pepper (2012) modelled on Jarvis & Crossley (2012) • Applications of the “detection-based argument” (Jarvis 2010) • Automatically detect the L1 using text classification methods, in particular Linear Discriminant Analysis (LDA) • Major advantages of this approach • Able to recognize positive as well as negative transfer • Able to detect very subtle patterns that otherwise escape notice • Applicable to a wide range of learner corpus problems • Goal today • Explain the methodology in a non-technical way

3. The modal verb skulle is greatly overused by Dutch speakers The masculine form of the indefinite article en is overused by German speakers (and the neuter form et correspondingly underused) On the other hand, the neuter form et is overused by English speakers (and the masculine form correspondingly underused) Polish and Russian speakers exhibit diametrically opposed patterns in their use of the pronouns den and det English speakers overuse the finite form of the copula er Most L1 groups underuse the infinitive form of the copula være, but English speakers overuse it Findings

4. The preposition på ‘on’ is overused by English speakers and underused by German speakers The preposition i ’in’ is overused by Russian speakers and underused by Polish speakers The preposition til ‘to’ is underused by all L1 groups The conjunction eller ‘or’ is overused by English and German speakers, but greatly underused by Russian speakers Most of these patterns appear to be the result of cross-linguistic influence Few (if any) of them have been recognized earlier All of them are suitable topics for further (MA) research I’m not going to talk about them today! Instead I want to talk about the method I used – Linear Discriminant Analysis More findings

5. Linear Discriminant Analysis • A statistical technique which allows the researcher to study the differences between two or more groups of objects with respect to several variables simultaneously (Klecka 1980) • Two kinds of application • Predictive: Purpose is to ascertain group membership based on properties • Descriptive: Purpose is to describe how groups of objects differ, and which variables are the best discriminators • Input to both kinds of application is • a set of CASES (objects), • which can be divided into two or more GROUPS, • and which have FEATURES represented as numeric variables. • So how does it work? Seppo Mustonen explains...

6. Mustonen 1965: Simple example Seppo Mustonen. 1965. “Multiple discriminant analysis in linguistic problems” Statistical Methods in Linguistics 4 Aim:“to indicate the possibilities of applying statistical multivariate analysis to some linguistic problems” Method: “teaching the computer to decide to which language a given word most probably belongs” Caveat: The study was “only for fun” and therefore not to be taken too seriously … except as an example that can be extended “to have practical linguistic applications” Kiitos, Seppo!

7. Mustonen 1965: Data • 900 words (CASES) • 300 from each of English, Swedish and Finnish, chosen at random • 3 GROUPS (1 grouping variable) • Language (value EN, SW or FI) • 43 quantitative variables (FEATURES) • See box (Ex.=‘always’) • “The computer was not given any other special information to improve accuracy of discrimination” • No deterministic rules, e.g. “that a Finnish word never ends with two consonants, that the English alphabet does not contain Å, Ä, Ö, etc.” • Think of the data set as a 900 row by 45 column Excel spreadsheet:

8. Mustonen 1965: Procedure • Compute two “discriminant functions”… z = a1x1 + a2x2 + a3x3 + a4x4 + … + a43x43 (i.e. two sets of values for the coefficients a1, a2 … a43) • … such that the within-group variance is minimized and the between-group variance is maximized z1 = .021x1 + .028x2 + .414x3 + … – .043x43 z2 = –.024x1 + .067x2 – .209x3 + … + .166x43 • Compute z1 and z2 for the 900 cases z1(always) = (0.021· 5) + (0.028· 2) + … – (0.043· 0) z2(always) = (–0.024· 5) + (0.067· 2) + … + (0.166· 0) • Why two functions? • Because there are three groups (n = G–1)

9. z2 FÖRLÅTA ÅTANKE BRÅK JÄMFÖRA ANSTÄLLNING ÖVERENSKOMMA ÄNG ORDNING KARJAKKO TYTTÖ FLAGG KIRKKO z1 SW KÄSIKÄHMÄ FATTA JÄTTÄÄ ÄRA KAPITALIST DRIVER PLUMP MORAL JONINJOUTOVA GIVA HAT FI PERFUME ANEMONE STACK HUNT VAI EN AURINKO SUOPUNKI REMOTE DISEASE HUPPU FEEL ZONE CLUB KIELTÄYTYÄ NIELAISTA CHORUS TOLERANCE CEREMONY MECHANIC CROWD Mustonen 1965: Result • To test the model’s accuracy • New random sample • 100 words from each language • Calculate z1 and z2 for each new word • Plot in [n =] two dimensions • Calculate accuracy • 76.3% success rate • Confusion matrix

10. Mustonen 1965: Discussion • Mustonen achieved an overall success rate of 76.3% • Expected rate according to the null hypothesis: 33.3% • Binomial test: p < 2.2e-16 ➔ result is statistical • This kind of classification cannot compete with deterministic methods based on a strict set of rules • (assuming such rules can be formulated, which in this case they cannot) • Mustonen comments: • Deterministic classification is not very interesting, since it hardly can reveal anything new about the subject. All the information about the rules of classification must be given by the investigator. In the method based on Discriminant Analysis no rules for classification are given in advance (p.44) • Summary • 900 words,3 languages, 43 features ➔ 76.3%CASES GROUPS FEATURES predictive accuracy

11. Jarvis & Crossley 2012 • Jarvis, Castañeda-Jiménez & Nielsen • 446 Charlie Chaplin texts, 5 L1s, 53 1-grams ➔76.9% • Jarvis & Paquot • 2,033 ICLE texts, 12 L1s, 722 n-grams ➔53.0% • Crossley & McNamara • 903 ICLE texts, 4 L1s, 19 indices of cohesion, lexical sophistication, syntactic complexity and conceptual knowledge (Coh-Metrix) ➔67.6% • Bestgen, Granger & Thewissen • 223 ICLE texts, 3 L1s, 46 error types ➔65.0% • Jarvis, Bestgen, Crossley, Granger, Paquot, Thewissen & McNamara • 223 ICLE texts, 3 L1s, combination of 50 n-Grams, 19 Coh-Metrix indices and 46 error types ➔79.4% • Pepper (2012) • 1,000 ASK texts, 5 L1s, 53–55 1-grams ➔51.2–57.6%

12. Analyze > Classify > Discriminant… OK… Demo

13. Other applications in linguistics • Bellinger (1979) – analysis of the pattern of change in mothers’ speech. • Marckworth & Baker (1980) – discriminant function analysis of co-variation of 36 syntactic devices (sentence type, focus phenomena, verb structure, conjoining and embedding, etc.) in five prose genres. • Fletcher & Peters (1984) – exploratory study of language impairment in children. • Fox (1991) – discriminant analysis of yes-no questions in Quebec French. • Nicoladis (1994) – PhD thesis on code-mixing in French-English bilingual children. • Ganschow & Sparks (1996) – psychological study of anxiety about language learning among high school women, which examines the relationship between three levels of anxiety (high, average, low) and nine measures of NL skill and FL aptitude. • Peng & Hengartner (2002) – quantitative analysis of the literary styles of nine authors, including Shakespeare, Dickens, Kipling and London.

14. Prediction and description • Predictive application of DA of limited interest in transfer research • Can be used to demonstrate the existence of transfer • Automatically provides the first two kinds of evidence called for by Jarvis • minimized within-group variance = intragroup homogeneity • maximized between-group variance = intergroup heterogeneity • Descriptive application more interesting • Can describe how L1 groups differ • i.e. which interlanguage features characterize various L1 groups • Stepwise method permits selection of most characteristic features • Further analysis can determine which L1 groups they discriminate between • e.g. Tukey’s HSD or Student-Newman-Keuls • Contrastive Interlanguage Analysis (Granger 1996) can then provide the third kind of evidence • But that’s another story…

15. References Bestgen, Yves, Sylviane Granger & Jennifer Thewissen. 2012. Error patterns and automatic L1 identification. In Jarvis & Crossley, 127‑153. Crossley, Scott & Danielle S. McNamara. 2012. Detecting the first language of second language writers using automated indices of cohesion, lexical sophistication, syntactic complexity and conceptual knowledge. In Jarvis & Crossley, 106‑126. Granger, Sylviane. 1996. From CA to CIA and back: An integrated approach to computerized bilingual and learner corpora. In Karin Aijmer, Bengt Altenberg and Mats Johansson (eds.) Languages in Contrast. Lund: Lund University Press, 37–51. Jarvis, Scott. 2010. Comparison-based and detection-based approaches to transfer research. EUROSLA Yearbook 10, 169‑192. Jarvis, Scott & Scott A. Crossley (eds.) 2012. Approaching Language Transfer through Text Classification. Explorations in the detection-based approach. Bristol: Multilingual Matters. Jarvis, Scott, Gabriela Castañeda-Jiménez & Rasmus Nielsen. 2012. Detecting L2 writers’ L1s on the basis of their lexical styles. In Jarvis & Crossley, 34‑70.Jarvis, Scott & Magali Paquot. 2012. Exploring the role of n-grams in L1 identification. In Jarvis & Crossley, 71‑105. Jarvis, Scott, Yves Bestgen, Scott A. Crossley, Sylviane Granger, Magali Paquot, Jennifer Thewissen & Danielle S. McNamara. 2012. The comparative and combined contributions of n-Grams, Coh-Metrix indices and error types in L1 classification of learner texts. In Jarvis & Crossley, 154‑177. Klecka, William R. 1980. Discriminant Analysis. Quantitative Applications in the Social Sciences 19. London: Sage Publications. Mustonen, Seppo. 1965. Multiple discriminant analysis in linguistic problems. Statistical Methods in Linguistics 4, 37‑44. Pepper, Steve. 2012. Lexical Transfer in Norwegian interlanguage: A detection-based approach. Masters thesis, University of Oslo.