Representatıvness, balance and samplıng ın a corpus Lınguistıcs

1. Representatıvness, balance and samplıng ın a corpus Lınguistıcs

2. Introduction • A corpus is designed to respresent a particular language or language variety. It is impossible to analyse every extant utterance or sentence of a given language. • Sampling is unavodiable. • How can you be sure that the sample you are studying is representative of the language or language variety under consideration?

3. One must consider • Balance • Sampling • to ensure representativness

4. Representativeness in Corpus Linguistics • Biber (1993:242) : “Representativness refers to the exent to which a sample includes the full range of variability in a population.” • Population: a sample of a language or a language variety in a corpus.

5. 2 factors to determine the representativeness of acorpus • The range of genres included in a corpus (i.e. balance). • How text chunks for each genre are selected (i.e sampling).

6. Criteria used to select texts for a corpus • External : situational; genre / register • Internal: linguistic / text types

7. Internal criteria is problematic and circular • Internal criteria like the distribution of words or grammatical features cannot be the primary parameters for the selection of corpus data. • The corpus has been skewed by design.

8. Sinclair (1995): The texts or parts of texts to be included in a corpus should be selected according to external criteria so that their linguistic characteristics are independent of the selection process.

9. Another aspect of representativeness: Change over time • View of a corpus as a • Static (It applies to a sample corpus) • Dynamic (It applies to a monitor corpus) • language model

10. Static view of language • Helsinki Diachronic Corpus • Lancester — Oslo / Bergen Corpus (LOB) • Freiburg — LOB (FLOB) • Sampling frame • LOB : British English, early 1960s • FLOB : British English, early1990s

11. Representativeness of General and Specialized Corpora • General Corpora (e.g., BNC) serve as a basis for an overall description of a language or language variety. It should involve samples from a broad range of genres. • Specialized Corpora tend to be domain (e.g., medicine or law) and genre (fiction, newspaper texts, academic prose) specific.

12. Balance • The range of text categories included in the corpus determines how balanced the corpus is. • The acceptable balance of corpus is determined by its intended uses. • A balanced corpus usually covers a wide range of text categories which are supposed to be representative of a language or language variety under consideration. • These text categories are typically sampled proportionally for inclusion in a corpus.

13. Balance • There is no scientific measure of corpus balance. • A more typical approach to corpus balance is that corpus-builders adopt an existing corpus model when building their own corpus, assuming that balanced will be achieved from the adopted model. • BNC is generally accepted as being a balanced corpus. • American National Corpus • Korean National Corpus • Polish National Corpus • Russian Reference Corpus

14. Balance • Balance is a more importan issue for a static sample corpus than a dynamic monitor corpus. • The builders of monitor corpora finds size of corpora more important than the balance of it. They assume that corpus will balance itself when it reaches a substantial size.

15. Composition of written BNC

16. Composition of spoken BNC

17. Sampling • To obtain a representative sample from a population: • define the sampling unit and boundaries of population • Written texts: sampling units are books, periodicals, or newspaper. • List of sampling units is referred to as sampling frame.

18. LOB corpus Target population: All written English texts published in the United Kingdom in 1961 Sampling frame: British National Bibliograhpy Cumulated Subjet Index 1960-1964, for books Willing’s Press Guide 1961, for periodicals

19. Sample size • Full texts (i.e., whole texts) • Text chunks (ideal: sample text fragments, 2,000 running words)

20. A population can be defined in terms of • Language production (demographically oriented) • Language reception (demographically oriented) • Language as a product (text category / genre of language)

21. Different samplingtechniques • Simple random sampling: all sampling units within the sampling frame are numbered and the sample is chosen by the use of table of random numbers • Stratified random sampling, first divides the whole population into relatively homogeneous groups (strata) and samples each stratum at random. • Demographic sampling (i.e., categorize sampling units on the basis of speaker/writer age,sex and social class) is also a type of stratified sampling.

22. Stratified Ramdom Sampling: Brown and LOB Corpora The target population for each corpus was first grouped into 15 text categories such as news reportage, academic prose, different types of fiction. Samples then drawn from each text category.

23. Conclusion • In constructing a balanced, representative corpus, • Stratified random sampling is to be preferred. • For written texts, a text typology established on the basis of external critearia is relevant. • For spoken data, demographic sampling is appropriate. It must be complemented by context-goverend sampling so that some context goverend linguistic variations can be included in the resulting corpus.