Evaluating semantic similarity and sameness in studies of polysemy and synonymy

1. Evaluating semantic similarity and sameness in studies of polysemy and synonymy Jarno Raukko (U. Helsinki)

2. For a full version of the PPT, seehandoutdistributedOct 28, 2010. SKY webpage version

3. Examples Are thrifty and stingy synonyms? EXPECTED ANSWER: ”Well, not quite.” 2. Are violin and fiddle synonyms? EXPECTED ANSWER: ”Well, almost.” (SYNONYMY) 3. Does back have the same meaning in My back hurts and I came back? EXPECTED ANSWER: ”Not at all. Different.” 4. Does back have the same meaning in I came back and I got it back? EXPECTED ANSWER: ”Well, almost.” (POLYSEMY)

4. Relevance of semantic similarity (vs. difference) • In synonymy: you expect similarity for a pair/(set) of items to be of interest • In polysemy: primarily, you expect difference for a pair/(set) of items to be of interest; secondarily, you group items according to similarity and difference

5. Yet…

6. synonymy --- polysemy ? • Dirk Geeraerts tomorrow in Helsinki: • ”The problem of synonymy and the problem of polysemy are essentially the same” • Dylan Glynn & Justyna Robinson (eds, in press) • Polysemy and Synonymy. Corpus methods and applications in Cognitive Linguistics. Amsterdam: Benjamins.

7. synonymy WORD 1 WORD 2 If their semantic content is similar or the same, this is a case of synonymy. If their semantic content is (very) different, a researcher of synonymy ignores this case.

8. polysemy MEANING 1 MEANING 2 OF WORD 1 OF WORD 1 The starting point is that Word 1 has at least 2 (different) meanings. If meanings 1 and 2 are very similar, this might be a case of vagueness. If meanings 1 and 2 are totally different (and not related semantically), this might be a case of homonymy. If meanings 1 and 2 are somewhat different but somehow relatable (or a bit similar), this is probably a case of polysemy. henceforth W = word M = meaning

9. scale of similarity: synonymy The meaning of W1 and W2 is… THE SAME --------------------------------- DIFFERENT perfectnear-weakNOT synonymysynonymysynonymyWORTH fullsemi-quasi-DISCUSSION synonymysynonymysynonymy

10. scale of similarity: polysemy The meaning of M1 and M2 (of W1) is… THE SAME --------------------------------- very DIFFERENT twovague-polysemyhomonymy instancesness(ambiguity) of the same meaningtwo instancesinstances of the same of different (yet related) meaning type meaning types

11. Main question • Is semantic similarity somehow different when we look at polysemy than we look at synonymy?

12. Differences so far • Which is the default, similarity or difference? • In synonymy, we idealize on the extreme of the scale, but mainly look at the part of the scale which is (fairly) close to the extreme. • In polysemy, we operate pretty much on the whole scale, with focus on the middle.

13. synonymy --- polysemy ? • when you study synonymy, the polysemy of the items gets in the way • can you ever say “W1 and W2 are synonymous”? • should you always say “Mx of W1 and My of W2” are synonymous? • when you study polysemy, you often use synonyms to talk about meanings • “Are get ‘receive’ and get ‘arrive’ meanings of the same verb?”

14. synonymy --- polysemy ? • Synonymy occurs when meaning is shared (but form differs) • Polysemy occurs when form is shared (but meaning differs) • Synonymy is a relational lexical-semantic property that unites (parts of the semantic potential of) “accidentally” coinciding words • The forms of words involved in the synonymy relationship are arbitrary (although the relationships might be non-arbitrary, cf. Levin this morning) • The semantic value (that is shared) is motivating enough that two or more forms coincide on it • It is typical that one meaning can be expressed with two different words.

15. synonymy --- polysemy ? • Polysemy is a semantic property of one word at a time that unites meanings. The relationship between them is motivated, but it is only sometimes predictable. • It is not accidental or arbitrary that words acquire polysemy. It is in their nature. :-) • It is typical for semantic value to be flexible, extended, and “multiplied”. • Polysemy is about categorization, both between words (W1 covers a semantic territory) and within a word (M1 and M2 are categories too). • One form : One meaning • a principle that cognition may strive for / take as a default • synonymy breaks it • polysemy breaks it

16. synonymy --- polysemy ? • The role of co(n)text • You can evaluate synonymy in identical co(n)texts: • I like to play the fiddle in bars. I like to play the violin in bars. • Usually you evaluate polysemy in non-identical co(n)texts • I got to Zabriskie Point. I got to a point in my life where… • But you can use identical co(n)texts as well. • I got to be the last one. I got to be the last one.

17. evaluating • To study shades of semantic similarity, we need to evaluate it. • A corpus cannot tell us if two instances are semantically similar • It requires human judgement • The main use of evaluating in this paper: • How informants / test subjects / speakers • evaluate the semantic similarity (or difference) • of linguistic items in a more or less experimental setting (e.g., similarity rating test) • ≈ Data elicitation ≈ Population test

18. evaluating • quantitative: • Estimate the degree of synonymy (or semantic distance between two meanings in polysemy) • qualitative: • Justify / explain / explicate the nature of / the reason for semantic similarity

19. evaluating takes place in real life as well • synonymy (examples) • in linguistic production, you e.g. estimate which of the near-synonyms might suit your needs best • in comprehension, you e.g. estimate whether near-synonyms that you have encountered refer to the same semantic value • in communication, when you negotiate meaning, you e.g. operate with synonymous alternatives • polysemy (examples) • in production, you e.g. apply words to new contexts • in comprehension, you e.g. approximate meanings according to related meanings of the same word • jokes often exploit polysemy • polysemy may cause misunderstandings • in communication, when you negotiate meaning, you e.g. cross-check with polysemy of other words

20. (Back to experiments/elicitation.) Expected difference between synonymy and polysemy, 1 • If an informant is asked to rate the semantic similarity/difference of two words, • the very fact that they are different words might cause her/him to presuppose that there is at least some semantic difference. • Therefore, rating two words ”semantically identical” requires a marked choice. • However, if the informant realizes that the researcher is after synonymy, then evaluating W1 and W2 as semantically similar is more likely.

21. Expected difference between synonymy and polysemy, 2 • If an informant is asked to rate the semantic similarity/difference of two meanings of one word, • the very fact that they are uses/instances of the same word might cause her/him to presuppose that there is at least some semantic similarity. • Therefore, rating two words ”semantically totally/very different” requires a marked choice. • However, if the informant realizes that the researcher is after polysemy, then evaluating M1 and M2 as semantically different is more likely.

22. Factors that influence • In both cases (synonymy and polysemy) • it matters a great deal • Which test (type) we use • What the instructions (exact phrasings) are • Whether there is an example rating given by the researcher • What the selection of stimuli is • What the linguistic context of each stimulus is • Which types of cases have been placed in the beginning of the test (or, the order in general)

23. Factors that influence • Should we expect (total) consensus? • No. There will be subjective differences. • Why? • The nature of semantics: • Based on intersubjective convention • Based on negotiation and flexibility • Must allow for variability and variation

24. Examples from (more or less) experimental studies on synonymy and polysemy

25. Whitten & al. 1979 (synonymy) • “Indicate the degree to which two words have the same meaning by writing a digit from 1 to 7.” • 7 =excellent synonymy • 1 = poor synonymy • All 464 stimulus noun pairs were listed as synonyms in standard references. • The rated degree of synonymy ranged from 6.79 to 2.24. The median was 5.08. • If placed within context of nonsynonym pairs, the ratings for the low end might have been higher.

26. Whitten & al. 1979 (synonymy) cont’d • Stimulus pairs at the high end: • purchase – buy 6.79 • lawyer – attorney 6.78 • autumn – fall 6.72 • penny – cent 6.71 • taxi – cab 6.71 • Stimulus pairs close to the median • college – university 5.12 • output – yield 5.10 • expert – authority 5.09 • effort – attempt 5.08 • servant – maid 5.08 • soldier – warrior 5.07

27. Whitten & al. 1979 (synonymy) cont’d • Stimulus pairs at the low end: • thunder – clap 2.72 • patient – invalid 2.55 • visit – chat 2.52 • suburb – neighborhood 2.34 • needle – spike 2.24 • Although instructions said that all stimuli are nouns, some of these are more common as verbs: buy, purchase, visit, chat • The polysemy is obvious in many cases: fall, authority, clap, patient, invalid

28. Whitten & al. 1979 (synonymy) cont’d • The main variable that they paid attention to was the order of the two stimuli: ½ of the informants got “forward order”, ½ got “back order”. • In 1979 one of their main aims was to study the structurings of the mental lexicon and lexical access. • Example: purchase => buy 6.72 buy => purchase 6.86 • On average, perceived synonymy was affected by word order. • For 21 word pairs, the effect of the order was significant.

29. Whitten & al. 1979 (synonymy) cont’d • Some of the 21 word pairs where the order played a significant role in the rating of the degree of synonymy: motive => reason 6.28 reason => motive 5.56 quarter => fourth 6.24 fourth => quarter 5.00 mission => task 5.66 task => mission 4.84 era => age 5.80 age => era 4.60 appetite => hunger 5.18 hunger => appetite 4.24 nectar => honey 4.94 honey => nectar 3.68 aborigine => native 4.52 native => aborigine 3.22 • Generalization: a more specific, more academic, and less polysemous word prompts a positive synonymy judgement more readily than vice versa.

30. Whitten & al. 1979 (synonymy) cont’d • Variance (between informants) • Mostly .50–1.20 at the end of 50 most synonymous • Exceptionally high variance at the high synonymy end: • murder => homicide 2.75 (cf. homicide => murder 1.03) • Mostly 2.00–3.00 at the median of the scale • Exceptionally low variance: province => territory 1.55 • Exceptionally high variance: congress => legislature 3.79 • Mostly 2.50–4.00 at the end of 50 least synonymous • That is, there was little consensus at the lower end of the scale.

31. Raukko 1994 (polysemy) • “Decide whether the word get carries the same meaning or two different meanings in the sentences.” • 0 = the same meaning • 2 = somewhat different meaning • 4 = very different meaning (heuristic post hoc: 4 might mean homonymy; 0 would refer to two instances of the same meaning type; typical polysemy would be 1...3)

32. Raukko 1994 (polysemy)(cont’d) • Data from my 1994 test, see handout.

33. Comparisons so far • Whitten & al. / synonymy • scale 1...7 (1 = very different meaning, 7 = same meaning) • synonymy ratings ranged 2.24...6.79 • median 5.08 (most pairs were viewed at least somewhat synonymous) • Raukko / polysemy • scale 0...4 (0 = same meaning, 4 = very different meaning) • polysemy ratings ranged 0.45...3.13 • average rating 1.55, median 1.34 (most pairs were viewed as having fairly similar but not identical meaning)

34. Comparisons so far • Whitten & al. / synonymy • informants saw synonymy where they were supposed to • Raukko / polysemy • informants did not see large meaning difference for the most part => get is polysemous, not homonymous • they saw some similarities, some differences, as predicted => they saw polysemy • both • differing degrees of similarity were apparent • many ratings make sense, some don’t • method is useful but there are skewing effects and irreliability in several details of the setting

35. Conclusions so far • In both synonymy and polysemy studies, semantic intuitions vary. • In both synonymy and polysemy studies, finding a scale of semantic similarity is useful. • Cf. Sandra & Rice 1995: 125 • “[researchers of prepositional polysemy] cannot propose extremely fine-grained distinctions without bothering about empirical data” • “language users’ mental representation [...] is [in fact] characterized by a high degree of granularity”

36. quantitative => qualitative • Whitten & al’s and Raukko’s similarity rating tests did not include informants justifying and explaining their ratings. • E.g., Liu (this symposium) reports tests with informants explaining their choices. • In Raukko’s study, qualitative results come from other types of tests • sorting test: (1) combine stimuli into categories, (2) give names to categories, etc. • production test: (1) produce examples of the use of polysemy, (2) explain links you find between them, etc. • Vanhatalo 2005

37. Vanhatalo 2005 (synonymy) • her PhD, The use of questionnaires in exploring synonymy • several types of tests • choose most likely components • rate components • choose better alternative (cf. Liu) • complete as sentences (only the word given) • define typical frames • spell out semantic differences

38. Vanhatalo 2005 (cont’d) • several factors investigated • 18 Finnish verbs of “nagging”, 17 Estonian verbs of nagging • the gender and age of the portrayed speaker (the subject of “nag”) • the degree of irritation of the portrayed speaker and hearer • the volume of the vocal act • 2-4 Finnish adjectives ‘important, central, crucial, significant’: open questions mainly

39. Vanhatalo 2005 (cont’d) • main results (Vanhatalo 2005: 40-45): the questionnaire method • helped to trace differences in the meaning and use of synonyms • many differences not documented before in dictionaries • sometimes consensus, sometimes deviation • useful especially for large groups of semantically similar words • (Vanhatalo did not use the method for placing synonyms on a scale of similarity) • both open questions and ratings should be used

40. Vanhatalo 2005 (cont’d) • main results (Vanhatalo) (cont’d) • helped to find differences between related words in Estonian and Finnish • sociodemographic variables caused fairly little variation • age and education affected a bit more than gender • answers critique

41. Vanhatalo 2005 (cont’d) • main results (Vanhatalo) (cont’d) • when both corpus method and questionnaire method were applicable, they yielded similar results • however, justification of results was different • questionnaire method dug up semantic properties that corpus method could not • in addition, can tackle low-frequency words • results of questionnaire method can be utilized in the production of electronic dictionaries

42. Other studies of synonymy that employ experimental techniques • Arppe & Järvikivi 2002, 2007 • Divjak & Gries 2008 • Liu, in this symposium • Oversteegen, in this symposium • etc.

43. polysemy / qualitative • In experimental settings (e.g., the sorting test): • An informant gives a name to a meaning type, a category within polysemy • An informant spells out the semantic link between two meanings • An informant draws a hierarchy between macrotypes and microtypes (more general and more specific meaning types) • An informant pinpoints at cases difficult to evaluate

44. And… • to conclude…

45. Evaluating semantic similarity • Both synonymy and polysemy operate on the scale of semantic similarity vs. difference. • Knowing about the degree of similarity is one useful property of both. • The way to find out about it is to use elicitation/experiments. • There is deviation in informants’ ratings. • A simple explanation: informants use different criteria for evaluation. • Solutions: let them explicate the criteria. use multiple methods.

46. Synonymy vs. polysemy • Evaluating semantic similarity between the meanings of two separate words (synonymy) is a matter of evaluating the match between two separate ”semantic events” • There should be mismatch, but there isn’t. • Evaluating semantic similarity/relatedness/ difference between the meanings of one word (polysemy) is a matter of comparing the applications of one single category. • There should be match between the semantic events.

47. Synonymy vs. polysemy • When you evaluate near-synonyms, you balance between (i) the ideal of what would constitute a perfect match and (ii) the nuances of the near-synonyms • When you evaluate meanings of a polysemous word, you balance between (i) the assumption that some meaning should be shared and (ii) the actual semantic profile of the uses

48. Synonymy vs. polysemy • In evaluating synonymy, the idealized equivalence can be taken from the semantic description of either of the two words. • In evaluating polysemy, the common factor (”core meaning”, ”shared meaning”) may be hard to find, or become too abstract. Maybe the first task is easier?

49. General relevance • ”Insights in the equality or similarity of meaning may shed light on meaning itself” (Oversteegen / SKY 2010, Helsinki) • The question of “identical meaning” is a crucial basis for e.g. typology and language comparisons: the problem of tertium comparationis • Cf. Haspelmath’s plenary on Saturday

50. References Arppe, Antti & Juhani Järvikivi 2007. Every method counts – Combining corpus-based and experimental evidence in the study of synonymy. Corpus Lingustics and Linguistic Theory 3: 2: 131-159. Colombo, Lucia & Giovanni B. Flores d’Arcais 1984. The meaning of Dutch prepositions: a psycholinguistic study of polysemy. Linguistics 22: 51-98. Divjak, Dagmar & Stefan Gries 2008: Clusters in the mind? Converging evidence from near-synonymy in Russian. The Mental Lexicon 3: 2: 188-213. Geeraerts, Dirk – in this symposium Liu, Dilin – in this symposium Oversteegen, Eleonore – in this symposium Raukko, Jarno 2003. Polysemy as flexible meaning: experiments with English get and Finnish pitää. In Brigitte Nerlich & al (eds) Polysemy. Flexible patterns of meaning in mind and language. 161-193. CONTINUED...