Prosody and NLP

1. Seminar by Nikhil : 06005004 Adith : 06005005 Prachur : 06D05011 Prosody and NLP • We have a presentation this Friday ? • We have a presentation this Friday ? • We have a presentation this Friday ?

2. Abstract Speech Processing and Natural Language Processing share a common area of study: Language. However, over time, they have grown to have little in common regarding theoretical models or methods of analysis. NLP takes written text as the starting point for it's analysis,  however, a lot of valuable information is lost in encoding speech as merely text. It is commonly accepted that intonational features of spoken language can greatly aid NLP tasks (like adjective scope resolution). We explore the foundations of the study of Prosody and observe some approaches that use prosodic cues to aid NLP.

3. Motivation • Language is not text driven but speech driven. • NLP currently has written text as the starting point for it's analysis (primarily due to the abundance of such data). • A lot of information is lost on ignoring spoken features and just looking at the text.

4. A Way Out ? • Utilize spoken features for NLP tasks • NLP needs all the help it can get. • Dealing at the pragmatics or discourse level is extremely untenable • Prosodic cues carry useful pragmatic information

5. What is Prosody, exactly? • Comes from Poetry, prosody refers to the study of poetic meters[1] (rhythms)‏ • Written text treats words as the basic building blocks of language. • Spoken language treats syllables as the basic building block. • Syllables : Consisting of Onset, Nucleus and Coda

6. What is Prosody, exactly? (contd.) • Wikipedia has this to say : Prosody is the rhythm, stress, and intonation of connected speech (as opposed to smaller elements like syllables or words). Prosody may reflect : • features of the speaker, • emotional state of a speaker, • features of the utterance, • ironic or sarcastic, • emphasis, contrast, and focus

7. Intonation ? • Conveys paralinguistic information, emphasis and contrast. • Intonation on a particular word could differentiate between sentence moods. • You are finISHED (interrogative)‏ • You are FINIshed (imperative)‏ Image courtesy Google Image Search

8. And Stress! • Stress is applied on Content Words in spoken utterances. • cOntent - Noun. • "I really liked their presentation's content.“ • contEnt - Verb. • "I have done my best. I am content.“ •  Stress on a pair of words distinguishes between the syntactic role played by each word in the pair. • tight<pause>rope : A rope that is held taut. • tight-rope : A circus-act uses this contraption :)‏ • [2]

9. Courtesy tom The Dancing Bug

10. Prosodic cues  Prosodic functions important for linguistics are • Marking of boundaries (syntactic, semantic or dialogue units.)‏ • Relative duration of phonetic segments • At syllable level : Energy, intensity, duration and intonation of syllable. We shall see two approaches of using these features in tasks central to NLP.

11. Prosody-Augmented Syntax Grammars[3] • Cue Used : • Relative duration of phonetic segments • Aim : • To improve the parsing of ambiguous sentences. • Method : • Augmenting the syntax grammar with a few • non-terminals and rules.

12. Concept of “Word Break Indices” used to show prosodic decoupling between neighboring words. • E.g. - • Andrea 1 moved 1 the 0 bottle 3 under 0 the 0 bridge. • Andrea 1 moved 3 the 0 bottle 1 under 0 the 0 bridge. Break indices were generated by analysing the coda that have a pause. Coda is the final sounded consonant of a word E.g. – cup , milk

13. Grammar Modification Original grammar rules like S -> NP VP etc. are changed to S -> NP link1 VP. “Link” non-terminals are used for the word-break indices. The Parse Trees for the aforementioned example become…

14. Example 1 • Andrea 1 moved 1 the 0 bottle 3 under 0 the 0 bridge. S NP Link1 1 VP V moved Link2 1 Link4 3 PP N NP Link5 0 Andrea NP P under D the N bottle D the N bridge Link3 0 Link6 0

15. Example 2 • Andrea 1 moved 3 the 0 bottle 1 under 0 the 0 bridge. S VP NP Link1 1 V moved Link2 3 NP N PP Andrea D the Link4 1 N bottle Link5 0 P under NP Link3 0 D the N bridge Link6 0

16. Results • The incorporation of prosody resulted in a reduction of about 25% in the number of parses found . • Parse times increase about 37%. • Extremely common cases of syntactic ambiguity can be resolved with prosodic information, and that grammars can be modified to take advantage of prosodic information for improved parsing

17. Using Prosodic Features in Language Models[4] • The outlined approach uses syllable-based prosodic cues, namely • Duration of the syllable • Average energy (intensity)‏ • The average F0 (fundamental frequency of the syllable) contour • The slope of the F0 contour (visualised as intonation-rising or falling/flat)‏

18. Recognition of Prosodic Features Courtesy [4]

19. Prosody in Language Model • We want to measure P(wn | wn-1,wn-2,…,F) • Naively modelled by linear interpolation : • Assumption : prosody features independent of previous words (not true!!). P(wn | wn-1,wn-2,…,F) = αP(wn | wn-1,wn-2,…) + (1- α)P(wn|F) • We want something better

20. Factored Language Model • Instead of a word W we will deal with a set of word-factors F={f1,f2..fk} (Factors may include the word itself)

21. Here, F is chosen as {W, prosodic features} • The four prosodic features are encoded as binary numbers(s0 to s15). • These numbers are assigned to each syllable of the word . • For e.g. the prosodic representation for the word “Actually” can be either ‘s10s12s6’ or ‘s10s15s6’ .

22. Conclusion • Prosodic Cues can play an important role as a heuristic for many NLP tasks. • All is not one way traffic though. POS tagging (since its relatively accurate) is used to aid speech synthesis tasks which conventionally used only prosodic cues[5]

23. Future Work • Handling prosodic information is a first step towards integration of Speech Processing and NLP Courtesy ZITS

24. References • Fromkin, Rodman and Hyams, An Introduction to Language, 7th Ed, Thomson and Wadsworth,2002 • En.wikipedia.org : Prosody, Austrian School of Thought, Intonation • John Bear and Patti Price (1990), “Prosody, Syntax and Parsing”, in proceedings of the 28th annual meeting of the ACL • Songfang Huang and Steve Renals (2007), “Using Prosodic Features in Language Models for Meetings”, IRTG annual meeting • http://speech.iiit.net/~raghavendra/Webpage/ppprts.pdf