Equal-party Conversation System for Language Learning

Equal-party Conversation System for Language Learning Chih-yu Chao <chihyu@mit.edu> (advisor: Stephanie Seneff) April 14th, 2006 Dialogs on Dialogs Reading Group

Overview • Motivation • Approach • Implementation • Future Work • References

Motivation (1 of 3) • Computer-Assisted Language Learning • Reading & writing • Listening (few) • Speaking (fewer) • Conversation (none?!)

Motivation (2 of 3) • Human tutors vs. computers • Confidence, stress • Making mistakes • Patience • Accessibility • Feedback

Motivation (3 of 3) • The proposed system • Domain-specific conversations • Sentence formation • Speaking • Listening • Less stressful environment • Game-like interaction process • Remember the new vocabulary/sentence patterns more efficiently

Approach (1 of 5) • Domain • Activity scheduling (cf. database-accessing) • Part of the lesson plan • Not simply form-filling • Activity proposal • Confirmation of likes/dislikes • Activity negotiation • Schedule checking • Multiple turns of these actions

Approach (2 of 5) • Taking the initiative • Forming different types of sentences by playing a different role in each dialogue

Approach (3 of 5) • Languages of interest • Mandarin Chinese  English • The game • Persona • A schedule from yesterday to day after tomorrow • Preference of activities • The virtual buddy also has a persona represented internally • Respond and/or negotiate based on the given persona until an agreement is reached • New persona

:setup ( {c like :min_count 3 :max_count 5 :choices ( "swimming" "basketball" "television" "movies" "football" "baseball" "tennis" "dancing" "video games" "shopping" ) :unique 1 :pattern ":choices" } {c dislike :min_count 3 :max_count 5 :choices ( "swimming" "basketball" "television" "movies" "football" "baseball" "tennis" "dancing" "video games" "shopping" ) :unique 1 :rule_out "like" :pattern ":choices" } {c schedule :min_count 3 :max_count 7 :choices ( "swimming" "basketball" "television" "movies" "football" "baseball" "tennis" "dancing" "shopping" "studying" "video games" ) :ordinates ( ":day" ":time" ) :pattern ":choices" :day ( "yesterday" "today" "tomorrow" "day after tomorrow") :time ( "morning" "afternoon" "evening" ) :rule_out "dislike" } ) :features {c features :day ( "yesterday" "today" "tomorrow" "day after tomorrow") :time ( {c morning :start_hour 9 :end_hour 12 } {c afternoon :start_hour 13 :end_hour 17 } {c evening :start_hour 19 :end_hour 22 } )

A: "what are you doing day after tomorrow afternoon?"B: "I am going to study the day after tomorrow in the afternoon"A: "are you free day after tomorrow morning?"B: "yes I am going to do nothing the day after tomorrow in the morning"A: "would you like to play tennis with me the day after tomorrow in the morning"B: "yes I do like to play tennis"A: "okay let's get together the day after tomorrow in the morning to play tennis"B: "yes that would be great" ====================== New Persona ====================== A: "what are you doing tomorrow evening?" B: "I am going to do nothing tomorrow evening" A: "would you like to play football with me tomorrow evening" B: “to play football is not bad" A: "okay let's get together tomorrow evening to play football" B: “okay to play football is not bad"

Approach (5 of 5) • Other features • Get help by speaking in the native language • 2 recognizers, threshold • Score • Keeps track of the difficulties • Same material (vocabulary or sentence pattern) will be reviewed until the user succeeds

Implementation (1 of 3) • Interlingua-based approach • Natural language input  language-independent semantic frame • TINA: syntactic and semantic analyses • Genesis: semantic frames  well-formed surface strings in the target language • Based on the language-dependent generation rules (Pluto)

{c verify :auxil “link” :topic {q pronoun :name “you” :number “pl” :domain “LanguageLesson” :sro “are you free day after tomorrow afternoon” :ilang “english” :olang “pinyin” :num_oreds 2 :pred {p attribute :topic “free” } :pred {p temporal :topic {q rel_date :name “day after tomorrow” } :num_preds 1 :pred {p daypart :topic {q daytime :name “afternoon” } } } }

Implementation (3 of 3) • 2 simulated users • Dialogues automatically generated by the system • Without eliciting input from human subjects • Acquire a large amount of data with very little cost • Good for different types of bootstrapping • Coverage of language generation rules • Accuracy of speech recognition • Quality of speech synthesis

Future Work • Interface • Phone & web • Web only (with audio GUI) • Peer-to-peer interaction mode • (Delayed) interaction between 2 learners • Target language only • User study • Curriculum design • Evaluation • Software • Pedagogy

References [1] V. Zue, S. Seneff, J. R. Glass, J. Polifroni, C. Pao, T. J. Hazen, and I. L. Hetherington. Jupiter: A telephone-based conversational interface for weather information. IEEE Trans. on Speech and Audio Processing, 8(1):100--112, 2000. [2] S. Seneff and J. Polifroni. Dialogue Management in the Mercury Flight Reservation System. In Proceedings of ANLP-NAACL 2000, Satellite Workshop, pp. 1--6, Seattle, WA. [3] S. Seneff. TINA: A Natural Language System for Spoken Language APplications. In Computational Linguistics, 18(1):61--86, 1992. [4] L. Baptist and S. Seneff. GENESIS-II: A Versatile System for Language Generation in Conversational System Applications. In Proceedings of ICSLP, pp. 271--274, Beijing, China, October 2000. [5] B. Cowan. PLUTO: A Preprocessor For Multilingual Spoken Language Generation. Master's thesis, MIT, Cambridge, MA, February 2004.

Questions?

Equal-party Conversation System for Language Learning