1 / 22

Collecting, Storing, Coding, and Analyzing Spoken Tutorial Dialogue Corpora

Collecting, Storing, Coding, and Analyzing Spoken Tutorial Dialogue Corpora. Diane Litman LRDC & Pitt CS. ITSPOKE Tutorial Dialogue Corpora. Students engage in spoken dialogue with tutors, in the qualitative physics domain human tutors (fully automated) computer tutors

jwaldo
Download Presentation

Collecting, Storing, Coding, and Analyzing Spoken Tutorial Dialogue Corpora

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Collecting, Storing, Coding, and Analyzing Spoken Tutorial Dialogue Corpora Diane Litman LRDC & Pitt CS

  2. ITSPOKE Tutorial Dialogue Corpora • Students engage in spoken dialogue with tutors, in the qualitative physics domain • human tutors • (fully automated) computer tutors • ‘wizard’ computer tutors

  3. Data Collection • Speech-enhanced computer interfaces • Head-mounted microphones • Currently no video • Humans can be at different locations • Human and Wizard Tutoring • Dialogue speech files • Computer Tutoring • Utterance speech files • Coordinated system logs

  4. Data Storage • Wav, raw audio, ogg formats • Sampling • 16k samples per second • 16 bits per sample • Stereo (dialogue level) and mono (utterance level)

  5. Coding and Analysis • Initially WaveSurfer • Open Source tool for sound visualization and manipulation • Speech/sound analysis • Sound annotation and transcription • Praat is similar • Recently moved to NXT (NITE XML Toolkit) • Also Open Source • http://groups.inf.ed.ac.uk/nxt/

  6. NXT • Mature open-source libraries to support heavily annotated corpora whether they be multimodal; textual; monologue; dialogue • A powerful integrated query language • Built in tools for common tasks + Java API for custom tools • Media sync built in • Command line tools for data analysis

  7. NXT meets the ICSI Corpus Jean Carletta and Jonathan Kilgour University of Edinburgh HCRC Language Technology Group

  8. ICSI Meeting Corpus • 75 natural meetings from research groups • close-talking and far-field microphones • orthographic transcription • "speech quality" tags (e.g., emphasis) • dialogue acts • hot spots

  9. The NITE XML Toolkit • library support for data handling and search using a data model that can express both timing and complex structure • multiple file stand-off XML data storage • some standard GUIs, data utilities • library support for writing tailored GUIs

  10. Stand-off XML extract from Bdb001.A.speech-quality.xml <speechquality nite:id="Bdb001.emphasis.16" type="emphasis"> <nite:child href="Bdb001.A.words.xml#id(Bdb001.w.1,342)..id(Bdb001.w.1,344)" /> </speechquality> extract from Bdb001.A.words.xml <w nite:id="Bdb001.w.1,342" starttime="356.39" endtime="" c="W">time</w> <w nite:id="Bdb001.w.1,343" starttime="" endtime="" c="HYPH">-</w> <w nite:id="Bdb001.w.1,344" starttime="" endtime="356.59" c="W">line</w>

  11. Tasks • pre-NXT: up-translation and tokenization • hand annotation (topic segmentation, dialogue acts, extractive summaries, ...) • automatic annotation/indexing by query match

  12. Queries in NXT ($a w):(TEXT($a) ~ /th.*/):: ($s speechquality):($s ^ $a) && ($s@type="emphasis") • Find instances of words starting with “th” • For each find instances of speech quality tags of type "emphasis" that dominate the word • Discard words that are not dominated by at least one such tag Use queries to understand data, verify quality, index.

  13. NXT as Meeting Browser • Browser = display + signal indexing + search • NXT data displays: • synchronize with signal • highlight search results

  14. Issues • Already can't load all the ICSI data at once on some machines • NXT supports display of one meeting at a time but browsing may be over several meetings • Really complicated queries are often too slow for browser response times Key: Pre-indexing of query results, tailored data builds

  15. NXT meets the BEETLE Corpus Johanna Moore’s Group University of Edinburgh

  16. Coding Tutorial Dialogue • Partitioned the dialogue into a set of non-overlapping segments with the following category names: • Content • Dialogue that contains information relevant to the topics in the lessons. • Management • Dialogue that does not contain information relevant to the lesson topics, but deals with the flow of the lesson. • Metacognition • Dialogue that contains the student or tutor’s feeling about his or her understanding of the lesson material or each other. • Social • Dialogue that serves as motivation, encouragement, humor, or establishing rapport.

  17. Coding Student Utts for Sig Events NOVELTY1 ACCURACY2 & CONFIDENCE3 ACCURACY2 & DEPTH4 DEPTH4 INITIATIVE5 • Constructivism / generative learning • Osborne & Wittrock, 1983 • Impasses • Van Lehn, et. al., 2003 • Accountable talk • Wolf, Crosson & Resnick, 2006 • Deep processing / cognitive effort • Thomas & Rohwer, 1993 • Motivated, self-directed learner • Thomas & Rohwer, 1993 • Student produces a lot of new information • Student utts are incorrect or correct w/ low confidence • Student utts are both accurate & deep • Student utts are deep (regardless of accuracy)‏ • High frequency of internally motivated student utts • Consider common theories of effective learning events

  18. Student Utterance Coding • Five major dimensions • Accuracy • Correct, some missing, some errors, incorrect • Signs of “deep” processing or cognitive effort • Present versus absent • Explain/justify/support statement with evidence/reasoning • Summarize or paraphrase • Express relationships or make connections between constructs • Questions or challenges statements from lesson or tutor • Wolf, Crosson & Resnick (2006)‏ • Signs of low confidence • Present versus absent (Bhatt, Evens & Argamon, 2004)‏ • Origin • Externally versus internally motivated • Novelty • Old versus new information

  19. Accountable Talk: utt83a: student: both bulbs A and C will go out because this scenario would act the same as if there was an open circuit Accuracy = Correct; Cognitive Processing = Present utt69: student: A and C will not light up  Accuracy = Correct; Cognitive Processing = Absent Non-Accountable Talk: battery utt122a: student: bulb a will light but b and c won't since b is damaged and breaks the closed path circuit Accuracy = Incorrect; Cognitive Processing = Present Cognitive Effort and Potential Impasse: A C B X Potential Impasse: utt97: student: both would be either dim or not light I would think  Accuracy = Partially Correct; Cognitive Processing = Absent; Signs of Low Confidence = Yes Question: If bulb B is damaged, what do you think will happen to bulbs A and C?

More Related