430 likes | 619 Views
Lecturing with Digital Ink. Richard Anderson University of Washington. Lessons learned from the Classroom Presenter project. Classroom Pedagogy Teaching with ink HCI Ink based presentation Multimedia Analysis of lecture artifacts. Classroom Presenter.
E N D
Lecturing with Digital Ink Richard Anderson University of Washington
Lessons learned from the Classroom Presenter project • Classroom Pedagogy • Teaching with ink • HCI • Ink based presentation • Multimedia • Analysis of lecture artifacts
Classroom Presenter • Integration of slides and digital ink using Tablet PC • Key ideas: • Ink overlay on images • Distributed application • Many other systems also support ink and slides
Classroom Presenter as a distributed application • Designed as distributed application for distance learning • Enables many scenarios • Mobility • Walking and talking • Sharing materials with students • Note taking • Classroom interaction • Student submissions
Deployments • Estimated use in at least 100 courses • Wide use inside of computer science • Push for adoption outside of CS • Lecture archives from UW Professional Master’s Program • Several hundred hours of recorded audio, video, and ink.
Ink use in presentation • Cognitive load • Only limited attention available for computer while lecturing • Linkage with speech • Close tie between ink and speech
Cognitive load • Limited feature use • Even color change unusual • User interface must be simple (and robust) • Cannot give feedback to user • Many actions appear to minimize mental effort • Color change only for contrast • Reliance on screen erase
Understanding Attentional Marks • Properties • Brief, simple markings • Occur with speech • Augment meaning of speech • Ad hoc form • Is there a linguistic context in which to understand these marks?
Spontaneous Hand Gestures Spontaneous Hand gestures [McNeill]: • are synchronous w/speech • are co-expressive w/speech • lack standard of form Attentional marks share these properties.
Analysis of digital ink • Understand ink usage • Motivation: inform development of ink based applications • Archiving • Search, Summarization, Transcription • Lecture based • Improved rendering, note taking, accessibility
Ink classification • Textual • Diagrammatic • Attentional Coding of six hours of lecture
Goals • Understand usage “in the wild” • Cannot expect lecturers to modify behavior • Determine opportunities for automatic analysis • Identify challenges
Methodology • Study of recorded classes • Best data set: Professional Master’s Program • Distance courses • Audio, Video, Ink archives • HCI, Compilers, Programming Languages, AI, Transaction Processing
Attentional ink • Problem – content matching • Identify slide content referred to by ink • Study • Implement basic algorithms to match attention marks to slide content • Compare results with human coders
Attentional ink • Determine the lecturer’s intent: • Determine level to parse the content
Attentional ink • Challenges • Recognition of attentional ink on text • Difficult example:
Handwriting • How well does handwriting recognition work on “typical” instructor writing? • Domain has many challenges
Recognition Study • Studied isolated words/phrases written on slides • Removed non-textual ink • Fed through the Microsoft Handwriting Recognizer • No training
Recognition Examples • The Good: • The Bad: • The Ugly:
Joint Writing and Speech Recognition • Can we use handwriting recognition with speech recognition together to improve accuracy? • Co-expression of ink and speech • Are written words spoken as well? • Can speech disambiguate handwriting? • Can handwriting disambiguate speech?
Examples • Difficult for Speech and Ink Recognition • Difficult Written Abbreviations • Speech/Ink Used to Disambiguate Ink/Speech
Experiment • Examined instances of isolated word writing • Selected word writing episodes at random but uniformly from the various instructors • Generated transcripts manually from the audio • Checked whether the instructor spoke the exact word written • Measured the time between the written and spoken word
Activity Recognition • Identifying slide corrections
Diagrammatic ink • How do instructors use diagrams • Basic legibility • Observed behaviors • Diagram phasing • Locality of expression
Typical diagram • Basic, irregular shapes • Difficult labels • Attentional ink
Locality in diagrams Separate wins indicated together Top arrows: “Not there”
Summary • Pedagogy with ink • How is ink used in conjunction with content and speech to express information • Presentation with ink • Low attention task • Analysis of ink usage • Extracting meaning from archived lectures
Resources • cs.washington.edu/education/dl/presenter/ • Software Downloads • Papers • Contact info • Richard Anderson, anderson@cs.washington.edu • Ruth Anderson, ruth@cs.virginia.edu • Craig Prince, cmprince@cs.washington.edu