Data collection and Multimodal Annotation Tools

Data collection and Multimodal Annotation Tools Dagstuhl 2001 Workgroup 2

Permanents Lisa Harper Michael Kipp Emiel Krahmer Jean-Claude Martin Dagmar Schmauks Visiting Scientists Harry Bunt Kioto Hasida John Lee Thomas Rist Laurent Romary Members

Needs • What is a multimodal corpus? • What corpora do exist? • How to collect a corpus? • How to develop/choose a coding scheme? • What tool to develop/choose? • What is the organizational infrastructure? • What is the future?

Multimodal Corpus • Sound • human speech (e.g. MPEG7) • transcription, (morphology) • part-of-speech • syntax (linguistic DS) • binary relations • thematic roles • rhetorical relations • co-reference • computer voice, sound, music • environmental sounds

Multimodal Corpus (2) • Vision • head: movement, gaze, facial expression • gesture: hands/arms • basic phases • formal features(handshape, trajectories, direction, location etc.) • encode qualities (Laban efforts?) • functional/semiotic categories (emblem, iconic, deictic, self-adaptors etc.) • posture: including feet/legs • computer graphics (charts/tables), characters • static/dynamic environment (people/objects): • moving camera

Multimodal Corpus (3) • Haptic • pressure of feet/hands/back/on seat, texture • force feedback • Biometric • heartrate, eye dilation, skin sensitivity, eyebrow movement, breathing • Smell & taste (VR) • Balance (VR) • Thermal (VR) • body/object temperature, conduit properties

Multimodal Corpus (4) • Within-modality/cross-modality relations • mirror behavior, synchronized behavior, repeated behavior, postural congruence • distance and touch • Behavioral/Social units?often across modalities!

Existing Corpora:Meta Survey • Existing surveys: • ISLE and NIMM (D8, EU & US) • ELRA (EU) • COCOSDA (Japan) • LDC (US) • TalkBank (US)

Existing Corpora:Dagstuhl 2001 • Survey with Dagstuhl participants • Collected 28 questionnaires • From 24 different institutes

annotated modalities speech: 20 gestures: 17 facial expression: 5 gaze: 3 posture: 3 file format analogue: 4 digital: 12 I don't know: 4 tool own tool: 9 other tool: 3 no tool: 8 I don't know: 1 application areas tourism/navigation (10), consumer electronics, info kiosk, realty, storytelling, instruction, cinema, graphical design, everyday gestures, education, car, face guessing, games, talk shows Questionnaire

Languages English: 11 German: 5 French: 2 Japanese: 3 Italian: 2 Dutch, Swedish, Finnish: 1 Planning to collect: 21 Questionnaire (2)

Data Collection:Methodology? • Legal issues: • ethical • commercial • country dependent legislature • Practical guidelines (best practice) • technical setup for recording • field-specific coder training, models for coding manuals • Specify meta-data

Coding Schemes • Survey on existing schemes: ISLE D9 • Guidelines for developing schemes: • encoding vs. inference • can scheme accommodate semantics or generation languages for MM players (MPEG) • Standardization • partial standards like in speech • standards for computer output log files (graphics output, locations, xml, trajectories, time-stamping, granularity)

Tools • Surveys of existing tools: • ISLE D11 • (Bigbee, Loehr, Harper 2001) • TalkBank proposal • Underlying frameworks: • track-based • annotation graphs • spatial annotation?

Tools: Checklists • Checklist for coding support • fast and efficient annotation • efficient view, search & find, customizable • extensibility of annotation • easy access to scheme definitions (online) • automatic extraction of modality-specific specimen (images, sound bits, transcription sequences) • Checklist for multi-coder support • update/merge, concurrent coding, reliability • Checklist for Import/Export

Tools: Visions • Bootstrapping (semi-automatic or fully automatic annotation) • Use MM techniques for coding tools (3D, haptic, VR) • Standardized analysis (e.g. metrics) and visualization (metaphors) • Modular generic framework for • tools • schemes

ML classifier Coding Scheme parser specific analysis Coding Tool data viewer general analysis Tools (4) Logical Layer Annotation Framework (Tracks, types, objects etc.)

ML classifier gaze parser analysis module speech gesture Coding Tool scheme framework Annotation Framework (Tracks, types, objects etc.)

Organizations • Initiatives: • EAGLES/ISLE • ATLAS, MATE/NITE • TalkBank, Childes • International: • ELRA/ELDA, US? Asia? • National agencies (Eurospeech): • BAS, LDC, MPI Nijmegen

Future • Data collection project • sample videos with illustrative MM data • pre-coded minimal data (speech transcription) • Comparison/integration of schemes • Encourage collaborative coding?

Future (2) • Workshop on LREC Language Resources and EvaluationCanary Islands! May 2002 • deadline: 20 Nov 2001 • paper on Dagstuhl and follow-ups • coding excercise based on data coll. • questionnaire based on Dagstuhl survey

Data collection and Multimodal Annotation Tools

Data collection and Multimodal Annotation Tools

Presentation Transcript

Data collection tools: Interview and Questionnaire Methods

Data and Data Collection

Data and Data Collection

Towards an integrated scheme for semantic annotation of multimodal dialogue data

LDMT MURI Data Collection and Linguistic Annotation

Data Collection Tools

Data Collection and Analysis Tools

Tools and Tips Collaboration and Data Collection

Data Collection Tools Interview and Questionnaire

Web-based analysis and annotation tools

Participatory Tools and Methods for Data Collection

Collection of multimodal data Face – Speech – Body

An overview of EMMA—Extensible MultiModal Annotation

Data Annotation Tools

Data Annotation Tools Market

Data Annotation

Data Collection, Annotation platform, AI training data

Data Annotation Services

Data Annotation Tools Market

Data Annotation Tools Market Valuable Growth Prospects 2035

Data Annotation

Data Annotation Tools ppt