AMI community of interest meeting

1. Automatic segmentation of meeting recordings AMI community of interest meeting David van Leeuwen, Stephan Raaijmakers, Wessel Kraaij

2. TNO is active in five core areas • Facts & Figures: • - Annual turnover: 553 Mio euro • Employees: 5100 Automatic segmentation of meeting recordings

3. A unique Dutch ICT innovation centre About TNO ICT • Established: 1 January 2003 • Bundling of former KPN Research with TNO’s ICT related departments • One of the largest ICT knowledge centres in Europe Features and unique selling points • Independent • Frontrunner • Multidisciplinary: • Conceptual and hands-on • Technical, economical and sociological • In-depth Telecom and IT expertise Key figures • Annual turnover: EUR 40 Mio • 375 professionals • 10 high-quality patents per year • Locations in Delft, Groningen and Enschede Automatic segmentation of meeting recordings

4. Indexing meeting recordings • Characteristics of meetings: • Lack of structure • Low information density • Rich in non-verbal cues • Challenge: • identify segments • annotate segments • TNO focus: • robust features • multi-level segmentation • low tech requirements Automatic segmentation of meeting recordings

5. Application scenario • Enabling step: building a browsable meeting recording archive: • multimodal analysis: • video analysis (e.g. motion zones), • speech analysis (e.g. diarization, laughter detection) • transcript analysis (e.g. topic segmentation, summarization, sentiment analysis). • Usage Scenario:searching/filtering interesting segments, • As soon as meeting segments have been detected and annotated, they can be exploited for any a search or summary generation application. • E.g. all positive comments on the company’s new flagship product expressed by marketing consultant “Joe. ”. Automatic segmentation of meeting recordings

6. TNO proposition • TNO seeks participation of COI members for a mini-project for the application of multi-level segmentation and annotation of meeting (or lecture) recordings. • A feasibility study of the application of TNO technology for CoI member product line. • Technologies ready for evaluation in a mini-project: • Speaker diarization: who spoke when (practical when speakerphones or central microphones are used) • Topic segmentation: would like to perform a test on an archive with real data. • Sentiment classifcation: state of the art labeling performance, would like to perform a test on meeting data • Motion zone classification: finding hot spots Automatic segmentation of meeting recordings

7. Speaker Diarization • Answers the question Who spoke When? • No prior information from participants required • no training necessary • but absolute identity therefore not resolved • we can use speaker recognition technology for this • Useful for finding out • who talks most • who interacts with whom • who says important things (using transcript) • … Automatic segmentation of meeting recordings

8. TNO solution for Speaker Diarization • Technology • requires unobtrusive distant microphones • uses acoustic properties of the voice • uses direction of signal • if multiple microphones are available • Performance • is evaluated in NIST Rich Transcription benchmark evaluations • is among the best performing teams • good co-operation with these teams Automatic segmentation of meeting recordings

9. Hot spot segmentation • Motion: pixels different from background estimation • Motion is measured in zones • Per person 3 zones: • head • hands • close-up camera • Motion gives indication about speaker activity • Gesture activity • Head movement • Cue for ‘hot spots’ Automatic segmentation of meeting recordings

10. Feature browser: motion zones Hot spot? Automatic segmentation of meeting recordings

11. Subjectivity in meeting transcripts • Focal point of TNO ICT: sentiment analysis in texts • Determine if texts (like movie reviews) are positive, negative or neutral (global sentiment classification; see paper) • Determine local sentiment (phrase level) • Find subjective and objective statements • For AMI: apply subjectivity detection to speech transcripts • Align subjectivity with hot spot information and speaker segmentation • Integrated browser for multimodal sentiment cues: • Motion (gestures) • Speaker information • Subjectivity information Automatic segmentation of meeting recordings

12. Topic segmentation • Automated division of texts (like meeting transcripts) into separate topics • Main topics • Fine-grained subtopic structure • A Machine Learning problem: learn on basis of segmented texts • AMI data is hard • Low interannotator agreement • Highly technical and overlapping vocabulary • TNO: two approaches • An approach based on Conditional Random Fields, using sequential (contextual) information, optimized for standard error metrics • An SVM-based approach, optimized for a new and better error metric • Both approaches significantly outperform the baseline LCSEG algorithm, a well-known and quite good algorithm Automatic segmentation of meeting recordings

13. Meeting transcript (ground truth) • ==========:1 • so um the thing we have to know is you already know what we're going to do , you also read what this the things or , not yet , okay . so um , yeah , it has to be original , trendy , user-friendly that's what we're going to design . uh first we have uh uh three steps of uh making the the remote control . fir the first thing is th the functional design , that's very important . we have to look what the needs are , the effects of the functional design , and and how the mm the the remote control works , so that's where we're going to look in the functional design , it's for the f next meeting . • ==========:2 • yes . • the the second thing is the conceptual design , that's what it that's uh the spe the specifications of the components and the properties and the specifications of the user interface . and we have to look what uh the market is doing for what kind of uh remote controls are in the market . and the third thing is uh the detailed design um and that's exa yeah , you know what it is , it's exactly how it looks and whatever . okay so uh no , this is a these are two smartboards , with the uh f uh s an introduction of that one . • and you already saw you know all that that you here can put uh things in the the red project uh map . folder , okay . Automatic segmentation of meeting recordings