Thematic Alignment of Static Documents with Meeting Dialogs

1. Thematic Alignment of Static Documents with Meeting Dialogs Dalila Mekhaldi Diva Group Department of Computer Science University of Fribourg

2. Outline • Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Thematic Segmentation • Alignments Grouping • Conclusion & Perspectives

3. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Introduction In document-centric meetings (lectures, teleconferencing, press reviews, etc.): • Static documents are present • Should be integrated in a common multimedia archive  Need to build links between documents and other media

4. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Document Alignments Several way to link static documents with other meeting data: • Document/Image alignment • Document/Speech alignment

5. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Document/Speech Alignment • Links static data (documents) to temporal data (audio). • Enriches the documents with temporal indexes and thematic links. • Helps: • Building document-based browsing interfaces. • Improving documents search and retrieval.

6. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Document/Speech Alignment • 3 alignment categories • Thematic: lexical similarity of document/speech parts • Quotation of a document part • Reference to a document part • Text decomposition into segments • Document • Logical • Syntactic • Speechtranscript • Turns • Utterances Speech Transcript Document Logical Turns Syntactic Utterances

7. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives <sentence id="77">Rendu public mardi 15 juillet, le rapport de la commission d'enquete parlementaire sur la gestion des entreprises publiques, presidee par Philippe Douste-Blazy, secretaire general de l'UMP.</sentence > <sentence id="78">Tres critique sur la gestion de France Telecom et d'EDF - leurs politiques d'acquisitions ont ete menees sans que les moyens humains, techniques, financiers aient ete adaptes en consequence ..</sentence >… Similarity based matching Speech Transcript <Turn id=”1”> … <utterance id="3" StartTime="47.429" EndTime="61.062" speaker="spk2"> Alors euh.. mardi 15 juillet, hier, euh.. la commission d'enquête parlementaire en France a rendu un rapport euh..sur la gestion des entreprises.. entreprises publiques. </ utterance > <utterance id="4" StartTime="61.062" EndTime="71.806" speaker="spk1"> Euh.. Très critique sur la gestion de France Telecom et d'EDF, leurs politiques d'acquisitions ont été menées sans que les moyens humains, ... </ utterance > … </Turn> Thematic Alignment

8. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Speech transcript segments Document segments similarities S1’ S2’ S3’ S1 S2 … Thematic Alignment • Similarity based matching  Vectors of weighted terms: S1V1={t1, t2,..}; S1’V1’={t1’, t2’,..} a. Stop-words removing, Stemming b. Similarity metrics between units • Jaccard = |V1 V1’| / |V1 V1’| • Dice = 2 × |V1 V1’| / |V1| + |V1’| • Cosine =|V1 V1’| /  |V1| |V1’| • Two strategies: One-best and multiple alignments

9. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Cosine Cosine Precision Recall Dice Dice 1 1 Jaccard Jaccard 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 sent/utt utt/sent turn/logic sent/utt utt/sent turn/logic One-best Alignment Evaluation • Manual ground truth for 8 meetings • N1: alignments to found (manual ) • N2: alignments found (automatic) => • N3: correct alignments found (automatic) • Precision: N3/ N2 • Recall: N3/ N1 • Improve the similarity metrics with a semantic dictionary

10. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Doc/Speech Thematic Alignment Meeting Thematic Segmentation A4 A1 A2 A3 A5 <Thematic_Segment id=” S1”> … <utterance id="3" StartTime="47.429" EndTime="61.062" speaker="spk1"> Alors euh.. mardi 15 juillet, hier, euh.. la commission d'enquête parlementaire en France a rendu un rapport euh..sur la gestion des entreprises.. entreprises publiques. </utterance > <utterance id="4" StartTime="61.062" EndTime="71.806" speaker="spk1"> Euh.. en gros, ça dit que le modèle français des entreprises publiques ne répond plus aux nouvelles exi.. exigences internationales et européenne. </utterance > … </Thematic_Segment > … Multiple Alignments Evaluation

11. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives  nodes  nodes size The most connected sub-graphs  Thematic regions 3 77 78 79 80 81 84 85 Sentences 77 4 78 5 3 79 4 80 7 5 81 8 9 (84, 9) 0.42 7 84 8 10 85 9 11 10 12 11 Similarity value (91, 8) 0.25 13 91 12 14 13 utterances Document sentences Speechutterances Meeting Thematic Segmentation • Thematic alignment (e.g sentences/utterances) • Alignability  arcs • Similarity  weights

12. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives A1 A2 A3 A4 A5 Document sentences S1 S2 S3 S4 S5 a. Bi-graph representation of the multiple alignment pairs. Meeting Themes b. Densest regions extraction (using clustering) c. Segments extraction (clusters projection) Speech utterances Meeting Thematic Segmentation

13. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Thematic Segmentation Evaluation • Manual ground-truth for 22 meetings 1. Speech: 2 main sets • Stereotyped: 2.7 utterances/turn (ratio>2) • Non-stereotyped: 1.3 utterances/turn (ratio<=2) 2. Documents: 2 main sets • Mono-document • Multi-documents • Comparison with 2 mono-modal methods: Texttiling, Baseline • Speech Baseline: turn-based segmentation • Documents Baseline : reflexive alignment/clustering

14. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Bi-modal Texttiling Baseline 0.7 0.8 0.6 0.7 0.5 0.6 0.4 Pk 0.5 0.3 0.2 0.4 Pk 0.1 0.3 0 0.2 Stereotyped Non- Stereotyped Non- 0.1 stereotyped stereotyped 0 Mono-document Multi-documents Meetings Stereotyped Non-stereotyped Meetings Thematic Segmentation Evaluation • Pk(Beeferman) metric  0 for a perfect segmentation. a. Speech b. Documents

15. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives A1 A2 A3 A4 A5 A1 A2 A3 A4A5 A1 A2A3 A4 A5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 Analysis Document Our bi-modal methodoutperforms standard mono-modal methods: • more precise in computing the segments number • detects the similar segments • bridges the gaps between documents and speech transcript Documents greatly help structuring meetings

16. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignment • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives b. Combine the 3 alignments categories (Thematic, Quotations and References) to improve the document/speech alignment Speech Document <Turn> <Them_Align with Logic> <utterances> <utt> <Them_Align with Sent> <Quotations with Sent> <References with Logic> <Logic> <Them_Align with Turns> <sentences> <Sent> <Them_Align with utt> Alignments Grouping Document L1 L2 S1 S2 Speech T1 T2 U1 U2 U3 • Implementation of a framework that: a. Combine the various levels, to correct the false alignments pairs, e.g. (sentences x utterances) & (logical blocks x turns)

17. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignment • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Alignments Grouping 2. A tool for the visualization that: • Highlights the alignment categories(Thematic, Quotations, References) • Represent the various structures of the documents/speech as Layers. Speech Document

18. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Conclusion • Thematic Alignment of documents with meeting dialog • Is a solution for integrating static documents into multimedia archives: • Conference • Lectures, etc.

19. Introduction • Thematic Alignment • One-best Alignment • Multiple Alignments • Meeting Segmentation • Alignments Grouping • Conclusion & Perspectives Perspectives • Automatic transcription of the speech • Generalize the alignment on other: • documents types with few text (e.g. slides, agenda) • meeting kinds where documents are discussed irregularly (e.g. conferences)