What Did I Miss?

1. What Did I Miss? In-Meeting Review using Multimodal Accelerated Instant Replay (AIR) Conferencing Kori Inkpen John C. Tang Rajesh Hegde Chris Brooks Univ. Saskatchewan Zhengyou Zhang Sasa Junuzovic

2. Goal Make meetings (and play) better with telepresence technology

3. Meetings are important

4. Do you ever ….

5. Partial Meeting Review Verizon Conferencing Report - 2003

6. What did I just miss?

7. DVR Functionality -> Live Conference

8. What makes it challenging?

9. Previous Research • Meeting Capture • Cutler et al., 2002; Jian, Kim & Li, 2003; Ranjan, Birnholtz, & Balakrishnan, 2008 • Summaries • Video Skimming & Summarization - Smith & Kanade, 1995; He, Gupta & Grudin, 1999; Christel, 2006; Money & Agius, 2008; • Indexing Recorded Meetings • Moran et al., 1997 • Time-Compression • Omoiguiet al., 1999; Orr, 1971; Wildemuthet al., 2003 • Real-time Catchup • Tucker et al., 2010

10. Prototype System AIR Conferencing

11. AIR Conferencing System 1/2 Live

12. AIR Conferencing System 2/2 Playback

13. AIR Conferencing - User Study #1 Explore the benefits (and challenges) of Accelerated Instant Replay for meetings

14. Experimental Design • 18 Participants (1 female) • Three conditions • Background Interviews • Task: Live Status Meeting • 3-way Videoconference • Each person gets a chance to present • Each person gets interrupted 4 times and has to catch up • Recall assessment (quiz on facts + explanations)

15. AIR – Interview Results • All occasionally miss parts of meetings • Current catch-up strategies (most indicated they do not want to disrupt the meeting) • often just listen and try to deduce what they missed, possibly interrupt with a question if they are confused • wait for a recap by the presenter • browse meeting material to figure out what was missed • discreetly ask someone what was missed (whisper or IM)

16. AIR – Interview Results • Useful to replay what was missed during a videoconference? • Yes (10), maybe (7), no (1). • “Often you miss critical conversations when you step out or are interrupted during a meeting and then you try to play catch-up during the rest of the meeting. Getting to know what was covered and who said it and the body language would put me back into the meeting very quickly.” • “It would depend mostly upon the importance of the meeting, followed by the duration of how much I missed, and finally, on how discreetly I could review the video”

17. AIR – Recall Results • Full Replay enabled users to recall as much as they did when they were not interrupted.

18. AIR – Preference Results – • Full Replay: • “it was fast, easy to concentrate and auto catch-up” • “easiest to follow; leads most smoothly into rejoining live” • “seems like the only way to catch up in a focused way” • “fast audio was really cool …. I feel like I can cheat in time with fast forward”

19. AIR – Divided Attention Results • Past + Present: • “I can listen to audio while watching the live slide show and transcript” • “you can listen to current conversations and read transcripts”

20. New Questions • How important were the additional modalities (over just audio alone)? • How successful would the system be with perfect speech-to-text transcription? • How can we better support multitasking?

21. AIR Conferencing - User Study #2 Examining individual modalities and impact of perfect speech-to-text transcription

22. Prototype System Videos Speech-to-text Transcript Shared Desktop

23. Experimental Design • 58 Participants (25 female) • Pre-recorded status meeting • Mixed Design • Everyone did audio only + 1 other condition • Focused review (no divided attention)

24. Modalities

25. User study #2 – Recall results 1 Significantly better than audio only but worse than live

26. User study #2 – Recall results 1 Significantly better than audio only and no difference with live 2 Significantly better than audio only but worse than live

27. User study #2 – Recall results • Users were significantly more confident with their answers in the Audio + All condition (p<.05). 1 Significantly better than audio only and no difference with live 2 Significantly better than audio only but worse than live

28. User study #2 – Recall results • “I used the audio to see who was talking, and every time I missed something, I had the transcript which kept a recording of everything and I could just look back at it.” • “Being able to see who was talking during catch-up helped to associate a face, name, and voice with the answers given.” 1 Significantly better than audio only and no difference with live 2Significantly better than audio only but worse than live

29. User study #2 – Transcript Only 1 Significantly better than audio only 2Significantly worse than audio only

30. User study #2 – Transcript Only • Transcript alone was way too fast to really understand what was going on. I could kind of skim, and I got some information but I felt like I was just bouncing along. 1 Significantly better than audio only 2Significantly worse than audio only

31. AIR Conferencing Conclusions • Enhanced audio catch-up is superior to audio-only review. • Users preferred, felt more confident with, and performed better with enhanced-audio than audio-only review. • As good as live! • Results consistent between two studies • Audio + All showed the strongest benefit • Audio + Shared Workspace improved recall of facts • Audio + Transcript improved recall of explanations • Speech-to-Text transcript not as beneficial as expected

32. AIR Conferencing Conclusions • Cost/benefit tradeoff: • importance of information vs. overhead • Ensure seamless use so as to not detract from the meeting • Next Steps: • Enhanced timeline index • Multi-tasking • AIR for face-to-face meetings (with mobile phones/laptops & headsets)

33. AIR Vision • Replaying life should be “a given” • But sometimes participation in real-time is important, so we need to enable catch-up

34. Thank you! Questions?