Livenotes

1. Livenotes A System for Cooperative and Augmented Note-Taking in Lectures Matthew Kam, Jingtao Wang, Alastair Iles, Eric Tse, Jane Chiu, Daniel Glaser, Orna Tarshish and John Canny University of California, Berkeley, USA

2. Video

3. Outline • Motivation • Solution • Experiment • Results • Implications • Conclusion

4. Motivation: Problem Statement • Constructivism • Learners are not blank slates that teachers write on • Learners need to actively construct their own understanding and knowledge • But large lecture classes are not conducive for active learning • Passive mode of oral dissemination • Lack of interactivity among students • Lack of interactivity with instructor

5. Motivation: Precedents • Face-to-Face Tutored Video Instruction (TVI), Distributed TVI (Gibbons, Stanford) • Group review of pre-recorded lectures • Regular pauses for small-group discussion • Students using DTVI received grades 0.5 std dev higher than non-TVI students (Smith et al. 1999) • Peer Instruction (Mazur, Harvard) • Lecture pauses for small-group discussion with neighbors • Improvements in conceptual understanding and problem-solving (Crouch and Mazur 2001)

6. Motivation: Small-Group, Cooperative Learning • More than 375 research studies since 1898 (Johnson and Johnson 1989) • Cooperative group learning results in greater • Efforts to achieve • Higher-level reasoning • Transfer from original context to new situations • Generation of new ideas and solutions

7. Motivation:Background Lecture Notes • (Hartley 1978, Kiewra et al. 1988) Experiments on note-taking that compared students annotating over: • Complete lecture notes provided by instructor, vs. • Skeletal (i.e. partial) notes, vs. • No background notes • Results: students were found to achieve maximum retention with skeletal notes

8. Livenotes Recap • Both a technology and educational practice • Large lecture classes • Small-group discussions in ongoing lecture • Cooperative note-taking: Combines real-time note-taking with discussion • Augmented note-taking: Skeletal slides for students to annotate over

9. Related Systems • No interaction between students • Classroom Presenter (Washington) • StuPad, eClass (Georgia Tech) • No real-time interaction between students, i.e. sharing of notes takes place after lecture • NotePals (Berkeley) • Limited real-time interaction between students • OneNote (Microsoft)

10. Outline • Motivation • Solution • Experiment • Results • Implications • Conclusion

11. Livenotes Evolution • 2000: Implemented in Java, for WinCE Clios • Late 2000 to early 2003: 5 small-scaledeployments using Clios, laptops and Tablet PCs • Spring 2003: Medium-scale experiment in undergraduate class using Tablet PCs • Since 2003: Ported to Microsoft .NET

12. Livenotes User Interface Group awareness (e.g. each user’s page number) Unique user colors Import background slides Pen and keyboard input

13. Client-Server Topology Group 1: • 802.11b networking • Large class broken down into many small groups (3-7 students) • One Tablet per group is set to server mode • Other members’ Tablets connect wirelessly to group’s server Server Clients … Group n: Server Clients

14. Outline • Motivation • Solution • Experiment • Results • Implications • Conclusion

15. Hypotheses Cooperative note-taking:Shared whiteboard interface enhances learning through cooperative note-taking and discussion Augmented note-taking:Background slides enhances learning by augmenting student note-taking

16. Experiment • Spring 2003 undergraduate HCI class • 21 volunteers, randomly partitioned into • Cooperative note-takers • Individual note-takers (control group) • 4 weeks (7 lectures) • Preloaded skeletal PowerPoint slides

17. Previous Observation • From 5 previous deployments, we learned that • Graduate students engaged spontaneously in group discussions • Undergraduates were not used to discussing lecture material with one another • For this experiment (with undergraduates), we held short, live group discussions in the classroom

18. Data Collection • Short quizzes (4 lectures) • Survey questionnaires • First week of deployment (~38% response rate) • End of semester (~29%) • Qualitative interviews (3 users) • Transcripts of students’ notes (~1581 pages)

19. Quantitative Analysis • Unit of analysis: mark • Spatio-temporally contiguous segment of user input • E.g.: “This lecture is very interesting” • Quantitative hand-coding of ~1581 pages

20. Taxonomy of Marks • Note-taking: someone taking notes on lecture • Commentary: someone making a statement • Question: someone soliciting a response • Answer: response to a question, clarification • Reinforcement: contribution to an existing thread

21. Outline • Motivation • Solution • Experiment • Results • Implications • Conclusion

22. Cooperative Note-Taking:Richer Notes • Cooperative note-taking group engaged in more than twice as much activity as individual note-taker

23. Cooperative Note-Taking:Richer Notes • Almost one quarter of marks made by cooperative note-takers were attributed to group interaction

24. Student Learning • Survey question: “How did Livenotes, if at all, assist your learning in lecture?” • Early survey after 2 sessions with Livenotes:75% of respondents self-reported affirmatively • Survey after semester (i.e. 7 lecture sessions): 83% of respondents self-reported affirmatively

25. Cooperative Note-Taking: Taking Turns to Take Notes • 66% of survey respondents agreed that cooperative note-taking is more useful • “Someone else might note something that I missed or hadn’t realized.” • “I liked how note-taking became a cooperative effort … someone can take over if another user is still inputing some notes, but the prof [had] moved on already.”

26. Cooperative Note-Taking: Paying Greater Attention • 36% of students who self-reported learning benefit explicitly attributed that to social aspect of cooperative note-taking:“Helped me to focus more in lecture. Often I fall asleep/lose attention in lecture. Having group members to respond to kept me better on track.”

27. Cooperative Note-Taking: Dual Conversations • Need to keep up with both lecture and on-tablet conversation:“It is helpful to be able to discuss questions. However, this does take attention away from the lecture if you are focusing on answering/asking a question.”

28. Cooperative Note-Taking: Decreasing Distraction • Is “running Livenotes during class distracting?” (1 = extremely distracting, 5 = not distracting at all) • Survey after two lectures: 2.6 out of 5 • Survey after deployment ended: 3.83 out of 5 • From student notes, “playful” behavior were observed to disappear almost completely after 2 lectures

29. Cooperative Note-Taking:Unanswered Group Questions • Students did not have time to answer some questions because they needed to keep up with lecturer • Some questions were unanswered because no group member knew the answer

30. Cooperative Note-Taking:Interaction During Pauses • Group interaction during pauses in lecture accounted for over half of group activity

31. Redeeming PowerPoint • Criticisms leveled at Microsoft PowerPoint • The Cognitive Style of PowerPoint (Edward Tufte) • Death by PowerPoint webpage • I  Powerpoint (David Byrne, Talking Heads) • Too boring, passive, does not promote active engagement with material • But students commented that augmented note-taking is “like having a conversation with the professor”

32. Augmented Note-Taking: Observed Behaviors • Elaborated on bullets • Appended bullets to list • Concurred and disagreed with bullet • Noted gist of HCI principles • Noted advantage and disadvantage of HCI technique • Answered questions in bullets

33. Augmented Note-Taking:Elaborated on Bullets

34. Augmented Note-Taking:Appended Bullets to List

35. Augmented Note-Taking:Answered Questions in Bullet

36. Augmented Note-Taking: Answered Questions in Bullet • Students responded to questions in bullets even when when they were not cooperative note-takers • Each group responded to 35% of the questions • Each question received a response from 36% of the groups

37. Augmented Note-Taking: Student Learning • Several high-quality notes in both individual and cooperative note-taking groups resulted from students “working off” bullets • Possibly due to bullets focusing student attention to relevant portions of lecture • A larger proportion (55%) of students who self-reported learning benefit attributed it to augmented note-taking, compared to cooperative note-taking. • Half of this sub-group attributed that to having slides at hand to annotate over

38. Quiz Scores • No statistical significance • But sample size was too small due to poor attendance at end of semester [1]Cooperative note-takers. [2]Individual note-takers. [3]Quiz scores presented in this table are normalized on a scale of 100.

39. Outline • Motivation • Solution • Experiment • Results • Implications • Conclusion

40. Student-Instructor Interaction • To help instructor assess student learning, we deployed feedback feature in last two sessions • Students provide instructor with real-time, anonymous lecture feedback • Recently allowed students to alert instructor that they have questions

41. Recommendation:Background Slides as Scaffold • Bullets are a lightweight means for lecturer to engage actively with students during class • Posing questions • Counter-intuitive bullets • Provocative statements • Direct student attention to critical parts of lecture • E.g. prompts such as “Pros?” and “Cons?” with blank spaces for students to fill in

42. Outline • Motivation • Solution • Experiment • Results • Implications • Conclusion

43. Conclusion • Cooperative note-taking • Richer variety of notes, higher-order thinking • More than twice as much notes as individuals • Members took turns to take notes • Students kept awake to interact with group • Augmented note-taking • Observed dialogue with bullets • Reflected higher-order thinking • High-quality notes resulted from “working off” bullets

44. Acknowledgement • Corporation for Education Network Initiatives in California • Microsoft Research • National Science Foundation • Qualcomm • Volunteers from Computer Science 160, Spring 2003 • Public domain source code by James R. Weeks

45. Questions? Livenotes can be downloaded from: http://www.cs.berkeley.edu/~mattkam/livenotes Matthew Kam, Ph.D. student Department of Electrical Engineering and Computer Sciences, and Berkeley Institute of Design University of California at Berkeley, USA mattkam@eecs.berkeley.edu