Monitoring MOOCs : Which Information Sources Do Instructors Value?

1. Monitoring MOOCs:Which Information Sources Do Instructors Value? Kristin Stephens-Martinez, Marti A. Hearst, Armando Fox UC Berkeley

2. Main Takeaways • Goal: Information display for MOOC instructors • Survey: What information MOOC instructors value? • Results: • Quantitative information valued same as smaller online courses • Assignment grades, demographics, activity patterns • Qualitative information is important • Forums, student surveys • Visualization mockups useful while course running and for review

3. Prior Work: Data to Visualize • Performance • Grades, cumulative concept performance • Access and Activity Patterns • What, how much, how long, and when content opened • Path through content • Assignment status • Forum Discussions • Student forum activity • Post meta data • Post structure • Student Demographics • Location, age, learning style • Course goals

4. Prior Work: How Visualize Data • Standard Graphs • Scatter plots, bar graphs, line graphs, pie charts, heat maps • Innovative Visualizations • Node graphs, icons, 3D, timeline spiral graphs • Interactive Techniques • Sorting • Filtering • Drill down • Clustering

5. Why need a survey? • Prior work on • Long distance learning • Course management systems • What about MOOCs? • Massive student body • Open enrollment “Small” Courses

6. Survey Participants • Recruit • Coursera • edX • Udacity • 92 MOOC Instructors • 67 complete survey (73%)

7. Survey Participants 95% Create 1 or more MOOCs 94% Run 1 or more MOOCs

8. Survey Participants

9. Survey Participants

10. Course Monitoring Goals • During class • Assignment issues • Find trouble students and what struggle with • Gauge difficulty of exam problem • Course is appropriate difficulty level • After class • Find most engaging content • Struggling parts of class • TA effectiveness • Improve topic presentation • Least interesting content .

11. Information Sources • Quantitative • Assignment grades • Student answers to problems • Student’s view pattern of online content • Number of times students attempt a problem • Grades for self-check questions • Qualitative • Discussion forum • Class survey • Discussion with students • Ask the TA • Chat room logs

12. Cross: Goals X Info Sources Assignment issues Assignment issues Assignment issues Find trouble students and what struggle with Find trouble students and what struggle with Gauge difficulty of exam problem Course is appropriate difficulty level Find most engaging content Struggling parts of class Find trouble students and what struggle with TA effectiveness Improve topic presentation Least interesting content

13. Cross: Goals X Info Sources Assignment issues Assignment grades Assignment grades Assignment grades Assignment grades Assignment grades Student answers to problems Student answers to problems Student answers to problems Student answers to problems Student answers to problems Student’s view pattern of online content Student’s view pattern of online content Student’s view pattern of online content Student’s view pattern of online content Student’s view pattern of online content Number of times students attempt a problem Number of times students attempt a problem Number of times students attempt a problem Number of times students attempt a problem Grades for self-check questions Grades for self-check questions Grades for self-check questions Grades for self-check questions Number of times students attempt a problem Discussion forum Discussion forum Discussion forum Discussion forum Class survey Class survey Class survey Class survey Grades for self-check questions Discussion with students Discussion with students Discussion with students Discussion with students Ask the TA Ask the TA Ask the TA Ask the TA Discussion forum Chat room logs Chat room logs Chat room logs Chat room logs Class survey Find trouble students and what struggle with Discussion with students Ask the TA Chat room logs

14. Cross: Goals X Info Sources Use Would use if available Do not/Would not use Assignment issues Assignment grades Assignment grades Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Student answers to problems Student answers to problems Student’s view pattern of online content Student’s view pattern of online content Number of times students attempt a problem Number of times students attempt a problem Grades for self-check questions Grades for self-check questions Discussion forum Discussion forum Class survey Class survey Discussion with students Discussion with students Ask the TA Ask the TA Chat room logs Chat room logs Find trouble students and what struggle with

15. Value of Information Sources: Across Goals

16. Value of Information Sources: Across Goals

17. Value of Information Sources: Across Goals

18. Value of Information Sources: Across Goals

19. Value of Information Sources: Across Goals Outlier Qualitative Quantitative

20. Value of Information Sources: By Goal Use/Would Use Skip Not Use

21. Value of Information Sources: By Goal Use/Would Use Skip Not Use

22. Value of Information Sources: By Goal Use/Would Use Skip Not Use Qualitative Quantitative

23. Value of Information Sources: By Goal Use/Would Use Skip Not Use Qualitative Quantitative

24. Value of Information Sources: By Goal Qualitative + Quantitative No Agreement Qualitative + some View Patterns Use/Would Use Skip Not Use Qualitative Quantitative

25. What does qualitative mean? • Qualitative Data: • Forums • Class Surveys • Student Discussions • Teaching Assistant Student Voice People

26. Visualization Mockups • Useful • (Likert response) • Easy to understand • (Likert response) • When use it? • While the course is running • When preparing new material • When preparing by reviewing past courses

27. Visualization Mockups • Majority participants find useful and understandable • Majority participants use • During class ⇨ All • Review ⇨ All, except (quantitative) forum

28. Limitations • Survey • Some questions ask predict future behavior • (people not necessarily good at) • Instructors not experienced with all information sources • Forum • Small sample of student body

29. Other Work: Metrics Tab Stanford edX Branch

30. Summary • Goal: Information display for MOOC instructors • Survey: What information MOOC instructors value? • Results: • Quantitative information valued same as smaller online courses • Qualitative information is important • Visualization mockups useful while course running and for review • Future work: • More visualizations • Improve communication of student voice to instructor

31. Spare Slides

32. Mockup: Boxplot Grades Distribution

33. Mockup: Stacked Bar Grades Distribution

34. Mockup: Stacked Bar Attempts Distribution

35. Mockup: Stacked Bar Views Distribution

36. Mockup: Line Graph Forum New Posts Views