1 / 36

Monitoring MOOCs : Which Information Sources Do Instructors Value?

Monitoring MOOCs : Which Information Sources Do Instructors Value?. Kristin Stephens-Martinez, Marti A. Hearst, Armando Fox UC Berkeley. Main Takeaways. Goal: Information display for MOOC instructors Survey: What information MOOC instructors value? Results:

miya
Download Presentation

Monitoring MOOCs : Which Information Sources Do Instructors Value?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Monitoring MOOCs:Which Information Sources Do Instructors Value? Kristin Stephens-Martinez, Marti A. Hearst, Armando Fox UC Berkeley

  2. Main Takeaways • Goal: Information display for MOOC instructors • Survey: What information MOOC instructors value? • Results: • Quantitative information valued same as smaller online courses • Assignment grades, demographics, activity patterns • Qualitative information is important • Forums, student surveys • Visualization mockups useful while course running and for review

  3. Prior Work: Data to Visualize • Performance • Grades, cumulative concept performance • Access and Activity Patterns • What, how much, how long, and when content opened • Path through content • Assignment status • Forum Discussions • Student forum activity • Post meta data • Post structure • Student Demographics • Location, age, learning style • Course goals

  4. Prior Work: How Visualize Data • Standard Graphs • Scatter plots, bar graphs, line graphs, pie charts, heat maps • Innovative Visualizations • Node graphs, icons, 3D, timeline spiral graphs • Interactive Techniques • Sorting • Filtering • Drill down • Clustering

  5. Why need a survey? • Prior work on • Long distance learning • Course management systems • What about MOOCs? • Massive student body • Open enrollment “Small” Courses

  6. Survey Participants • Recruit • Coursera • edX • Udacity • 92 MOOC Instructors • 67 complete survey (73%)

  7. Survey Participants 95% Create 1 or more MOOCs 94% Run 1 or more MOOCs

  8. Survey Participants

  9. Survey Participants

  10. Course Monitoring Goals • During class • Assignment issues • Find trouble students and what struggle with • Gauge difficulty of exam problem • Course is appropriate difficulty level • After class • Find most engaging content • Struggling parts of class • TA effectiveness • Improve topic presentation • Least interesting content .

  11. Information Sources • Quantitative • Assignment grades • Student answers to problems • Student’s view pattern of online content • Number of times students attempt a problem • Grades for self-check questions • Qualitative • Discussion forum • Class survey • Discussion with students • Ask the TA • Chat room logs

  12. Cross: Goals X Info Sources Assignment issues Assignment issues Assignment issues Find trouble students and what struggle with Find trouble students and what struggle with Gauge difficulty of exam problem Course is appropriate difficulty level Find most engaging content Struggling parts of class Find trouble students and what struggle with TA effectiveness Improve topic presentation Least interesting content

  13. Cross: Goals X Info Sources Assignment issues Assignment grades Assignment grades Assignment grades Assignment grades Assignment grades Student answers to problems Student answers to problems Student answers to problems Student answers to problems Student answers to problems Student’s view pattern of online content Student’s view pattern of online content Student’s view pattern of online content Student’s view pattern of online content Student’s view pattern of online content Number of times students attempt a problem Number of times students attempt a problem Number of times students attempt a problem Number of times students attempt a problem Grades for self-check questions Grades for self-check questions Grades for self-check questions Grades for self-check questions Number of times students attempt a problem Discussion forum Discussion forum Discussion forum Discussion forum Class survey Class survey Class survey Class survey Grades for self-check questions Discussion with students Discussion with students Discussion with students Discussion with students Ask the TA Ask the TA Ask the TA Ask the TA Discussion forum Chat room logs Chat room logs Chat room logs Chat room logs Class survey Find trouble students and what struggle with Discussion with students Ask the TA Chat room logs

  14. Cross: Goals X Info Sources Use Would use if available Do not/Would not use Assignment issues Assignment grades Assignment grades Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Use Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Would use if available Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Do not/Would not use Student answers to problems Student answers to problems Student’s view pattern of online content Student’s view pattern of online content Number of times students attempt a problem Number of times students attempt a problem Grades for self-check questions Grades for self-check questions Discussion forum Discussion forum Class survey Class survey Discussion with students Discussion with students Ask the TA Ask the TA Chat room logs Chat room logs Find trouble students and what struggle with

  15. Value of Information Sources: Across Goals

  16. Value of Information Sources: Across Goals

  17. Value of Information Sources: Across Goals

  18. Value of Information Sources: Across Goals

  19. Value of Information Sources: Across Goals Outlier Qualitative Quantitative

  20. Value of Information Sources: By Goal Use/Would Use Skip Not Use

  21. Value of Information Sources: By Goal Use/Would Use Skip Not Use

  22. Value of Information Sources: By Goal Use/Would Use Skip Not Use Qualitative Quantitative

  23. Value of Information Sources: By Goal Use/Would Use Skip Not Use Qualitative Quantitative

  24. Value of Information Sources: By Goal Qualitative + Quantitative No Agreement Qualitative + some View Patterns Use/Would Use Skip Not Use Qualitative Quantitative

  25. What does qualitative mean? • Qualitative Data: • Forums • Class Surveys • Student Discussions • Teaching Assistant Student Voice People

  26. Visualization Mockups • Useful • (Likert response) • Easy to understand • (Likert response) • When use it? • While the course is running • When preparing new material • When preparing by reviewing past courses

  27. Visualization Mockups • Majority participants find useful and understandable • Majority participants use • During class ⇨ All • Review ⇨ All, except (quantitative) forum

  28. Limitations • Survey • Some questions ask predict future behavior • (people not necessarily good at) • Instructors not experienced with all information sources • Forum • Small sample of student body

  29. Other Work: Metrics Tab Stanford edX Branch

  30. Summary • Goal: Information display for MOOC instructors • Survey: What information MOOC instructors value? • Results: • Quantitative information valued same as smaller online courses • Qualitative information is important • Visualization mockups useful while course running and for review • Future work: • More visualizations • Improve communication of student voice to instructor

  31. Spare Slides

  32. Mockup: Boxplot Grades Distribution

  33. Mockup: Stacked Bar Grades Distribution

  34. Mockup: Stacked Bar Attempts Distribution

  35. Mockup: Stacked Bar Views Distribution

  36. Mockup: Line Graph Forum New Posts Views

More Related