Achieving Learning Objectives Online: Not All Platforms are Equal!

1. Achieving Learning Objectives Online: Not All Platforms are Equal! Nick FeamsterGeorgia Tech

2. A Tale of Two MOOCs • A free Coursera MOOC on Software Defined Networking to over 50,000 enrolled students • Scalable delivery, assignments, and forums • A for-credit Udacity “MOOC” on Computer Networking for Georgia Tech’s Online MS program, to 200+ enrolled students • Not so scalable… Subtle differences in platform design have significant effects on a course’s ability to scale.

3. MOOC #1: Software Defined Networking • Extremely hot, new topic • No existing “real world” courses on the topic (yet) • A chance to develop an archetype course (and material) that others might use in their own courses • I wanted to be copied. This seemed like a good way. • Boutique topic means that there were no “set in stone” ways of teaching the course • Easier to think outside the box, since there was no existing material anywhere http://blog.sflow.com/2012/05/software-defined-networking.html • Two Offerings So Far: • Summer 2013(6 weeks) • Summer 2014 (8 weeks)

4. Who Takes The Course? • 50,000+ people registered each year • 10,000 people active each week • 1,000-3,000 watching lectures(3,500+ in 2014) • 500+ turn in programming assignments each year

5. Other Demographics (Summer 2014) • 55,000+ Enrolled • 197 Countries • 31% from the United States • 15% from India • Sex and Age • 86% Male • 74% 20-39 years old • Education and Employment • 58% Employed full-time • 26% Full-time students • 38% have Undergrad degree, 31% have Masters

8. What the Instructor Sees

9. How Was the Course Developed? • Course lesson plan, with learning objectives • Top down design: • Figure out modules, continue subdividing until you get 10-minute “lectures” • Production • Lecture Filming and Production • Assignments • Syllabus • Operation • Forums

10. Filming Take 1: Studio • Started in a studio at Georgia Tech • This quickly proved intractable • Studio is distracting. Lots of people watching makes recording nerve-wracking. • Studio time is limited, contentious, and expensive. • Post-production quality was not to my liking. • Faculty members need scheduling autonomy. • A lot different from a TV interview!

11. Filming Take 2: Camtasia • Studio people will claim that quality suffers. • I believe this is bogus, for several reasons. • People can take their time recording, take breaks, record when they are “in the zone”. • Screen captures from a laptop are easy. • Recording quality from a good laptop is quite fine. • Nobody has ever complained about the quality of the recordings. Content is what matters most. • Takes a little getting used to at first (looking at the camera, etc.) • You have to do your own editing.

12. Producing Lectures in Camtasia • General rule:10-15 minutes of lecture == 2 hours of recording and editing • This does not include time to prep slides, etc. • This is way different than a usual lecture! • Biggest lesson: Silence is quite easy to edit! • Stumbling is OK: Just say the same thing again • You can also “practice” while the video is rolling • Need to find a quiet, well-lit location • Takes patience, but it starts to be quite fun

13. General Lecture Production Process • Determine learning objectives • Research the topic • Write slides • General format: Summary/outline, 7-10 content slides, wrap-up • Plan demonstrations • Plan on-screen landscape • Record (one sitting for consistency…big challenge!) • Edit and export • Upload • Note: You can prepare lectures out-of-order!

14. New Modes of Lecture Production • On-screen demonstrations • Videos lend themselves very nicely to on-screen demonstrations • User can see things in “life size” • User can pause, rewind, etc. Very helpful tutorials for working through assignments • Interviews with luminaries in the field • Google “On Air” Hangouts • Can be done anywhere, and people are willing! • Process: Invite, script questions, send, revise. Stay roughly on script • Many people have since copied this idea (Nick McKeown, Alex Orso, Mike Hicks, etc.)

15. Lessons from Lecture Production • Lots of money is not needed. Students want content, clearly delivered. Camtasia works. • A 45–60-minute lecture is about 15 minutes of well-polished video. Content distills well. • Lecturer is always “ahead” of the students. This creates interesting time-shifting effects, with advantages and disadvantages. • MOOCs are asynchronous and autonomous for the student. This is a big reason for their success. Production should be the same.

16. How Was the Course Developed? • Course lesson plan, with learning objectives • Top down design: • Figure out modules, continue subdividing until you get 10-minute “lectures” • Production • Lecture Filming and Production • Assignments • Syllabus • Operation • Forums

17. Developing Scalable Quizzes: Stick to the Basics • Problem:The platform is not very good at handling anything that’s not multiple choice or simple numerical answer. • Short answers become massive regular expressions • …will likely improve as the software gets better • Stick to multiple choice if possible • Main differences from a normal MC quiz • Helps to write up explanations for each option • Need to figure out if students can re-take the quiz, and if so, how many times

18. Developing Assignments is Hard • Wanted to teach concepts and programming • Not a “heavy-duty” programming course, but some concepts are best explained and learned through short code exercises. • Code submission possible, but grading must be done on regular expression evaluation of output • Problem: How to guarantee uniform programming environments?

19. Uniform Environment: Virtual Machine • Asked students to: • Install VirtualBox (or equivalent) virtual machine emulator • Download 64-bit VM image that had everything pre-packaged (no asking students to install software themselves). Kind of like a lab at home. • This has generally worked. A few problems • VM is huge (~1 GB), so have to get it right the first time • Not everyone has a 64-bit machine • Certain performance, even when run in an emulator, varies depending on underlying hardware

20. How Was the Course Developed? • Course lesson plan, with learning objectives • Top-down design: • Figure out modules, continue subdividing until you get 10-minute “lectures” • Production • Lecture Filming and Production • Assignments • Syllabus • Operation • Forums

21. Communicating with 10,000 Students Can Scale! • I feared absolute disaster • Small glitches in assignments in a classroom of 50 students create mayhem. • I feared that I would be spending my life answering minutia in online forums. • Thankfully, this did not happen! • Lesson: Giving an assignment to 10,000 students in a MOOC is easier than giving it to 50 in a classroom!

22. Why Assignments Can Scale A student’s blog post describing part of the assignment setup in detail… • First of all, you do your best to work out the bugs, mostly out of fear  • We attempted everything that we assigned, sometimes on multiple platforms • More importantly: Self-selection! • Everyone doing the assignments wants to be there • This makes a huge difference • People not only fix issues themselves, they rewrite documentation • Caveat: Course difficulties vs. platform difficulties are currently hard to tease apart. Some of this is “user error”.

23. Some Feedback…

24. A Tale of Two MOOCs • A free Coursera MOOC on Software Defined Networking to over 50,000 enrolled students • A for-credit Udacity “MOOC” on Computer Networking for Georgia Tech’s Online MS program, to 200+ enrolled students

25. “MOOC” #2: Online MS at GT • Online version of CS 6250: Computer Networking • Two offerings thus far(Spring and Summer 2014) • About 120+ students in each offering • Paying tuition for a “degree” • Not specialists in networking • Technical difference: Platform

26. How Was the Course Developed? • Top-down planning, with “scripts” • Several full-time staff members • Course developer from Udacity • Video editor from Udacity • Part-time from others • Three paid TAs/Graders • Many other people handling logistics • Course recorded in studio, ahead of time, with custom equipment • Recording done in “marathon stints” • Not possible to integrate interviews • No auto-grading, little visibility of content, etc. • Initially, all material released at once (huge mistake)

27. Comparing the Approaches

28. Leave the Pedagogy to the Instructors • Substance trumps style. • A MOOC platform should allow instructors to deliver the content in whatever format they see fit. • Mandating a particular “style” emphasizes style over substance.

29. GT OMS Degree “MOOCs” Are Neither Massive nor Open • One of the important features of MOOCs that allows scaling is community support. • Things that dilute this support: • Students who payand expect staff contact • Students who are not interested in the content, but rather trying to get a degree • “Go at your own pace” style (students are on their own, course staff can’t keep up) • Other platform-specific factors prevent scaling • Instructor has less autonomy in creation and management of content • Support for auto-grading is critical!

30. Lesson #1: MOOCs Change the Game for Instruction • Everyone is watching you • In seven years, never got feedback (or acknowledgment) about on-campus course • Now, everyone’s eyes are on you • Huge pressure to do a good job • MOOCs are a “forcing function” for introducing new media into the classroom • I wouldn’t have thought to do “on air” interviews for a classroom; people would likely decline them anyhow • I will likely re-use some MOOC content in the “real” classroom

31. Lesson #2: Time Investment Pays Off • Typical CourseCreation (45-minute lecture) • 1-2 hours to create slides • 1 hour to deliver the lecture • Number of students: 50-100 • Total: 3 hours per lecture • … but then you have to do it again! (can’t save an awesome lecture, students can’t rewatch, etc.) • MOOC Course Creation (10-minute video) • 1-2 hours to create slides • 2-3 hours to produce the lecture • Total: 3-5 hours per lecture • Number of students: 500-100,000 • … but, you have the archive. Can keep improving it, use it to enhance in-classroom discussion, etc.

32. Lesson #3: Certain Aspects of MOOCs Scale Surprisingly Well • Lectures are watchable by thousands • The scale can be a motivator for guest lectures! • Programming assignments can be graded automatically, if designed well • Self-motivated, self-selected students fix glitches, help each other out

33. Conclusions • Scalable MOOCs require • Community engagement • Low barriers to creating content • Support for a diversity of delivery modes to match the material • Not all platforms are equal • A platform should give the instructor autonomyand visibility. • For-credit “MOOCs” face far different scaling challenges than true MOOCs. • At the moment: neither massive, nor open