“ Crowdsourcing ” a Textbook: 120 Student Authors Writing on a Wiki Edward F. Gehringer

1. “Crowdsourcing” a Textbook:120 Student Authors Writing on a Wiki Edward F. Gehringer Department of Computer Science Dept. of Electrical & Computer Engineering North Carolina State University Supported by the National Science Foundation Feedback: http://tinyurl.com/expertiza-form Web site: http://tinyurl.com/expertiza-site

2. Introduction: Why wiki textbooks? The administrative burden Our course The process Survey results Working with student authors The software and the process Summary Outline Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

3. Introduction Wikis are one of the premier collaborative spaces on the Web. Editing in place No need to “disseminate” to others Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

4. Why wiki textbooks? • Important for students to “take ownership” of learning • Need to evaluate different points of view • Compatible with constructivism • No scientific evaluation of textbook effectiveness • No comparative studies • Even buying textbook doesn’t help … • Contrast that with wiki textbooks, where research exists Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

5. Why wiki textbooks, cont. “Writing across the curriculum” Students write for an audience of their peers Feedback helps them improve Allow direct dissemination without publisher Combat high textbook prices Important to learn to “find truth” on the Web Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

6. Outline Introduction: Why wiki textbooks? The administrative burden Our courses The process Survey results Working with student authors The software and the process Summary Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

7. The administrative burden Chapters need to be spaced out over the semester Precedence relationships must be obeyed Students are usually allowed to choose what they write on Electronic signup gives everyone an equal chance Multiple deadlines needed for each chapter Choose topic, submit, review, resubmit, etc. Students must be reminded … “Handles” for the wiki Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

8. Outline Introduction: Why wiki textbooks? The administrative burden Our courses The process Survey results Working with student authors The software and the process Summary Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

9. Spring course CSC/ECE 506, Architecture of Parallel Computers Taken by all Computer Engineering students Optional for Computer Science students Small class … 15 students New textbook Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

10. The textbook supplement Good book, but there is room to augment its coverage … covers theory and design fundamentals, a bit thin in showing how current machines realize them Two goals Cover topics not covered in text. Apply concepts to current parallel machines Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

11. Fall course • CSC/ECE 517, Object-Oriented Design • 4 textbooks for different topics • None cover all topics of interest • Very broad range of material • Large course • 109 students finished the course Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

12. Outline Introduction: Why wiki textbooks? The administrative burden Our course The process Survey results Working with student authors The software and the process Summary Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

13. Signing up Every two weeks, students sign up for chapters Depending on size of course There are more or fewer topics 2 to 10 More or fewer authors allowed to sign up for each topic 2 or 3 Students encouraged to work in pairs Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

14. Exercise 1: Sign up • Go to http://expertiza.ncsu.edu, • log in with “eli#” and password “eli”, • accept terms of use, • click on “ELI demo”, • & then “Signup sheet”. • Choose oneof the topics. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

15. Writing on the wiki • Same user-IDs on the wiki as on Expertiza • … unless handles are used! • We tell them how to name their pages. • ELI_demo/username Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

16. Exercise 2: Create a Wiki Page • Go to http://tinyurl.com/ELI-demo • You will be redirected to this page: • Log in with “eli#” and password “eli”. • Return to ELI_demo, click on your user # and start editing. • Save your page. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

17. Exercise 2 (cont.): Submit your page • Go back to http://expertiza.ncsu.edu. • Select “Others’ work”. • Click on the topic you want to review. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

18. Our first assignment writeup Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

19. Which kind of feedback do you use? a.  Feedback by instructor b.  Feedback by TA c.  Peer feedback, not anonymous (e.g., face to face) d.  Anonymous peer feedback e.  Feeback by teammates Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

20. Peer review • Submissions are peer-reviewed through our Expertiza system. • ≈ 5 reviewers give feedback to each team. • Teams improve work based on feedback • Reviewers suggest a grade. • Instructor/TAs review student recommendations, assign final grades. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

21. Exercise 3: Review a Wiki Page • After you save the page you were editing, go back to http://expertiza.ncsu.edu. • Click on “Others’ work”, then on “Begin” for one of the reviews. • Assign scores and give text comments. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

22. Initial experience Students were not clear about the purpose … Substantially duplicated the textbook Did not concentrate on recent developments As a result, we began pre-reviewing the student submissions. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

23. More hurdles Our initial feedback served to tell students what they needed to re-do. Students had just gone through the first few search hits. They sometimes hit on info covered in next chapter of text. So … we met weekly to discuss upcoming chapters made list of material to be covered Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

24. Later assignment writeup Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

25. Later assignment writeup, cont. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

26. Differences in a large class • 3 teams/topic instead of 2 • “Too many” topics needed • Students suggest topics; instructor approves • No time to read all submissions … Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

27. Outline Introduction: Why wiki textbooks? The administrative burden Our course The process Survey results Working with student authors The software and the process Summary Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

28. Progress? Students’ second wiki chapters received almost the same average grade (82.7%) as the first (82.8%). But reviews were more demanding. Perceived quality of contributions is better. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

29. Survey results — contributions Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

30. Survey results — process Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

31. Survey results — software Only the first question above received an unfavorable average response. Since then, Expertiza has been modified to send e-mail reminders. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

32. Outline Introduction: Why wiki textbooks? The administrative burden Our course The process Survey results Working with student authors The software and the process Summary Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

33. Working with student authors • Discouraging plagiarism • Iterative improvement • Organization • Permission Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

34. Outline Introduction: Why wiki textbooks? The administrative burden Our course The process Survey results Working with student authors The software and the process Summary Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

35. Lessons learned 1 • Signup sheet is very useful • The only reliable way to manage topics in a large class. • Tells reviewer if author wrote on correct topic • Handles are not always a win • Students would change handles, • then forget and use their old handle. •  Expertiza wouldn’t find their work • Simpler just to turn off wiki “history.” Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

36. Lessons learned 2 • Students want to submit work explicitly • Confused by concept of Expertiza “finding” work on wiki • Review system sometimes confused about which assignment submission is for. • Students need e-mail reminders to stay on track • Otherwise, there are too many deadlines to remember • This may contribute to “too much rating for this course.”. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

37. Practical ideas • Topic suggestions • Social bookmarking • Start slowly—wiki only … Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

38. Outline Introduction: Why wiki textbooks? The administrative burden Our course The process Survey results Working with student authors The software and the process Summary Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

39. Summary • A wiki is an excellent medium for collaborative writing. • With appropriate software support, a class can engage collaboratively in a large project. • Students in a parallel-architecture course made significant contributions to their textbook. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

40. Exercise 1: Sign up • Go to http://expertiza.ncsu.edu, • log in with “eli#” and password “eli”, • accept terms of use, • click on “ELI demo”, • & then “Signup sheet”. • Choose oneof the topics. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

41. Exercise 2: Create a Wiki Page • Go to http://tinyurl.com/ELI-demo • You will be redirected to this page: • Log in with “eli#” and password “eli”. • Return to ELI_demo, click on your user # and start editing. • Save your page. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

42. Exercise 2 (cont.): Submit your page • Go back to http://expertiza.ncsu.edu. • Select “Others’ work”. • Click on the topic you want to review. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form

43. Exercise 3: Review a Wiki Page • After you save the page you were editing, go back to http://expertiza.ncsu.edu. • Click on “Others’ work”, then on “Begin” for one of the reviews. • Assign scores and give text comments. Gehringer, “Crowdsourcing” a Textbook Feedback: http://tinyurl.com/expertiza-form