STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL RESOURCES FOR COMPUTER & INFORMATION SCIENCE: Teaching with Merlot C

1. STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL RESOURCES FOR COMPUTER & INFORMATION SCIENCE: Teaching with Merlot Collection Emrah Orhun, Ph.D. Troy State University Montgomery Alabama, USA

2. Outline • Introduction • Learning and Teaching Materials in The Merlot Collection • Students’ Satisfaction with Merlot Materials in CIS Classes • Conclusions

3. Introduction • Difficulties in locating relevant and affordable materials of high quality • World Lecture Hall at the University of Texas • 83 categories to browse and search utilities to locate materials • WebCT (2002) • 1,250 publisher course content packs in more than 50 subject areas • customizable materials around which to build courses combined with the functionality of WebCT's course management software.

4. Quality issue • World Lecture Hallmaterials • courses offered at accredited colleges and universities around the world • WebCT packs • editorial standards of the publisher and author of the text

5. MERLOT (Multimedia Educational Resource for Learning and Online Teaching) • Web-based open resource designed primarily for faculty and students in higher education • emphasizes the quality issue

6. MERLOT repository • Over 8,000 free learning materials and support materials that can be integrated within the context of a larger course

7. Searching and Browsing in MERLOT

8. Figure 1. Entries in the Applications sub/subcategory of Information Technology subcategory under Science and Technology category

9. Figure 2. Search results for TECHtionary

10. Support resources • sample assignments • evaluations of the learning materials • discipline panel peer reviews • individual user reviews

11. Peer Review Process • Discipline communities • Biology, business, chemistry, engineering, health sciences, history, information technology, mathematics, music, physics, psychology, teacher education, and world languages.

12. Discipline teams (“MERLOT Editorial Boards”) • selected by the MERLOT partner organizations in accordance with the MERLOT guidelines. • develop evaluation standards, • conduct peer reviews of learning materials within the discipline, and • make connections with their professional organizations

13. Peer review process • Evaluated by at least two team members who have relevant expertise. • Review process stages • triagereview to identify worthy candidates • review of candidates • composite review report sent back to the authors for feedback and permission to post the review on MERLOT. • approved review is posted to the MERLOT Web site by the discipline leader

14. The evaluation criteria: • quality of content, • potential effectiveness as a teaching tool, • and ease of use.

15. Each dimension evaluated separately (on a scale of 1-5 stars, with 5 being the highest), and an overall rating is computed • A review must average three stars in order to be posted to the site.

16. Figure 3(a). Peer Review of TECHtionary (Part a)

17. Figure 3(b). Peer Review of TECHtionary (Part b)

18. Figure 3(c). Peer Review of TECHtionary (Part c)

19. Figure 4. Member Comments for TECHtionary

20. Current state of MERLOT

21. Table 1 Growth of MERLOT materials, members, and peer reviews (May 28, 2002)

22. Table 1 Growth of MERLOT materials, members, and peer reviews (February 27, 2003)

23. Table 1 Growth of MERLOT materials, members, and peer reviewsfor IT Category

24. Ratio of Peer Reviews, Member Comments, and Assignments to Materials

25. Problem • Percentage of reviewed materials low Cause • Editorial peer review process too slow Potential Solution • Peer review by community of learners

26. STUDENTS’ SATISFACTION WITH MERLOT MATERIALS IN CIS CLASSES

27. Goal of Study • To find out to what extent MERLOT provides relevant and quality materials for the CIS courses.

28. Questions • Is the quality and quantity of learning materials in the MERLOT collection sufficient to satisfy CIS students’ knowledge/learning needs in specific courses? • Is the MERLOT collection ‘usable’ by the students? • Does the MERLOT collection induce the students to become members of learning communities? • Can peer review by students substitute for peer review by editorial boards?

29. Questions re-formulated • How do students rate their satisfaction with MERLOT collection? • How do students rate the learning materials as reviewers? Do students’ ratings of the learning materials differ from the ratings given by Discipline Panel members?

30. Teaching with the MERLOT Collection in CIS Courses • Summer-Fall 2002 semesters in the following CIS classes at Troy State University Montgomery: • CIS 5547 CAR Applied Systems Analysis • CIS 5547 CBR Applied Systems Analysis • CIS 3332 Software Engineering I • CIS 3357 Logical Structures of Computer Design • Overall, 41 students participated in the evaluation. A total of 252 evaluations were made.

31. Student Assignment • Evaluation of web-based learning materials • 1.  Visit MERLOT (www.merlot.org) site and register as a member. • 2.       For each chapter covered in the course: • a.      Find a relevant learning material in the IT subcategory. • b.      Review the material using the guidelines provided by MERLOT. In the table provided below, state what learning outcomes are supported by the material and post your review as a Member Comment on MERLOT. • c.      Rate your satisfaction with MERLOT collection on a scale 1-5, with 5 implying that you were fully satisfied with the quality and quantity of learning materials you found in connection with your task, and 1 implying that you were not satisfied at all. • d.      Find a relevant online learning material not contained in MERLOT, post it to MERLOT and add a Member Comment.

32. Example of Relevant MERLOT Material

33. Example of New MERLOT material posted

34. How do students rate their satisfaction with MERLOT collection?

35. COURSE Number of Students Number of Topics Number of Evaluations Average sdev CIS3357 Logical Structures of Computer Design 8 7 49 3.8 1.3 CIS5547 (A+B) Applied Systems Analysis 22 14 165 3.8 0.8 CIS3332 Software Engineering I 11 4 38 3.8 1.0 Students’ mean satisfaction ratings for MERLOT collection

36. Ch Topic N Average sdev Min Max Percent rated 4-5 1 Binary Numbers & Codes 8 4.7 1.4 3 5 87.5 2 Digital Circuits 8 4.6 1.5 3 5 75.0 3 Combinational Circuits 8 4.3 1.9 1 5 62.5 4 Sequential Logic Systems 8 4.2 1.9 1 5 62.5 5 Registers & Counters 8 3.8 2.1 1 5 50.0 6 Memory & Programmable Logic 5 3.2 1.5 1 5 40.0 7 Register Transfer & Computer Operations 4 3.8 1.9 1 5 75.0 CIS3357 Students’ satisfaction with MERLOT collection

37. Ch Topic N Average sdev Min Max Percent rated 4-5 1 Process 13 4.2 0.8 1 5 76.9 2 Project Management 12 4.2 0.4 2 5 91.7 3 Requirements Analysis I 11 3.8 1.2 1 5 72.7 4 Requirements Analysis II 10 3.6 1.2 1 5 70.0 5 Software Architecture 13 4.2 0.8 1 5 76.9 6 Detailed Design 12 4.2 0.4 2 5 91.7 CIS3332 Students’ satisfaction with MERLOT collection

38. Ch Topic N Average sdev Min Max Percent rated 4-5 1 Players in the System Game 22 3.6 0.8 2 5 59.1 2 Information System Building Blocks 20 3.7 0.8 2 5 55.0 3 Information System Development 19 3.6 1.2 1 5 63.2 4 Project Management 10 4.1 0.7 3 5 80.0 5 Systems Analysis 12 3.8 0.6 3 5 75.0 6 Requirements Discovery 12 3.9 0.8 3 5 66.7 11 Application Architecture and Modeling 10 3.9 0.7 3 5 70.0 CIS5547 Students’ satisfaction with MERLOT collection

39. A Object-Oriented Analysis and Modeling 9 3.7 0.7 3 5 55.6 B Object-Oriented Design and Modeling 9 3.9 0.8 3 5 66.7 13 Output Design and Prototyping 9 4.0 0.5 3 5 88.9 14 Input Design and Prototyping 9 4.2 0.4 4 5 100.0 15 User Interface Design 9 3.9 0.8 3 5 66.7 16 Systems Construction and Implementation 7 4.1 0.9 3 5 71.4 17 Systems Operations and Support 8 4.1 0.8 3 5 75.0 CIS5547 Students’ satisfaction with MERLOT collection (contd.)

40. How different are students’ evaluations from Editorial Peer Reviews?

41. Distribution of materials in IT

42. Sample of Materials from IT Category

43. Sample of Students • CIS3332 SOFTWARE ENGINEERING (16) • CIS5547 SYSTEMS ANALYSIS (27)

44. Students’ Ratings

45. Students’ ratings for TECHtionary Editorial Board Rating: 3.00

46. Students’ ratings for Web-Database Integration Editorial Board Rating: 3.67

47. Is the mean student rating different from the rating given by the Editorial Board? • TECHtionary • Different (one-group t-value = 8.773, p=0.0001) • Web-Database Integration • Insufficient evidence (one-group t-value = 1.333, p=0.1975)

48. Conclusions • Students are satisfied with the quality and quantity of learning materials they found in the IT category. • Current data suggest that students rate materials differently than editorial boards. • The present limited study does not provide evidence for the notion that peer review by students can substitute for peer review by editorial boards.