480 likes | 620 Views
STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL RESOURCES FOR COMPUTER & INFORMATION SCIENCE: Teaching with Merlot Collection. Emrah Orhun, Ph.D. Troy State University Montgomery Alabama, USA. Outline. Introduction Learning and Teaching Materials in The Merlot Collection
E N D
STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL RESOURCES FOR COMPUTER & INFORMATION SCIENCE: Teaching with Merlot Collection Emrah Orhun, Ph.D. Troy State University Montgomery Alabama, USA
Outline • Introduction • Learning and Teaching Materials in The Merlot Collection • Students’ Satisfaction with Merlot Materials in CIS Classes • Conclusions
Introduction • Difficulties in locating relevant and affordable materials of high quality • World Lecture Hall at the University of Texas • 83 categories to browse and search utilities to locate materials • WebCT (2002) • 1,250 publisher course content packs in more than 50 subject areas • customizable materials around which to build courses combined with the functionality of WebCT's course management software.
Quality issue • World Lecture Hallmaterials • courses offered at accredited colleges and universities around the world • WebCT packs • editorial standards of the publisher and author of the text
MERLOT (Multimedia Educational Resource for Learning and Online Teaching) • Web-based open resource designed primarily for faculty and students in higher education • emphasizes the quality issue
MERLOT repository • Over 8,000 free learning materials and support materials that can be integrated within the context of a larger course
Figure 1. Entries in the Applications sub/subcategory of Information Technology subcategory under Science and Technology category
Support resources • sample assignments • evaluations of the learning materials • discipline panel peer reviews • individual user reviews
Peer Review Process • Discipline communities • Biology, business, chemistry, engineering, health sciences, history, information technology, mathematics, music, physics, psychology, teacher education, and world languages.
Discipline teams (“MERLOT Editorial Boards”) • selected by the MERLOT partner organizations in accordance with the MERLOT guidelines. • develop evaluation standards, • conduct peer reviews of learning materials within the discipline, and • make connections with their professional organizations
Peer review process • Evaluated by at least two team members who have relevant expertise. • Review process stages • triagereview to identify worthy candidates • review of candidates • composite review report sent back to the authors for feedback and permission to post the review on MERLOT. • approved review is posted to the MERLOT Web site by the discipline leader
The evaluation criteria: • quality of content, • potential effectiveness as a teaching tool, • and ease of use.
Each dimension evaluated separately (on a scale of 1-5 stars, with 5 being the highest), and an overall rating is computed • A review must average three stars in order to be posted to the site.
Table 1 Growth of MERLOT materials, members, and peer reviews (May 28, 2002)
Table 1 Growth of MERLOT materials, members, and peer reviews (February 27, 2003)
Table 1 Growth of MERLOT materials, members, and peer reviewsfor IT Category
Ratio of Peer Reviews, Member Comments, and Assignments to Materials
Problem • Percentage of reviewed materials low Cause • Editorial peer review process too slow Potential Solution • Peer review by community of learners
Goal of Study • To find out to what extent MERLOT provides relevant and quality materials for the CIS courses.
Questions • Is the quality and quantity of learning materials in the MERLOT collection sufficient to satisfy CIS students’ knowledge/learning needs in specific courses? • Is the MERLOT collection ‘usable’ by the students? • Does the MERLOT collection induce the students to become members of learning communities? • Can peer review by students substitute for peer review by editorial boards?
Questions re-formulated • How do students rate their satisfaction with MERLOT collection? • How do students rate the learning materials as reviewers? Do students’ ratings of the learning materials differ from the ratings given by Discipline Panel members?
Teaching with the MERLOT Collection in CIS Courses • Summer-Fall 2002 semesters in the following CIS classes at Troy State University Montgomery: • CIS 5547 CAR Applied Systems Analysis • CIS 5547 CBR Applied Systems Analysis • CIS 3332 Software Engineering I • CIS 3357 Logical Structures of Computer Design • Overall, 41 students participated in the evaluation. A total of 252 evaluations were made.
Student Assignment • Evaluation of web-based learning materials • 1. Visit MERLOT (www.merlot.org) site and register as a member. • 2. For each chapter covered in the course: • a. Find a relevant learning material in the IT subcategory. • b. Review the material using the guidelines provided by MERLOT. In the table provided below, state what learning outcomes are supported by the material and post your review as a Member Comment on MERLOT. • c. Rate your satisfaction with MERLOT collection on a scale 1-5, with 5 implying that you were fully satisfied with the quality and quantity of learning materials you found in connection with your task, and 1 implying that you were not satisfied at all. • d. Find a relevant online learning material not contained in MERLOT, post it to MERLOT and add a Member Comment.
How do students rate their satisfaction with MERLOT collection?
COURSE Number of Students Number of Topics Number of Evaluations Average sdev CIS3357 Logical Structures of Computer Design 8 7 49 3.8 1.3 CIS5547 (A+B) Applied Systems Analysis 22 14 165 3.8 0.8 CIS3332 Software Engineering I 11 4 38 3.8 1.0 Students’ mean satisfaction ratings for MERLOT collection
Ch Topic N Average sdev Min Max Percent rated 4-5 1 Binary Numbers & Codes 8 4.7 1.4 3 5 87.5 2 Digital Circuits 8 4.6 1.5 3 5 75.0 3 Combinational Circuits 8 4.3 1.9 1 5 62.5 4 Sequential Logic Systems 8 4.2 1.9 1 5 62.5 5 Registers & Counters 8 3.8 2.1 1 5 50.0 6 Memory & Programmable Logic 5 3.2 1.5 1 5 40.0 7 Register Transfer & Computer Operations 4 3.8 1.9 1 5 75.0 CIS3357 Students’ satisfaction with MERLOT collection
Ch Topic N Average sdev Min Max Percent rated 4-5 1 Process 13 4.2 0.8 1 5 76.9 2 Project Management 12 4.2 0.4 2 5 91.7 3 Requirements Analysis I 11 3.8 1.2 1 5 72.7 4 Requirements Analysis II 10 3.6 1.2 1 5 70.0 5 Software Architecture 13 4.2 0.8 1 5 76.9 6 Detailed Design 12 4.2 0.4 2 5 91.7 CIS3332 Students’ satisfaction with MERLOT collection
Ch Topic N Average sdev Min Max Percent rated 4-5 1 Players in the System Game 22 3.6 0.8 2 5 59.1 2 Information System Building Blocks 20 3.7 0.8 2 5 55.0 3 Information System Development 19 3.6 1.2 1 5 63.2 4 Project Management 10 4.1 0.7 3 5 80.0 5 Systems Analysis 12 3.8 0.6 3 5 75.0 6 Requirements Discovery 12 3.9 0.8 3 5 66.7 11 Application Architecture and Modeling 10 3.9 0.7 3 5 70.0 CIS5547 Students’ satisfaction with MERLOT collection
A Object-Oriented Analysis and Modeling 9 3.7 0.7 3 5 55.6 B Object-Oriented Design and Modeling 9 3.9 0.8 3 5 66.7 13 Output Design and Prototyping 9 4.0 0.5 3 5 88.9 14 Input Design and Prototyping 9 4.2 0.4 4 5 100.0 15 User Interface Design 9 3.9 0.8 3 5 66.7 16 Systems Construction and Implementation 7 4.1 0.9 3 5 71.4 17 Systems Operations and Support 8 4.1 0.8 3 5 75.0 CIS5547 Students’ satisfaction with MERLOT collection (contd.)
How different are students’ evaluations from Editorial Peer Reviews?
Sample of Students • CIS3332 SOFTWARE ENGINEERING (16) • CIS5547 SYSTEMS ANALYSIS (27)
Students’ ratings for TECHtionary Editorial Board Rating: 3.00
Students’ ratings for Web-Database Integration Editorial Board Rating: 3.67
Is the mean student rating different from the rating given by the Editorial Board? • TECHtionary • Different (one-group t-value = 8.773, p=0.0001) • Web-Database Integration • Insufficient evidence (one-group t-value = 1.333, p=0.1975)
Conclusions • Students are satisfied with the quality and quantity of learning materials they found in the IT category. • Current data suggest that students rate materials differently than editorial boards. • The present limited study does not provide evidence for the notion that peer review by students can substitute for peer review by editorial boards.