260 likes | 291 Views
Design and Evaluation of Web-Based Collaborative Learning. Dina Goren-Bar & Tal Goori Department of Information Systems Engineering 23/6/2004. Agenda. Introduction & Motivation Research Objectives TEPCEL Framework TEPCEL Implementation TEPCEL Evaluation Current Status Experiment Results
E N D
Design and Evaluation of Web-Based Collaborative Learning Dina Goren-Bar & Tal Goori Department of Information Systems Engineering23/6/2004 Goren-Bar & Goori June 2004
Agenda • Introduction & Motivation • Research Objectives • TEPCEL Framework • TEPCEL Implementation • TEPCEL Evaluation • Current Status • Experiment Results • Conclusion • Further Research Goren-Bar & Goori June 2004
Introduction • What is Collaboration • What is Collaborative Learning? • What is ALN • Research Justification • Why should we perform a research in the field of Collaboration via ALN? • Shift from outcomes to interactions Goren-Bar & Goori June 2004
Research Objectives • Define a framework for the design of collaborative e-Learning tasks which combines the technological, educational and process oriented approaches. • Evaluation of the collaborative process generated during the implementation of a collaborative assignment as part of the HCI course. Goren-Bar & Goori June 2004
TEPCEL Framework • An acronym for Technological, Educational and Process oriented Collaborative E-Learning Framework • Enables the design and evaluation of synchronous as well as asynchronous collaborative learning environments and tasks. • Combines several approaches including the tools, outcome and process, document centric, and session centric approaches into one integrated framework Goren-Bar & Goori June 2004
TEPCEL Framework - 5 stages Objectives Definition Collaborative Features Settings Collaborative Tools Definition Design Stages Collaborative Assignment Definition Implementation & Evaluation Stages Evaluation • Each stage is characterized by a set of attributes that enable the design • & evaluation of the collaborative learning environment and tasks. Goren-Bar & Goori June 2004
TEPCEL Framework - Attributes Goren-Bar & Goori June 2004
TEPCEL Implementation • A web collaboration asynchronous environment. • An undergraduate course for Information Systems Engineering students. • The research was conducted in two students groups during years 2002-2003. Goren-Bar & Goori June 2004
TEPCEL Implementation • Each group received the same assignment structure with slightly different content. • Each group was further divided up into dyads. • Each subtask in each group was performed by two students (dyads). • The students were unable to control their collaborative group belonging. Goren-Bar & Goori June 2004
TEPCEL Implementation • Experimental Settings • Each asynchronous private group workspace (within the course website) was compounded of: • A list of all group members • A list of five sub-tasks • Collaborative Assignment description • A threaded asynchronous communication • Files Storage • Group email capabilities • An automatic email mechanism that informs the group members of a file upload operation performed by one of the dyads requesting for feedback. Goren-Bar & Goori June 2004
TEPCEL Implementation Goren-Bar & Goori June 2004
TEPCEL Evaluation - Tools • Feedback Questionnaire • To identify the level of collaboration within the groups and the successes of the collaboration process. • The questionnaire referred to the collaboration assignments and aimed to identify student 's self - preference towards collaboration Goren-Bar & Goori June 2004
Current Status • The 1st research results were used as a pilot study • The main conclusions from our Pilot Study were (Goren-Bar & Goori, 2004): • Feedback on uploaded files increased significantly from 2.6 to 6.5 (P<.05) • Students learned from experience, identified the main factors that influence successful collaboration and acted accordingly • Study on 2003 focused on the collaboration process (tested by a new feedback questionnaire) • Evaluation of the 2nd study in progress Goren-Bar & Goori June 2004
Feedback Questionnaire – Q1Grading Collaboration Methods • The students graded 5 collaborations methods (from 1 least preferred to 5 most preferred) to work with: Legend 1 – CL within ALN with 6-10 students 2 - F2F CL with 6-10 students 3 - Work alone 4 - F2F CL with 4 students at most5 – CL within ALN with max 4 students Anova: df=4, F=2.016388, P=0.0928 Goren-Bar & Goori June 2004
Number of selections Feedback Questionnaire – Q2Collaboration Disadvantages • Legend • Unequal workload division between group members • My influence on the final product is less than when I work alone • The coordination between group members requires too much effort • The noninvolvement of some group members damages the quality of the work • Group members who think different than me lead to undesired directions • Group collaboration does not promote individual's unique ideas and skills • Most of the students felt that the workload division between the group members was unequal (68%= 34/50) X2= 22.61427, df=4, P=0.000151 Goren-Bar & Goori June 2004
Feedback Questionnaire – Q3Collaboration Advantages Legend 1 - Group collaboration enables equal workload division between group members. 2 - The quality of deliverables created during group collaboration is better than those when working alone. 3 - Group collaboration enhances the variety of proposed ideas and solutions. 4 - Group collaboration reinforces the relations between group members. 5 - Discussing new ideas with other group members enables me to improve my knowledge. 6 - Group collaboration incites students' interest in the learning materials. • Group collaboration enhances brainstorming (3= the variety of proposed ideas and solutions 66% and 5= discussing new ideas & issues with other group members enables me to improve my knowledge 62%) • Interesting: option 1 which states that Group collaboration enables equal workload division between group members is an advantage and disadvantage! (question 2 – 34 students) X2= 34.344, df=4, P=0.000 Goren-Bar & Goori June 2004
Feedback Questionnaire – Q4&5 Legend 1 - "I am pleased with the task output I have delivered with my partner" 2 - "I am pleased with the final output my group delivered" 3 - "I have gained new knowledge as a result of the collaborative work" 4 - "The delivered presentation at the end of the assignment reflected the level of invested effort" 5 - "I would have gained better results had I performed the whole assignment by myself" * Probability associated with a Student's paired t-Test, with a one-tailed distribution Goren-Bar & Goori June 2004
Feedback Questionnaire – Q4&5 • The T - Test results indicate that in assignment 1 the students were more pleased with the task output they have delivered with their partner (P(t) = 0.0137). • Students were less pleased with the final output their group delivered (P(t) = 0.01) in assignment 2. • The implementation of assignment II was less successful due to bad timing (last weeks of the semester). Goren-Bar & Goori June 2004
Feedback Questionnaire – Q6&7 * Probability associated with a Student's paired t-Test, with a one-tailed distribution • In questions 6 & 7 students were asked to provide the number of group's members who did not collaborate or their level of involvement was low. • Only in group 5 in assignment 2 we can see a significant reduction in the number of uninvolved students (P(t) = 0.0477). • This stands in contrast to our hypothesis that the level of collaboration will improve from the first CL assignment to the second. Goren-Bar & Goori June 2004
Feedback Questionnaire – Q8 Goren-Bar & Goori June 2004
Feedback Questionnaire – Q8 • The level of collaboration improves when the group results are homogeneous (meaning we strive for a low percentage). • When the percentage is high it indicates that the group was not homogenous from the collaboration perspective, the effort was carried out by a small number of students within the group and most of the students were pretty much not involved in the process. • We have defined the threshold for measuring homogenous group as 30%. • Group 4 performed good collaboration (20%) and also groups 1 and 9 implemented good collaboration where most of the group members were involved in the process (30%, 33% respectively) Goren-Bar & Goori June 2004
Conclusions • Most students resented from working in big groups of 6-10 participants. • Most students would prefer to: • Work alone as this is the traditional working method. • Collaborate F2F or within the web medium but in smaller groups of 4 participants at most. • Most of the students felt that the workload division between the group members was unequal - “how can we balance the workload” This aspect will be handled in the framework • Almost all the students were pleased with the task output they have delivered with the partner - We assume that students were feeling uncomfortable with the thought of "complaining" about their best friend. Goren-Bar & Goori June 2004
Conclusions • In order to identify the level of collaboration based of students' credit we have defined three indicators. • "collaboration champions" presents the students who were 50% beyond the group average. • "non collaborative" which represented the number of students who were 50% under the group average • The Normalize Standard Deviation presents the variance of the feedbacks the students received based on the collaborative characteristics. I.e. a high value of N. STD states that the number of students who collaborated was low. A low value of N. STD states that the more students collaborated. Goren-Bar & Goori June 2004
Further Research • System Usability Scale (SUS) • Threaded asynchronous communication – Content Analysis based on: • Types of messages • Number of messages • Threaded asynchronous communication - Social Network Analysis • To analyze the collaboration interactions within the collaborative environment. • Participation distribution within group’s pairs • Distribution of initiator or replier within the type of messages and group members Goren-Bar & Goori June 2004