1 / 24

Designing Web-Based Collaborative Learning: TEPCEL Framework Evaluation

Explore TEPCEL Framework for designing collaborative e-learning tasks, combining technological, educational, and process-oriented approaches. Evaluate collaborative processes in HCI course assignments. Experiment results and further research discussed.

kylev
Download Presentation

Designing Web-Based Collaborative Learning: TEPCEL Framework Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Design and Evaluation of Web-Based Collaborative Learning Dina Goren-Bar & Tal Goori Department of Information Systems Engineering23/6/2004 Goren-Bar & Goori June 2004

  2. Agenda • Introduction & Motivation • Research Objectives • TEPCEL Framework • TEPCEL Implementation • TEPCEL Evaluation • Current Status • Experiment Results • Conclusion • Further Research Goren-Bar & Goori June 2004

  3. Introduction • What is Collaboration • What is Collaborative Learning? • What is ALN • Research Justification • Why should we perform a research in the field of Collaboration via ALN? • Shift from outcomes to interactions Goren-Bar & Goori June 2004

  4. Research Objectives • Define a framework for the design of collaborative e-Learning tasks which combines the technological, educational and process oriented approaches. • Evaluation of the collaborative process generated during the implementation of a collaborative assignment as part of the HCI course. Goren-Bar & Goori June 2004

  5. TEPCEL Framework • An acronym for Technological, Educational and Process oriented Collaborative E-Learning Framework • Enables the design and evaluation of synchronous as well as asynchronous collaborative learning environments and tasks. • Combines several approaches including the tools, outcome and process, document centric, and session centric approaches into one integrated framework Goren-Bar & Goori June 2004

  6. TEPCEL Framework - 5 stages Objectives Definition Collaborative Features Settings Collaborative Tools Definition Design Stages Collaborative Assignment Definition Implementation & Evaluation Stages Evaluation • Each stage is characterized by a set of attributes that enable the design • & evaluation of the collaborative learning environment and tasks. Goren-Bar & Goori June 2004

  7. TEPCEL Framework - Attributes Goren-Bar & Goori June 2004

  8. TEPCEL Implementation • A web collaboration asynchronous environment. • An undergraduate course for Information Systems Engineering students. • The research was conducted in two students groups during years 2002-2003. Goren-Bar & Goori June 2004

  9. TEPCEL Implementation • Each group received the same assignment structure with slightly different content. • Each group was further divided up into dyads. • Each subtask in each group was performed by two students (dyads). • The students were unable to control their collaborative group belonging. Goren-Bar & Goori June 2004

  10. TEPCEL Implementation • Experimental Settings • Each asynchronous private group workspace (within the course website) was compounded of: • A list of all group members • A list of five sub-tasks • Collaborative Assignment description • A threaded asynchronous communication • Files Storage • Group email capabilities • An automatic email mechanism that informs the group members of a file upload operation performed by one of the dyads requesting for feedback. Goren-Bar & Goori June 2004

  11. TEPCEL Implementation Goren-Bar & Goori June 2004

  12. TEPCEL Evaluation - Tools • Feedback Questionnaire • To identify the level of collaboration within the groups and the successes of the collaboration process. • The questionnaire referred to the collaboration assignments and aimed to identify student 's self - preference towards collaboration Goren-Bar & Goori June 2004

  13. Current Status • The 1st research results were used as a pilot study • The main conclusions from our Pilot Study were (Goren-Bar & Goori, 2004): • Feedback on uploaded files increased significantly from 2.6 to 6.5 (P<.05) • Students learned from experience, identified the main factors that influence successful collaboration and acted accordingly • Study on 2003 focused on the collaboration process (tested by a new feedback questionnaire) • Evaluation of the 2nd study in progress Goren-Bar & Goori June 2004

  14. Feedback Questionnaire – Q1Grading Collaboration Methods • The students graded 5 collaborations methods (from 1 least preferred to 5 most preferred) to work with: Legend 1 – CL within ALN with 6-10 students 2 - F2F CL with 6-10 students 3 - Work alone 4 - F2F CL with 4 students at most5 – CL within ALN with max 4 students Anova: df=4, F=2.016388, P=0.0928 Goren-Bar & Goori June 2004

  15. Number of selections Feedback Questionnaire – Q2Collaboration Disadvantages • Legend • Unequal workload division between group members • My influence on the final product is less than when I work alone • The coordination between group members requires too much effort • The noninvolvement of some group members damages the quality of the work • Group members who think different than me lead to undesired directions • Group collaboration does not promote individual's unique ideas and skills • Most of the students felt that the workload division between the group members was unequal (68%= 34/50) X2= 22.61427, df=4, P=0.000151 Goren-Bar & Goori June 2004

  16. Feedback Questionnaire – Q3Collaboration Advantages Legend 1 - Group collaboration enables equal workload division between group members. 2 - The quality of deliverables created during group collaboration is better than those when working alone. 3 - Group collaboration enhances the variety of proposed ideas and solutions. 4 - Group collaboration reinforces the relations between group members. 5 - Discussing new ideas with other group members enables me to improve my knowledge. 6 - Group collaboration incites students' interest in the learning materials. • Group collaboration enhances brainstorming (3= the variety of proposed ideas and solutions 66% and 5= discussing new ideas & issues with other group members enables me to improve my knowledge 62%) • Interesting: option 1 which states that Group collaboration enables equal workload division between group members is an advantage and disadvantage! (question 2 – 34 students) X2= 34.344, df=4, P=0.000 Goren-Bar & Goori June 2004

  17. Feedback Questionnaire – Q4&5 Legend 1 - "I am pleased with the task output I have delivered with my partner" 2 - "I am pleased with the final output my group delivered" 3 - "I have gained new knowledge as a result of the collaborative work" 4 - "The delivered presentation at the end of the assignment reflected the level of invested effort" 5 - "I would have gained better results had I performed the whole assignment by myself" * Probability associated with a Student's paired t-Test, with a one-tailed distribution Goren-Bar & Goori June 2004

  18. Feedback Questionnaire – Q4&5 • The T - Test results indicate that in assignment 1 the students were more pleased with the task output they have delivered with their partner (P(t) = 0.0137). • Students were less pleased with the final output their group delivered (P(t) = 0.01) in assignment 2. • The implementation of assignment II was less successful due to bad timing (last weeks of the semester). Goren-Bar & Goori June 2004

  19. Feedback Questionnaire – Q6&7 * Probability associated with a Student's paired t-Test, with a one-tailed distribution • In questions 6 & 7 students were asked to provide the number of group's members who did not collaborate or their level of involvement was low. • Only in group 5 in assignment 2 we can see a significant reduction in the number of uninvolved students (P(t) = 0.0477). • This stands in contrast to our hypothesis that the level of collaboration will improve from the first CL assignment to the second. Goren-Bar & Goori June 2004

  20. Feedback Questionnaire – Q8 Goren-Bar & Goori June 2004

  21. Feedback Questionnaire – Q8 • The level of collaboration improves when the group results are homogeneous (meaning we strive for a low percentage). • When the percentage is high it indicates that the group was not homogenous from the collaboration perspective, the effort was carried out by a small number of students within the group and most of the students were pretty much not involved in the process. • We have defined the threshold for measuring homogenous group as 30%. • Group 4 performed good collaboration (20%) and also groups 1 and 9 implemented good collaboration where most of the group members were involved in the process (30%, 33% respectively) Goren-Bar & Goori June 2004

  22. Conclusions • Most students resented from working in big groups of 6-10 participants. • Most students would prefer to: • Work alone as this is the traditional working method. • Collaborate F2F or within the web medium but in smaller groups of 4 participants at most. • Most of the students felt that the workload division between the group members was unequal - “how can we balance the workload” This aspect will be handled in the framework • Almost all the students were pleased with the task output they have delivered with the partner - We assume that students were feeling uncomfortable with the thought of "complaining" about their best friend. Goren-Bar & Goori June 2004

  23. Conclusions • In order to identify the level of collaboration based of students' credit we have defined three indicators. • "collaboration champions" presents the students who were 50% beyond the group average. • "non collaborative" which represented the number of students who were 50% under the group average • The Normalize Standard Deviation presents the variance of the feedbacks the students received based on the collaborative characteristics. I.e. a high value of N. STD states that the number of students who collaborated was low. A low value of N. STD states that the more students collaborated. Goren-Bar & Goori June 2004

  24. Further Research • System Usability Scale (SUS) • Threaded asynchronous communication – Content Analysis based on: • Types of messages • Number of messages • Threaded asynchronous communication - Social Network Analysis • To analyze the collaboration interactions within the collaborative environment. • Participation distribution within group’s pairs • Distribution of initiator or replier within the type of messages and group members Goren-Bar & Goori June 2004

More Related