Some findings from the evaluation of Collage authoring tool

1. Some findings from the evaluation of Collage authoring tool Davinia Hernández-Leo, Eloy D. Villasclaras,Juan I. Asensio-Pérez, Yannis Dimitriadis GSIC / EMIC group University of Valladolid, Spain TENCompetence Workshop Barcelona, June 22nd 2007 http://gsic.tel.uva.es

2. Our approach (I) • Challenges 1. Designing scripts that potentially elicits the desired interactions • How can teachers create “potentially effective” “computer-interpretable” collaboration scripts according to the necessities of their situation? • 2. Formalizing the scripts so that they are computer-interpretable • 3. Hiding computational representations of scripts to teachers

3. Reusing the “essence” of commonly used CSCL scripts (formulated as patterns) • CLFPs, Collaborative Learning Flow Patterns:Broadly accepted techniques repetitively used by practitioners when structuring the flow of types of (collaborative) learning activities(E.g. Jigsaw, Pyramid (Snowball) CLFPs) Our approach (II) 1. Not easy to design a script that potentially elicits the desired interactions

4. IMS Learning Design (LD) Our approach (IV) 2. Formalizing the scripts so that they are computer-interpretable

5. Integrate in authoring tools “a pattern-based design process for CSCL macro-scripts computationally represented with IMS LD” which allows teachers to create their own computer-interpretable script by reusing CLFPs as visual LD templates that can be assembled and refined Our approach (V) 3. Computational representations are not familiar to educators COLLAGE(COLaborative LeArning desiGn Editor) • (Based on Reload Editor. IMS-LD level A compliant)

6. Evaluation methodology (I) • Multicase study(Stake, 2005) • Authentic experiences with different audiences (contextual issues, achievement of educational benefits, impact of software systems) • Cross-case analysis • Not aiming at accurate generalizations but at valid conclusions • Mixed method combining qualitative and quantitative data gathering techniques(Martínez et al., 2003) • Triangulation: Comparative analysis of the data obtained with the different techniques

7. “Collage workshops” Case Study “Planet game” Case Study Creating CSCL scripts based on CLFPs using Collage “Network Managment” Case Study Solving a third-party scenario Putting into practice a CSCL script created with Collage Hands-onsessions Lab. uni. of Valladolidand Cádiz, Spain Pre-work Presenting Discussing ICALT conference Mini-cases Mixed method Two f2f anda distancesession Lab. Uni. OfValladolid Questionnaires Observations Focus groups Achieved CSCL script, papers, discussion ISSUE: Can we useCollage for creating a script representing a scenario proposed by a third-party? ISSUE: Does the design process implementedin Collage facilitate the reuse of CLFPs in the creation of particularized LD-represented CSCL scripts in a way that allows teachers to focus on the CSCL critical elements? Mixed method ISSUE: Can we use CSCL scripts created with Collage in real situations ? Evaluation methodology (II) • Multicase study(Stake, 2005) QUITAIN: The proposed pattern-based design process for CSCL macro-scripts computationally represented with IMS LD

8. Evaluation methodology (III) • Two main experiences with the target audience(teachers with interest in CL and CS) • University of Cádiz, Spain [UCA], University of Valladolid, Spain [UVA] • Creation of the script (for Network Management course) • Two smaller experiences (mini-cases): • CSCL practitioners [GSIC/EMIC], Educational technologist [UNFOLD] • Several different scripts • Data: Scripts, questionnaires, observations, discussion groups

9. “CTM2” script (applied in the “Network Management” case study) I want to design a collaboration script that guides the students in the collaborative understanding of a complex long technical paperthat can be divided into 3 different sections (3 versions of a network management protocol). I want the students to discuss and reach agreement on the main ideas of the paper… Teacher (Evaluation methodology) Sample creation of a script (I) • Optional undergraduate course on Network Management technologies

10. Checking educational benefits, types of problems, complexity • Reading information and examples (Evaluation methodology) Sample creation of a script (II) Selecting the CLFPs

11. (Evaluation methodology) Sample creation of a script (III) Authoring a CLFP-based LD • Combining the CLFPs

12. (Evaluation methodology) Sample creation of a script (IV) Authoring a CLFP-based LD • Refining the CLFPs

13. Four topics of interest • Pattern-based design process • Characteristics of the potential audience • Use of Collage • Focus on CSCL critical elements Evaluation methodology (IV) ISSUE: Does the design process implemented in Collage facilitate the reuse of CLFPs in the creation of particularized LD-represented CSCL scripts in a way that allows teachers to focus on the CSCL critical elements?

14. “… it saves the teacher a lot of specification workload [UVA-quest-final]” “I do not find the process too contrained since it enablesthe combination of patterns… and I think that it is easierfor a novice to have an already structured model [UVA-focus]” Support data Support data Support data Support data Support data Support data Support data All the GSIC/EMIC, 5 UNFOLD, 3 UVA and 10 UCA partici- pants agree on that the CLFPs are significant. 1 UVA and 3 UCA interviewees state that they are other relevant techniques but these are OK. The rest (2 UNFOLD, 1 UVA and 1 UCA participants) indicate that they do not know any CL technique. “… a minimum formation of patterns is necessary, for which Collage is helpful [UVA-focus]” The CLFPs are significant but there are more well-known strategies (e.g. for assessment) The combinations of patterns provides design flexibility Selection phase is critical and promotes the understanding of the patterns The interviewees’ ideas of the CLFPs coincide with what is presented in Collage “… I think that the patterns are perfectly transferredto the user workspace, reproducing the needed roles and activities for their execution [GSIC/EMIC]” “… it would be nice if other patterns could be added.However, as patterns can be combined, thesealready offer quite a lot of flexibility [UNFOLD]” “… no example is directly transferable…” “… the examples are useful especially if results are available… as in a community of practice…”[UVA-focus] “… no example is directly transferable…” “… the examples are useful especially if results are available… as in a community of practice…”[UVA-focus] “… I miss a pattern that I employ in my classes, which is randomly selecting a member of a groupto explain what the group has done [UVA-quest-final]” Pattern-based templates are probably more useful in the processof customizing a new situation than ready-to-run templates, but completeexamples are also useful Satisfactory trade-off between flexibility, keeping the essence captured in the CLFPs, hiding LD-specific technologicaldetails and providing a clear (but limited) set of design options Evaluation results • Pattern-based design process “Collage systematizes the selection of the patterns [UCA-quest-final]” • Is the selection of the CLFP-based LD templates and their representation useful and satisfactory? • Does the design process achieves a satisfactory trade-off between the reuse of the CLFPs and the creation of scripts contextualized according to the situational needs?

15. “… at last, patterns in practice!” “… less complicated and more intuitive than Reload on its own…”“makes Reload much easier to use for specific purposes” [UNFOLD] “… it may happen that advanced users cannot specifythings that they would like [UVA-quest-final]”“… the incorporation of activities that are not part of a pattern but that may be complementary [GSIC/EMIC]”“… if some students abandon the course in the thirdsession, then I need to re-structure [UVA-focus]” Support data Support data Support data “I am satisfactorily astonished by the user-friendliness of the tool; I thought it was going to be more difficult [UCA-quest-final]” “.. the participants create the designs without difficulties… [UVA-observer]” Most of the participants find Collage user-friendly and intuitive and are able to create partly completed examples during the workshops. Very few problems appear by they point out the importance of applying the scripts in real situations. Incorporating more CLFPs and other types of patterns (also non-pattern based elements), which would provide more flexible design options. (Level B and C of IMS-LD) “Preliminary view” of the script. The integration of Collage with instantiation and delivery systems should support flexibility. Collage is easier to use, specific to CL andit is the first IMS-LD editor providing pattern-based templates Evaluation results • Use of Collage • Can the teachers use successfully Collage? • How can Collagebe improved?

16. Interested in designing CL processes to be used with an LMS in f2f, distant or blended situations • teachers, novice or with experience in CL-some experience and interest in the use of ICT,but it is not necessary to be expert (IMS-LD) technologist “… it is going to be necessary to use this type of tools in our educational system. The EEES will also push to use these tools [UCA-quest-final].”“… for non-experts on CL this is really an eye-opener [UNFOLD]” Support data Evaluation results • Characteristicsof the potential audience [UCA-focus-group] • Which are the characteristics andmotivations of thepotential audienceof Collage?

17. Discussion (I) • A systematic multicase study • Focus on Collage workshops (2 experiences with target audience and 2 mini-cases), using a mixed evaluation method • Other two cases: • ICALT 2006 workshop (third-party scenario) • Experience with students in a real situation (Hernández-Leo et al., IEEE Transactions on Education, to be published in November 2007)

18. Discussion (II) • Positive results with respect to: • Reuse and contextualization • Focus on CSCL critical elements • Support of practitioners (teachers) • But also pointers for several enhancements and new research efforts

19. Thank you! http://gsic.tel.uva.es/collage Davinia Hernández-Leo davher@tel.uva.es

20. “… Since Collage guides the design process…, we are pushed to reflect on each element that shapes the activity [UVA-quest-final]” “the design implies the previous determination of allthese elements and their organization [UCA-quest-final]” Support data Support data Support data Support data “… it helps to determine the CLFP more suitable to foster particular objectives [UVA-quest-final]”“… it helps to envisage how the interactions willdevelp [UCA-quest-final]” “… it hightlights the interrelations among activities” “I think that the most important aspect of Collage is that ithelps to structure and systemtize the different types of activities… Collage has helped me to structure what I already do but I do not organize well [UCA-focus]” “… the structure of the (learning) process helps to check the roles that each group plays in each moment” [UCA-quest-final]“… Collage helps to conceptualize the groups [UVA-focus]“ The “selection phase” helps to determine the objectives related to CL that will be promoted and to select the task-type that will be solved by the students Many arguments support that the design process is very satisfactory concerning this element I helps to specify these elements, but it does Not provide suggestion of recommendedcontent or tools and the eventual structure within activities is textually-defined (or supported by tools) Evaluation results • Focus on CSCLcritical elements Does the design process: • help to determine the learning objectives, task-type and expected interaction that will be developed? • help to understand and determine the structure regarding the flow of activities and hierarchy of groups? • support also the definition of group-size, resource distribution, computer support and structure within activities?