1 / 41

TELOS TESTBED – 2006 Evaluating Learning Design Tools

TELOS TESTBED – 2006 Evaluating Learning Design Tools. Karin Lundgren-Cayrol Isabelle Savard Marcello Maina Claire Banville En collaboration avec: Theme 6.3 France Henri & Josianne Basque. Outline. 2006 - Testbed Context, Goals and Objectives

edda
Download Presentation

TELOS TESTBED – 2006 Evaluating Learning Design Tools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TELOS TESTBED – 2006Evaluating Learning Design Tools Karin Lundgren-Cayrol Isabelle Savard Marcello Maina Claire Banville En collaboration avec: Theme 6.3 France Henri & Josianne Basque

  2. Outline • 2006 - Testbed Context, Goals and Objectives • General Evaluation Plan & Formative Evaluation Design • Participants • Hands-on Evaluation Scenario • Results • Background questionnaire • Testbed Evaluation Scenario • Final Questionnaire • Preliminary conclusions • Connected Student Thesis Work • Isabelle Savard (Poster78 : Cultural Diversity and Reuse of Learning Objects) • Marcelo Maina (Adapting MISA to IMS Learning Design) • Questions

  3. Goal • To identify strengths and weaknesses of the TELOS Instructional Design Interface potentialTools

  4. Objectives 2006 • To test tools that are scheduled to be incorporated into the TELOS instructional design Interface. This year: • MOT+LD editor • Canadian LD Repository • Learning Design Methodology • To elicit information about practitioners ways and needs in terms of facilitating their instructional design tasks. • To send results to concerned teams about their tools in regards to their integration into TELOS. • Team 6.3 – ID’s opinions, tasks and perceived competency profile • Team 2.1 and 6.2 – Revisions of processes, tools and guides

  5. demands demands demands Identify needs Create testbed scenario Create testbed protocol Carry out testbed Analyse results Protocol & Testbed Cycle Revise TELOS Revise LKMS Revise LKMA Revise LKMA Core LKMS LKMA LKMP Disseminate

  6. Context 2006 • TELOS consists of four different interfaces and toolsets according to actor • Engineer (TELOS Kernel) • Technician (LKMS) • Instructional Designer (LKMA) • Learner and Teacher Interfaces (LKMP)

  7. Call for Participants • Requirements: • Experience in designing online learning / e-learning • Good computer knowledge (Windows) • Experience with web-based technology

  8. Testbed Agenda & Instruments • Complete Background Information concerning (15 min) • Work Context • Technical Profile and Tool Usage • ID Experience and Practice • Carry out the Evaluation Scenario • Demo Session (1h30) • Hands-on (5 h) • Complete a Final Questionnaire (30min) • Participate in a Focus Group (1h)

  9. Let’s visit the IDLD siteAnd carry out this EVALUATION Scenariohttp://www.idld.org

  10. Some Preliminary Results

  11. Completion Rates • Background questionnaire (N=11) • 9 Mauritians (Validation of instrument) • 11 Canadians • Demo session (N=9) • 1 in LORIT; 6 = ClickToMeet; 2 by LORIT Webcast & chat; 1 used • 2 abandon • Evaluation Scenario (N = 6 - 1) • 2 abandon; 1 finished but did not return it; • 3 about half; 1 all but adapting UoL; 1 all the tasks • Final Questionnaire (N = 6) • 4 most questions • 2 less than half • Focus Group = cancelled

  12. Background Information - Context • Mauritius (N=9) • F2F interviews allowed validation of BQ • All worked at the University (7 ID; 2 tutors) • 3 designed in English, 2 in French, and 4 in Both languages • 5 had LO repository access • Canada (N=11) • Geographically distributed (contact by email and phone) • 8 University, 2 College and 1 Government • 4 Prof/Teach; 5 ID; 1 Tutor ; 1 Training Manager • 7 designed in English, 1 in French, & 3 Both languages • 2 LO access; 2 under development; 6 NO &1 didn’t know

  13. Background Information – Context (1)

  14. Context (2) Tendency:Instructional designers work in any discipline, teachers referred to their domain expertise.

  15. Background Information – Canadians Tool Usage • 10/11 mostly used Windows • 6/11 used IE, 4 Mozilla & 1 Netscape

  16. Preferred Help Method Conclusion: Telos must provide a plethora of advisory systems as well as having human resources available through the use of sophisticated communication tools.

  17. Tool Usage (1)

  18. Tool Usage (2)

  19. Tool Usage (3) LO and Repository Conclusion: Most LO, IMS LD related tools are not widely known.

  20. Background Information – ID Experience and Practice (1) • Canada • 5/11 had more than 6 years • 3/11 had between 3 and 6 years • 3/11 less than 2 years • Mauritius • 1/9 had more than 6 years • 7/9 had between 3 and 6 years • 1/9 less than 2 years 16/20 more than 3 years Conclusion: TELOS must be able to provide flexible instructional design environments.

  21. Reusing LO’s Conclusion: LO’s are used in different ways and are starting to be a well known concepts.

  22. Validation process and results Validation process in Mauritius (Savard, 2006) • First version of the background questionnaire • Semi-structured interviews to find out their regular ID process and tasks • Similar to Gagné’s ADDIE Instructional Design Model. Results: • Comment on each question which led to • changes to Que’s 5, 10 & 21 • addition Que’s 17, 19 & 22 (ADDIE)

  23. ADDIE - Instructional Design Task • Analysis • Design • Development/Production • Implementation • Evaluation

  24. ADDIE - Instructional Design Task • Analysis Conclusions • In general, the Analysis task is already done when a demand comes to Mauritians Instructional Designer’s. • They often have to reformulate objectives. • For the Canadians, the Analysis is an important task.

  25. ADDIE - Instructional Design Task • Design Conclusions: • Instructional Structure is an important task for both ID from Mauritius and Canada.

  26. ADDIE - Instructional Design Task • Development/Production Conclusions: • Material production is an important task in both Mauritius and Canada. • In Mauritius the delivery and maintenance plan seem not to be their responsibility.

  27. ADDIE - Instructional Design Task • Implementation • Conclusions: • Small field testing doesnt take too much time for ID from both contries.

  28. ADDIE - Instructional Design Task • Evaluation Conclusions • The course evaluation is not a major concern in Mauritius. • 70% of Canadians do carry out this task

  29. ID Competencies (ibstpi) • Four groups of Competency statements for instructional designers: • A- Professional Foundations • B- Planning and Analysis • C- Design and Development • D- Implementation and Management • Validate which competencies were the most important in their job situation. http://www.ibstpi.org/

  30. ID Competencies (ibstpi)Results

  31. ID Competencies (ibstpi) - Results

  32. ID Competencies (ibstpi)Results

  33. ID Competencies (ibstpi)Results

  34. Evaluation Scenario

  35. Comments and Suggestions • Experience too time consuming because of all the documents to be studied • Several separated Demo Sessions are necessary • IMS LD very complex • MOTPlus has significant advantages • Graphical modeling is interesting • All tools were very interesting

  36. Final Questionnaire

  37. IDLD PORTAL Conclusion: Very divided opinions, we need more evaluators!!

  38. MOT+LD and Graphical Modeling Conclusion: At least useful and functional for those who tried

  39. PALOMA Resource Manager Conclusions: Divided opinions and Needs more evaluators!! • Suggestions: • Need a more user-friendly version • We need more reliable federated search protocols • Advanced search is a bit confused

  40. Some Preliminary Conclusions • The ADDIE Model and ibstpi competency • Generally practiced by ID’s (few differences) • Questions whether Planning and Evaluation is an ID task? • TELOS must be able to provide • Sophisticated communication tools to simulate human resource contact • flexible instructional design environments • Confirms complexity of the IMS LD concepts • Need to adjust vocabulary • Provide transparent & flexible tools • Provide more examples in the Canadian LD Respository

More Related