Immersive Reflective Experience-based Adapive Learning

1. 13th International Conference on Knowledge Management and Knowledge Technologies (i-KNOW)4-6 September 2013, Graz, AustriaEU-Day Synergy Workshop Immersive Reflective Experience-based Adapive Learning Evaluation in ImREAL: Lessons Learned from a Project on Augmenting Experiential Training Simulators Christina M. Steiner, Gudrun Wesiak, & Dietrich Albert Cognitive Science Section, Knowledge Management Institute,Graz University of Technology

2. Overview • Intro • Evaluation in ImREAL • Challenges and Solutions • Special Types of Evaluation • Wrap Up

3. Intro • Evaluation is a natural part of TEL projects • Defining evaluation mechanisms and metrics at proposal stage • Interests of scientific and industrial community • Evaluation goals: • Demonstrate added value of the developed learning technologies • Collecting information for advancing technologies • Testing and evaluating in a reproducible way • Allowing comparison of research results

4. Simulated Environmentfor Learning Evaluation in ImREAL Affective Meta-cognitive Scaffolding Pedagogy – Use Cases Evaluation – User Trials Augmented user modelling Making Sense of Digital Traces Integration Framework

5. Evaluation in ImREAL Overall Goal: Designing and carrying out methodologically sound evaluations of the project outcomes and the project itself Main Objectives: • Definition of a scientifically sound evaluation methodology • User trial evaluations

6. Evaluation in ImREAL Evaluation phases Year 1 Year 2 Year 3 Phase 2 Phase 1 Project phases First version of services and demonstrators Second version of services and demonstrators Final versions of services and demonstrators collecting benchmark data, informing design Formative Evaluation:initial and potential benefits, informing next development cycle Formative and Summative Evaluation:quality of second version of ImREAL technology, informing final refinements in development Development phases Baseline Evaluation Phase 1 Evaluation Phase 2 Evaluation Evaluation phases

7. Evaluation in ImREAL • Simulator Maturation Stages • Simulator creation • Creating a new simulation learning scenario • Simulator extension • Extending an existing simulationtosupportlearning • Simulator integration • Integrating a simulator in a training environment

8. Evaluation in ImREAL • Test beds • Two usecasesandsimulatorswithdifferent focus • Medical interview training • Maturesimulation • Integrated in concretetrainingcontext • Focus on simulatorextension(Phase 1),broadeningtosimulationcreation (Phase 2) • Business simulationandbuddyprogram • Creationofnewsimulationscenarios • Focus on simulatorcreationin Phase 1,on simulatorextension in Phase 2

9. Evaluation in ImREAL • Evaluation dimensions • Aligningevaluationmethodologywith • Goals ofevaluationphase • Stakeholdersofmaturationstage • Research focusofusecase integration Simulator Maturation Process extension creation use case A Use Case use case B phase 1 eval. phase 2 eval. baseline eval. Evaluation Phase

10. Evaluation in ImREAL • Evaluation dimensions • Aligningevaluationmethodologywith • Goals ofevaluationphase • Stakeholdersofmaturationstage • Research focusofusecase integration Simulator Maturation Process extension creation use case A Use Case use case B phase 1 eval. phase 2 eval. baseline eval. Evaluation Phase

11. Challenges & Solutions • Evaluation of ImREAL services, not the simulators • Formulation of evaluation question - analyses of expected benefits of services • Combination of evaluation studies • Specifically targeted on isolated services • Integrated in test beds and simulators • Incorporation of baseline evaluations • Sound investigation of service benefits

12. Challenges & Solutions • Evaluation methodology vs. evaluation practice • Evaluation plans before development is finished • Feasibility • Timing • Involvement of target and external stakeholders • Small numbers of experts • Academia vs. industry

13. Challenges & Solutions • Evaluation of user model augmentation • Users’ acceptance of exploiting their social digital traces • Evaluation of learning effects • Difficulty to demonstrate knowledge gain • Considering learning effects and experience in a more comprehensive way • Study learning activity rather than purely subjective reports • Analysing learners’ real activities with the simulators

14. Special Types of Evaluation • Cost-benefit analysis • Costs of integration of ImREAL services • How can these costs be balanced by the benefit/added-value • ‘Perspective of the market’ • Internal project evaluation • Formativereviewofresearchquestionsand formative evaluationoftheproject • Evaluation approach: • Interdisciplinarydialogue • Questionnaire

15. Wrap-up • Evaluation should be inherent part of iterative design process • Evaluation of sub-components, as well asintegration in real-world settings • Mind the gap: Realistic evaluation planning • Mixed-method approach combining objectivedata with subjective self-reports • Evaluation as dissemination activity • Cost-effectiveness and readiness for the market • Evaluation as instrument to stimulate reflection on the project

17. Highlights • Evaluation as inherent part of iterative design process • Mind the gap: Realistic evaluation planning • Mixed-method approach combining objective data with subjective self-reports

18. D7.5 Evaluation Methodology