Assoc. Prof. Dr. Eugenijus Kurilovas , Vilnius University , Lithuania

1. Evaluation of Quality of Learning Scenarios and Their Suitability to Particular Learners’ Profiles Assoc. Prof. Dr. Eugenijus Kurilovas, Vilnius University, Lithuania Inga Zilinskiene, Vilnius University, Lithuania Natalija Ignatova, Vilnius Pedagogical University, Lithuania ECEL 2011, Brighton

2. Outline • The aim of the paper • Literature analysis: the main notions and personalisation strategies • Research methodology • Learning scenarios quality criteria (model) • Method for evaluating quality of learning scenarios • Experimental evaluation results (iTEC project scenarios) • Conclusion and recommendations

3. The aim of the paper

4. To investigate, propose, and demonstrate examples of practical application of a model and methods suitable for the expert evaluation of quality of learning scenarios (LS). • A special attention is paid to LS suitability to particular learner groups (i.e., profiles), namely, to activist learners.

5. Literature analysis: the main notions and personalisation strategies

6. Learning Scenario’ (LS) / Unit of Learning – an aggregation of learning activities that take place in particular virtual learning environments (VLEs) using particular learning objects (LOs) • LO – any digital resource that can be reused to support learning • Learning activities (LAs) describe a role they have to undertake within a specified environment composed of LOs and services. Activities take place in a so-called “environment”, which is a structured collection of LOs, services, and sub-environments • VLE – a single piece of software, accessed via standard Web browser, which provides and integrated online learning environment

8. Evaluation – a process by which people make judgements about value and worth • Quality evaluation – a systematic examination of the extent to which an entity (part, product, service or organisation) is capable of meeting specified requirements • Expert evaluation – a multiple criteria evaluation of learning software aimed at the selection of the best alternative based on score-ranking results

9. Learning styles (Honey, P. & Mumford, A., 1982): • Activist: These people learn by doing. Have an open-minded approach to learning, involving themselves fully and without bias in new experiences. Their preferred activities are: brainstorming, problem solving, group discussion, puzzles, competitions, and role-play. • Reflector: These people learn by observing and thinking about what happened. • Pragmatist: These people need to be able to see how to put the learning into practice in the real world. • Theorist: These learners like to understand the theory behind the actions.

11. Research methodology

12. The main scientific problems: • A ‘proper’ set of LS quality criteria that should reflect the objective scientific principles of constructing a quality model (criteria tree) for LS suitable to particular learners’ groups (activist learners in our case) • ‘Proper’ methods to evaluate LS quality and suitability to particular learners’ profiles paying special attention to establishing proper weights of quality criteria by the novel method of consecutive triple application of AHP

13. MCEQLS (Multiple Criteria Evaluation of Quality of Learning Software) approach combines: • Model: • MCDA (Multiple Criteria Decision Analysis) principles for identification of quality criteria (Belton & Stewart, 2003) • Technological quality criteria classification principle (international standard ISO/IEC 9126-1:2001(E)) • Methods: • Fuzzy group decision making theory • Experts’ additive utility function using normalized weights of quality criteria

14. Learning scenarios quality criteria (model)

15. Principles of identification of quality criteria (relevant to all MCDA approaches): • Value relevance • Understandability • Measurability • Non-redundancy • Judgmental independence • Balancing completeness and conciseness • Operationality • Simplicity versus complexity

16. Quality criteria classification principle: there are two different groups / types of evaluation criteria – ‘internal quality’ and ‘quality in use’: • ‘Internal quality’ is a descriptive characteristic that describes the quality of learning software independently from any particular context of its use • ‘Quality in use’ is evaluative characteristic of software obtained by making a judgment based on criteria that determine the worthiness of software for a particular project or user / group (e.g., activist learners).

18. Method for evaluating quality of learning scenarios

19. , , X– LSalternative i = {1, … , m} – evaluation criteria (m=24) – rating (value) of criterion –‘normalised’ weights of criteria

21. Pair-wise comparison scale for AHP preferences: • 9 – Extremely preferred • 8 – Very strongly to extremely • 7 – Very strongly preferred • 6 – Strongly to very strongly • 5 – Strongly preferred • 4 – Moderately to strongly • 3 – Moderately preferred • 2 – Equally to moderately • 1 – Equally preferred

22. The proposed method consists of application of AHP in three consistent stages as follows: • Establishment of comparative weights of three different groups of LS quality criteria (i.e., LOs, LA and VLE) and weights aiof all quality criteria • Establishment of comparative weights of ‘internal quality’ and ‘quality in use’ criteria groups from activist learner point of view. The final ‘internal quality’ criteria weights are established in this stage • Establishment of final weights of quality criteria from activist learners point of view by application of AHP once again only for ‘quality in use’ criteria

26. Linguistic variables conversion into triangular non-fuzzy values: Excellent0.850 Good0.675 Fair0.500 Poor0.325 Bad0.150

27. Linguistic variables conversion into trapezoidal non-fuzzy values: Excellent1.000 Good0.800 Fair0.500 Poor0.200 Bad0.000

28. Experimental evaluation results (iTEC project scenarios)

29. Examples of LS in iTEC project: • LS1 “A Breath of Fresh Air”: http://itec.eun.org/web/guest/scenario-library • LS2 “Online Repositories Rock”: http://itec.eun.org/web/guest/scenario-library

30. Results of application of method of consecutive triple application of AHP: • Stage 1: experts evaluators prefer LOs and LA components more in comparison with VLE component (LOs – 39.7%, LA – 39.7%, and VLE – 20.6%). Weights ai for all 24 LS quality criteria were also calculated in conformity with AHP • Stage 2: the evaluators prefer ‘quality in use’ criteria in comparison with ‘internal quality’ criteria (respectively 69.4% Vs 30.6% for LOs, 72.2% Vs 27.8% for LA, and 61.1% Vs 39.8% for VLE) to analyse suitability of chosen LS to activist learners • Stage 3 has re-established the weights of ‘quality in use’ criteria

31. AHP stage 1 AHP stage 2 AHP stage 3 LOs LA VLE

32. AHP stage 1 AHP stage 3 AHP stage 2

34. In general case (G) when experts do not take into account particular learning style: • LS1: 72.7% according to trapezoidal method, or 63.8% according to triangle method Vs • LS2: 68.5% according to trapezoidal method, or 61.0% according to triangle method:

35. In particular case when experts take into account particular (activist) learning style (A): • LS1: 76.4% according to trapezoidal method, or 65.6% according to triangle method Vs • LS2: 69.7% according to trapezoidal method, or 61.8% according to triangle method:

36. Conclusion and recommendations

37. MCEQLS approach refined by the original method of consecutive triple application of AHP to establish criteria weights: • is applicable in real life situations when schools have to decide on use of particular LS for their education needs, and • could significantly improve the quality of expert evaluation of LS by noticeably reduce of the expert evaluation subjectivity level • Use of the method of consecutive triple application of AHP leads to establishing different weights of criteria for particular leaner groups, and respectively – to different LS alternatives evaluation results

38. Application of both triangle and trapezoidal fuzzy numbers shows similar alternatives’ quality evaluation results, i.e., the ranking of analysed alternatives have not changed while applying different fuzzy numbers methods • Proposed MCEQLS AHP method is quite objective, exact and simply to use for selecting qualitative LS alternatives for particular learner groups • Proposed LS personalised quality evaluation approach is applicable for the aims of iTEC project in order to select LS suitable for activist learners

39. Thank you for your attention !!! Questions ? Dr. Eugenijus Kurilovas Eugenijus.Kurilovas@itc.smm.lt Inga Zilinskiene Inga.Zilinskiene@gmail.com Natalija Ignatova Natalija.Ignatova@itc.smm.lt