e-MapScholar: Evaluation

1. e-MapScholar: Evaluation Canan Blake and the Evaluation Team at The Open University e-MapScholar: evaluation framework

2. Evaluation Framework • CIAO • Context • Interaction • Attitudes and Outcomes e-MapScholar: evaluation framework

3. Context • wider - framing the teaching • finer - components of the resource “how learners use the information technology” “rationale” e-MapScholar: evaluation framework

4. (Tracking the )Interaction Look in detail at students’ use of the software by analysing their interactions • qualitative data about the learning process • difficulties experienced e-MapScholar: evaluation framework

5. Attitudes and Outcomes Investigate whether students have learnt the subject matter that the program is designed to teach But also consider effective outcomes, e.g. changes in learners perceptions and attitudes e-MapScholar: evaluation framework

6. Evaluation Activities • Previous Digimap Evaluations, Needs Assessment, Workshops • Formative • expert reviews • student observations • Summative • in universities with voluntary students • online questionnaires e-MapScholar: evaluation framework

7. Formative Evaluations • Method and participants • observation • educational technology experts • students • Findings • usability issues - the use of graphics and text • interactivity • assessment questions • ... e-MapScholar: evaluation framework

8. Summative Evaluation Aims to assess the effectiveness of the learning material as a teaching/learning resource and identify how they can be successfully integrated into a teaching programme • November 2002 • Staff and students in two institutions • Students accessed the resources and sent feedback via online questionnaires e-MapScholar: evaluation framework

9. What are the things you liked about this resource? “Good and relevant use of the diagrams, animated diagrams, clarity of diagrams, Mars example (more interesting), summary and the reflection” “The ability for the user to interact with the resource and the visual aisd throughout the session” e-MapScholar: evaluation framework

10. Things disliked “Inconsistent use of bold to highlight words” “some of the visual aids are sometimes thought as ‘interactive’ but they are not” “In the long view version, it need a period of time to download the diagrams or pictures” e-MapScholar: evaluation framework

11. CMS Evaluation • Cognitive walkthrough on the prototype • Representative tasks that would be carried out by potential users (defined from the prototype) • Will users see the relevant control(button menu) for the required action? • Once users find the control, will they recognise that it produces the effect that they want? • After the action is taken, will users understand the feedback that is provided so that they can then go on to complete the task? e-MapScholar: evaluation framework

12. Issues • What can be evaluated and when? • Development team involvement - distance • Practical constraints and timing • Pedagogical context • re-usable Learning Objects approach • customising to provide context relevant information and images • A framework that allows different approaches to be combined e-MapScholar: evaluation framework