Pedagogic Evaluation

1. Pedagogic Evaluation Helen BeethamConsultant in Pedagogy JISC e-learning programme

2. Activities for this session • Discuss what is meant by ‘pedagogic evaluation’ • Identify project aims and rephrase as evaluation questions • Identify stakeholders in the evaluation • Consider appropriate means of data collection and analysis • Establish contact with ‘peer’ projects and begin sharing ideas/expertise

3. Lab testing • Does it work? • Functionality test • Compatibility test • Destruction test • …

4. Usability testing • Can other people make it work? • Are the menus clearly designed? • Is there a logical page structure? • ...

5. Pedagogic evaluation • Does anyone care if it works? • i.e. is it useful in learning and teaching contexts? • How is it useful? • Who needs software anyway? • …

6. Relating the three phases of evaluation • Software may progress from lab testing through usability testing to contextual evaluation… • … or (e.g. in RAP) through many iterative cycles, with users involved at each stage • Increasing authenticity of context • Evaluation moves from simple, lab-based to complex, authentic contexts of use • Different questions are asked, different kinds of data are collected, and different issues arise • Complex, authentic contexts rarely provide yes/no answers to development questions • Causative factors may be difficult to untangle • Findings may be highly context-related (so several different contexts are better for evaluation than one)

7. Three approaches to learning and teaching • There are basically three ways of understanding how people learn • Associative • Constructive • individual/cognitivist • social constructivist • Situative • Lead to different pedagogic strategies and approaches • Any of these approaches may be appropriate • depending on the priority outcomes and the needs of learners

8. Associative approach • In learning • Routines of organised activity • Progression through component concepts or skills • Clear goals and feedback • Individualised pathways matched to prior performance • In teaching • Analysis into component units • Progressive sequences of component-to-composite skills or concepts • Clear instructional approach for each unit • Highly focused objectives • In assessment • Accurate reproduction of knowledge or skill • Component performance • Clear criteria: rapid reliable feedback

9. Constructive approach(cognitivist) • In learning • Active construction and integration of concepts • Ill-structured problems • Opportunities for reflection • Ownership of the task • In teaching • Interactive environments and appropriate challenges • Encourage experimentation and the discovery of principles • Coach and model skills • Include meta-cognitive outcomes • In assessment • Conceptual understanding (applied knowledge and skills) • Extended performance • Processes as well as outcomes • Crediting varieties of excellence • Developing self-evaluation and autonomy in learning

10. Constructive approach(social) • In learning • Conceptual development through collaborative activity • Ill-structured problems • Opportunities for discussion and reflection • Shared ownership of the task • In teaching • Collaborative environments and appropriate challenges • Encourage experimentation, discussion and collaboration • Coach and model skills, including social skills • Learning outcomes may be collectively negotiated • In assessment • Conceptual understanding (applied knowledge and skills) • Extended performance • Process and participation as well as outcomes • Crediting varieties of excellence • Developing peer-evaluation and shared responsibility

11. Situative approach • In learning • Participation in social practices of enquiry and learning • Acquiring skills in contexts of use • Developing identity as a learner • Developing learning and professional relationships • In teaching • Creating safe environments for participation • Supporting development of identities • Facilitating learning dialogues and relationships • Elaborating authentic opportunities for learning • In assessment • Crediting participation • Extended performance, including variety of contexts • Authenticity of practice (values, beliefs, competencies) • Involving peers

12. 3 ways of learning about (educational) software • Does it work? • Routines of organised activity • Analysis into component units • Component performance • Highly focused objectives • Can other people make it work? • Ill-structured problems • Opportunities for reflection • Ownership of the task • Extended performance • Processes as well as outcomes • Is it useful in authentic (educational) contexts? • Creating supportive environments for use • Supporting development of users’ skills • Facilitating dialogues and relationships • Elaborating authentic opportunities • Extended performance, including variety of contexts • Authenticity of practice

13. Evaluation is learning! • Evaluation for development, not accountability • Sharing lessons (including failures) • Sharing concepts and approaches • Moving software into more authentic contexts of use • in order to find out how it is useful, and how it should be supported and embedded for effective use • Using the outcomes of evaluation to inform development and take-up • Learning across peer projects • Learning across different strands of the e-learning programme (and beyond) • Learning about your own software • how to have effective dialogues with users • range and/or specificity of application • usability implications?

14. Principles of evaluation • Ask the right questions • Involve the right people • Collect useful and reliable data • Analyse and draw conclusions appropriately

15. 1. Asking the right questions • How does the use of this e-tool support effective learning, teaching and assessment (LTA)? • What LTA activities? • Be specific and pragmatic • Understand how e-tool fits with existing LTA practice • But expect it to alter practice – sometimes unpredictably • Which users? • Range of user needs, roles and preferences • Consider stakeholders who are not direct users • What counts as ‘effective’? • Enhanced outcomes for learners? Enhanced experience of learning? • Enhanced experience for teachers/support staff? Greater organisational efficiency? • Consider what claims you made in your bid, your big vision • Effective in what LTA contexts? • Does e-tool support a particular pedagogicapproach? • Does it require a particular organisational context? • Consider pragmatics of interoperability, sustainability and re-use • Are you aiming for specificity of breadth of application?

16. Example: Interactive Logbook project • identify how the IL supports learners wrt ‘access, communication, planning & recording’ • How is access to learning resources improved? • How is communication for learning improved? • Does the IL provide a useful tool for planning and recording learning in the pilot programme? In what ways? • Does the IL support planning and recording outside of the pilot programme and does it provide a durable basis for future planning and recording? In what ways? • How does it compare with other systems offering similar benefits? • identify how the IL supports teachers or programme developers and organisations respectively, and understand how best ‘to implement and embed the Interactive Logbook… as distributed e-learning becomes more mainstream’. • What skills do learners and tutors need to make effective use of IL? What features of the programme support integration and use of IL (assessment strategies, support available, mode of access)? • What technical systems and support are needed for IL to be integrated effectively? What features of the organisation support effective use of IL by learners and teachers?

17. Over to you (1) • Evaluation should be interesting, so: • What do you really want to find out about your software? • What is the most important lesson your project could pass on to others? • Don’t set out to prove what we already know! • Look back at your project aims – what claims are you making for impact on LTA? • Translate these aims/claims into questions. Is there evidence of this impact? How does it happen? • Good claims are achievable but also challenging • Good questions are tractable but also interesting • Do your original aims fit with what interests you now? • Prioritise the issues that seem important now, with the benefit of insights from the development process • But use this as an opportunity to revisit and review • What are other projects investigating? • Do you have any questions in common in your peer group? Or questions that complement each other? • What could you usefully share of the evaluation process?

18. 2. Involving the right people • Who are your users? • What activities will they carry out with the system? • What functions of the system are important to them? • What roles do they play in those activities? • What are the important differences between them? • Significant for sampling (e.g. dyslexics for V-MAP) • walk-throughs, use cases (user testing and evaluation design) • real groups of learners and teachers (pedagogic evaluation) • Who are your other stakeholders? • Non-users whose work or learning may be impacted by use of the system in context e.g. administrators • Other ‘interested’ parties e.g. institutional managers, project funders, researchers/developers, potential users…

19. Over to you (2) • Who are your important stakeholder groups? • distinguish your user group in ways that are significant for your evaluation questions (e.g. learners with or without an existing e-portfolio) • consider non-users as stakeholders and as potential sources of information • Share your outcomes with your peer group. • What types of user do you need to include? (e.g. model users you have developed for walk-throughs) • How will you identify real user groups for evaluation? • How will you ensure all your significant types are included? • (NB including different types of learner is much easier than finding a ‘statistically representative’ sample: for this the proportions of different kinds of learner must be the same as in the target population)

20. 3. Collecting useful data • Data collected should be directly relevant to the questions! • Data should show triangulation: • using a variety of methods (e.g. focus group and questionnaire) • from a range of stakeholders (e.g. learners, teaching staff) • over a span of time (e.g. ‘before’ and ‘after’) • Quantitative data = How much? How often? How many? • Also providing yes/no answers to simple questions • Generalising from instances to rules • Converting opinions into data for analysis (Likert scales) • Qualitative data = explanatory, narrative • What happened? What was it like for you? Why? • Identifying themes and providing local evidence • Preserving the voices of participants • How authentic the context? If authentic, how will you support embedding and use of the software? • Feasibility and costs of data collection? • Skills and costs of analysis?

21. Over to you (3) • Use the matrix to plan what data you will collect • Data should be designed to answer specific questions (left column) and should be collected from specific stakeholder groups (top row) • Add details if possible, e.g. when (time) and how (method) this data could be collected • You need not fill all the boxes but try to have something in each row and column • You can merge boxes!

22. Final discussion: analysing and drawing conclusions • Basic choices for data analysis: • Quantitative analysis – statistical data that may be presented as pie charts, graphs etc; likert scales • Qualitative analysis within a given analytical framework –comparison, correlation, explanation • Qualitative analysis without a given analytical framework – case histories, narratives, themes • Outcomes need to be useful to different audiences • Your project, and other development projects • Implementers and users of your software • How will we draw conclusions across the different projects? • Peer review groups • Links to other projects in the e-learning programme • Pedagogy strand – refer to previous workshop • DeL regional pilots • ELF reference models (also standards community) • Sharing scenarios, roles and walk-throughs (model users)? • Mapping activities to a common framework (model uses)?

23. Learner Differences (‘model users’) • Refer to hand-out • But note that in many situations these differences will not be educationally significant • And in your context of use, there may be other important differences to consider (see your ‘stakeholder’ activity) • Could we develop a databank of ‘scenarios’ and roles’? • See Peter Rees-Jones’ work on scenarios for e-portfolios

24. Outline of a learning activity Identities: preferences, needs, motivations Competences: skills, knowledge, abilities Roles: approaches and modes of participating learner(s) specific interaction of learner(s) with other(s), using specific tools and resources, oriented towards specific outcomes learning activity learning outcome(s) learningenvironment New knowledge, skills and abilities on the part of the learner(s) Artefacts of the activity process Tools, resources, artefacts Affordances of the physical and virtual environment for learning other(s) Other people involved and the specific role they play in the interaction e.g. support, mediate, challenge, guide

25. reflect adapt do differentiate share respond Developmental processes Environment (tools, resources) can be adapted to meet the needs of learners, or provides a range of options for differentiation learner(s) Learning outcomes are captured for reflection, planning and review learning activity learning outcome(s) learningenvironment Outcomes can also be shared for formal assessment, informal feedback and peer review other(s)

26. Next steps • Appoint evaluators (if not already in place) • Finalise evaluation plan • Based on phase 2 bid • Using a pro-forma (optional) • Identify opportunities to liaise with other projects, e.g. to share • Evaluation questions and approaches • Actual data (comparative analysis?) • Process of analysis and drawing conclusions • I will be in touch to discuss these (or contact me at any time) helen.beetham@tiscali.co.uk