Higher learning skills, good feedback

1. Higher learning skills, good feedback EASiHE Workshop 10th February 2010

2. Introduction • Welcome • Introductions • Overview of the day

3. Competency Modelling Supporting Higher-Level Outcomes and Better Feedback Lester Gilbert Learning Societies Lab University of Southampton

4. Competence

5. Discriminate learning & teaching

6. Teaching

7. Learning

8. Technology support

9. The big picture

10. Competence hierarchy

11. Competency tree

12. Example nursing competencies Sitthisak, Gilbert, & Davis (2007)

13. Example competency 3D tree Sitthisak, Gilbert, & Davis (2007)

14. Questions, comments, discussion… Gilbert, L., & Gale, V. (2008).Principles of E-learning Systems Engineering.Chandos Publishing. ISBN 1-84334-290-1

15. HLOs (Higher-order Learning Outcomes) Bill Warburton EASiHE/iSolutions

16. A formative taxonomy Increasing level of abstraction • For FORMATIVE evaluation, Bloom et al. (1971) distinguished a hierarchy of levels of behaviour parallel to the cognitive taxonomy • Added value to the cognitive taxonomy by making it easier for teachers to identify ‘mal-rules’ – flaws in reasoning ABILITY TO MAKE APPLICATIONS ABILITY TO MAKE TRANSLATIONS SKILL IN USING PROCESSES & PROCEDURES KNOWLEDGE OF RULES & PRINCIPLES KNOWLEDGE OF FACTS KNOWLEDGE OF TERMS

17. Testing KNOWLEDGE OF FACTS(Bloom 1956: KNOWLEDGE RECALL) • Memorise something • Recognise something • Recall of information or identity: • Who, what, when, where, how ...? • Describe… • (School of Oceanography and Earth Science, University of Southampton)

18. KNOWLEDGE OF RULES & PRINCIPLES • Memorise and recall a general rule • What, when, where, how ...? • Describe… • Describe interrelationships among many items • Memorise and recall applications of a rule • Memorise and recall exceptions to a rule • Does *not* deal with Application of a rule

19. Testing KNOWLEDGE OF RULES & PRINCIPLES(Bloom 1956: COMPREHENSION) Example: In order to write a chemical formula, you have to know a. only the symbols of the elements that are in the compound b. only the proportions in which the atoms of elements combine c. both the symbols of the elements that are in the compound and the proportions in which the atoms of elements combine d. the atomic weight of the elements that form the compound (Bloom et al. 1971)

20. Skill in using Processes & Procedures Steps along the route to Mastery The difference between ‘Knowing That’ and ‘Knowhow’? Accuracy in use Practice makes perfect Expression of (justified) confidence Not formally represented in Bloom’s Taxonomy of the Cognitive domain but important for assessing Mastery Formative drills

21. Skill in using Processes & Procedures Example of process-based item (JISC/REAQ Report,2009)

22. ABILITY TO MAKE TRANSLATIONS Put idea in own words or use new examples of what is learned Transform a term, fact, rule, principle, process or procedure from one form to another Take a phenomenon presented in one mode/form and represent it by an equivalent form/mode Move from a verbal to a symbolic form Determine when a new illustration is appropriate or not Move from a concrete to a more abstract form, or from a general to a more specific illustration, and vice versa.

23. Testing ABILITY TO MAKE TRANSLATIONS (Bloom 1956: COMPREHENSION) Example: testing a participant’s ability to • Interpret (native Spanish speaker) or • Translate (second language speaker) parts of speech from English to Spanish (School of Humanities, University of Southampton)

24. ABILITY TO MAKE APPLICATIONS Recognize the essentials of the problem Use of a rule/principle learned in one context to solve a problem presented in a new or unfamiliar context Identify rules/principles/generalisations relevant to a problem Use ideas to solve a problem which is different from those previously encountered in the instruction or instructional materials. Most complex of the Formative categories - depends on other classifications but requires application of ideas in new situations or problems

25. Testing ABILITY TO MAKE APPLICATIONS (Bloom 1956: APPLICATION) Example: testing a participant’s ability to (i) solve problems (in this case, using Wien’s Law) … (ii) use facts, rules and principles (School of Physics and Astronomy, University of Southampton)

26. Testing ANALYTICAL skills 26 • Subdivide something to show how it is put together • Find the underlying structure of a communication • Identify motives • Separate something into component parts: • What are the parts or features of...? • Classify...according to... • Outline/diagram... • How does...compare/contrast with...? • What evidence can you list for...?

27. Testing ANALYTICAL skills This example tests a participant’s ability classify data according tospecific criteria: Q. Which countries' statistics are being reported in A, B and C? 1. A=South Korea; B=Kenya; C=Canada 2. A=Sri Lanka; B=Germany; C=Thailand 3. A=Sri Lanka; B=Thailand; C=Sweden* 4. A=Namibia; B=Portugal; C=Botswana (CASTLE project, University of Leicester)

28. Testing SYNTHETIC skills 28 • Create original products in verbal or physical form • Combine ideas to form a new whole: • What would you predict/infer from...? • What ideas can you add to...? • How would you create/design a new...? • What might happen if you combined...? • What solutions would you suggest for...?

29. Testing SYNTHETIC skills Example: testing a participant’s ability to predict/infer the 3D appearance of a 2D net. Requires the abstract abilities to accurately reconstruct solids, rotate them about three axes and combine the results with a predicted model. Q. The picture shows a cube that I have made. Which one of the shapes below, if cut out and folded, could make a cube the same as mine? (Thinking Skills Admission Tests, University of Cambridge)

30. Testing EVALUATION skills • Make value decisions about issues • Resolve controversies or differences of opinion • Develop opinions, judgements or decisions: • Do you agree that...? • What do you think about...? • Place the following in order of priority... • What criteria would you use to assess...? • What are the most important aspects of...?

31. Testing Evaluation skills Example: testing a participant’s ability to evaluate the link between cause and effect in terms of predefined criteria "The United States took part in the Gulf War against Iraq BECAUSE of the lack of civil liberties imposed on the Kurds by Saddam Hussein's regime." • The assertion and the reason are both correct, and the reason is valid • The assertion and the reason are both correct, but the reason is invalid* • The assertion is correct but the reason is incorrect • The assertion is incorrect but the reason is correct • Both the assertion and the reason are incorrect. (CASTLE toolkit, University of Leicester)

32. Mastering the art of item authoring Three principle challenges for new authors: • How to reducing the scope for guessing • How to testing ILOs at higher levels of abstraction • How to varying the difficulty of items

33. Effective questions and feedback Veronica Gale EASiHE/Independent Consultant

34. HLO questions • Write questions and feedback after finalising the ILO • Do students have enough information to be able to answer the question correctly? • Is the question focused on one main learning point? • If there is a score, is it clear to students how the scoring works? • If there is a ‘correct answer’ is this the only one that can be considered correct? • Does the question use terminology that is suitable for this stage in the course? • Have you piloted the assessment?

35. Example question from Bloom FACT: The number of shareholders in most large corporations has increased considerably during the last 30 years. CONCLUSION: Control of corporations has become more democratic in the last 30 years. A: The fact is good evidence to support the conclusion B: The fact is good evidence to disprove the conclusion C: Neither A nor B applies clearly To help them learn, what do the students who select A need in their feedback?

36. Effective feedback • Whether the response is correct or not • If the student is incorrect: an explanation of why, what the correct answer is and why this is correct • If the student is correct: confirmation of why this is correct • Recommended action e.g., reference back to the learning materials, remedial tuition, further questions for practice All to be: • Presented immediately • With the question, their response and the feedback visible at the same time

37. When there are no ‘right’ and ‘wrong’ answers • Scoring may be inappropriate • Discuss the merits and drawbacks of the different options • Give an expert’s opinion on the best choice, with full explanation • Consider a model answer, range of sample answers or demonstration • Ask a follow-up question on the rationale for the response

38. Visual feedback

39. Activities Group discussions

40. Activity One • Review the questions provided and identify: • what works well and • what needs to be improved

41. Activity Two • Drawing on your own experiences and thinking about the discussions in the workshop so far, create a poster listing: • what to do, and • what to avoid • when writing e-assessments and feedback for higher learning outcomes. • Your readership are academics creating new formative questions.

42. Lessons learned from project managing high volume e-assessment David Bacigalupo EASiHE/ECS

43. Conclusion and next steps Lester Gilbert EASiHE/ECS