Objective assessment

1. Objective assessment Bob Rotheram Reader in Assessment, Learning & Teaching b.rotheram@leedsmet.ac.uk

2. Intended learning outcomes (Biggs 2003) • write good quality objective test items; • answer (mostly) correctly some questions about today’s subject material; • identify what you need to do next.

3. Tales of a dissatisfied student • Maths • Social work

4. The importance of… • Assessment • Constructive alignment (Biggs 2003) • Feedback

5. Seven pillars of assessment wisdom (Elton 2003) • “The drunkard looks under the lamp post for the keys which he has dropped, not because that is where he has dropped them, but because that is the place where there is light.” • Reliability pursued at expense of validity. • Students should get a profile. • Assessments to be valid, reliability treated within constraints of validity.

6. Constructive alignment (Biggs) • State intended learning outcomes (ILOs) • Find verbs students must perform to achieve outcomes • Assessment tasks require students to enact the verbs • Course activities to be aligned with enabling achievement of ILOs

7. Feedback • Potentially very powerful • (Brown G, 2001) Best if: • timely • perceived as relevant • meaningful • suggests ways of improvement (within student’s grasp)

8. Computer-assisted assessment (CAA): a demo • Social Policy Question Resource (SPQR) 10-question demo

9. Summative assessment Diagnostic tests Formative tests Informal quizzes Revision aid Convey what social policy is Individual questions v. flexible, ‘granular’, ‘learning objects’: insert into lectures gather further info via web links preparation for classes … SPQR: some uses

10. Important background ‘Need to know before…’ ‘Remedial’ ‘Nice to know’ ‘Much-repeated’ ‘Learned by doing’ Individual feedback needed Stuff you don’t like to teach! Hard to grasp first time May be needed later, at short notice (Race 2007) Resource-based learning (& CAA) best with…

11. CAA in your discipline • Use? • Potential/advantages? • Limitations/disadvantages?

12. CAA: key points • Much ‘up-front’ effort needed • Best where: • large numbers of students on introductory courses • questions have long ‘shelf-life’ • teams can produce shared item banks

13. Objective tests • “require a user to choose or provide a response to a question whose correct answer is predetermined” (Bull & McKenna, 2001) • “only as objective as the test designer makes it” (CAA Centre) • Reputation mixed • Best for lower-level skills???

14. Bloom’s Taxonomy (1956) • knowledge • comprehension • application • analysis • synthesis • evaluation

15. multiple-choice multiple response matching text match response/fill in blank extended matching set ‘sore finger’ graphical ‘hotspot’ true/false, yes/no multiple true/false ranking Likert scale numeric assertion/reason essay Some question types

16. Knowledge-testing questions • What word means the same as…? • What is the most important difference between…? • Which one of the following sequences shows the correct order of…? • What are the major classifications of…? • Which method is the most useful for…? • What evidence best supports the theory of…?

17. Comprehension-testing questions (if not taught or discussed) • Which one of the following is closest in meaning to the term…? • The statement ‘…’ means that… • (Various facts are presented.) Which of the reasons listed below best explains this?

18. Feedback: minimum • statement on whether chosen option correct or not; • (if incorrect) statement giving correct answer and why correct; • clear up possible misunderstanding about wrong choice; • source of info on correct answer; • language clear and not discouraging.

19. Quality control • Editorial board? • Evaluate in use: • Level of difficulty • Discrimination – appropriate?

20. Sources of items • Students • Colleagues • Staff development events • Question-setting workshops • Publishers’ websites

21. Publishers’ websites • E.g. Oxford UP Test Banks, ‘VLE cartridges’ • E.g. Bryman, social research methods: • 260 MCQs • Check terms of use

22. Item-writing • See ‘PASS-IT’ Guide, Section 2 • Sample MCQs: • ‘PASS-IT’ guide, Appendix A

23. Item-writing exercise • (alone) create 1 MCQ, 1 matching Q, answers, feedback • (2s/3s) ‘shred’ + refine items • (whole group) further consideration of some items

24. Further issues • Constructing assessments • ‘Content-free test’ (Race) • Scoring • Computerisation, inc. interoperability • Metadata • Item banks

25. 1. Who, according to Bob Rotheram, said this: “Assessment is the most important thing we do for HE students.”? • Lewis Elton • John Biggs • George Brown • Sally Brown • Phil Race

26. 2. Which of the following statements best reflects John Biggs’ concept of constructive alignment? • State the learning outcome we want students to achieve. Find a verb that students need to perform to properly achieve the outcome. Assessment tasks should require students to enact that verb. • Don’t just tell students about topics and then give them exams on what we have told them about problem-solving. • Constructive alignment is joined-up programme design, making sure that all the components fit together properly. • Constructive alignment is used in quality enhancement/assurance in university teaching.

27. 3. Which of the following does Brown G (2001) recommend as components of good feedback on student work? • Meaningful • Perceived as relevant • Extensive • Suggesting ways of improvement (within the student’s grasp) • Timely • Brief

28. 4. Which two levels of Bloom’s Taxonomy is this question most obviously testing?

29. 5. In the two lists below, match the questions with their types

30. 6. In each case below, indicate whether the statement is true or false. • If software is ‘IMS QTI-compliant’, it separates questions from presentation. • Computerised questions need to be IMS QTI-compliant to export them from one CAA package and import them into another. • In 2005, Questionmark Perception did not fully meet the IMS QTI specifications. • Respondus is fully IMS QTI-compliant. • The ‘TOIA’ project and ‘JORUM’ repository take IMS QTI-compliance into account. • The ‘RELOAD’ project does not take IMS QTI-compliance into account. • The chance of getting all the answers to this question correct purely by guessing is one in seven.

31. Intended learning outcomes (Biggs 2003) • write good quality objective test items; • answer (mostly) correctly some questions about today’s subject material; • identify what you need to do next.

32. Where next? • Your next steps?

33. Sources (1) • Biggs, J (2003) Teaching for Quality Learning at University (2nd ed.), Maidenhead, Open University Press. • Bull, J and McKenna, C 2001, Blueprint for Computer-Assisted Assessment: Chapter 3 and Appendix I (Republished 2004 by Routledge Falmer) • Good Practice Guide in Question and Test Design. http://www.pass-it.org.uk/resources/031112-goodpracticeguide-hw.pdf

34. Sources (2) • Brown G, Bull J and Pendlebury M, (1997) Assessing Student Learning in Higher Education, London, Routledge. • Elton L, (2003) Principles for a fair and honest approach to assessing and representing students’ learning and achievement, HE Academy website. • Race P, (2007) The Lecturer’s Toolkit, (3rd ed.) London, Routledge.