Question setting

1. Question setting • IEA Insight is a key support for IEA members. • Through Insight you can tap into a range of materials in one place that can: • help you develop your own knowledge and skills in specific areas of assessment • help you develop your team’s knowledge and skills • give you access to information that we hope you will find stimulating and informative.

2. Question setting This short IEA Insight will give you a glimpse of some key aspects of question setting More detailed information on the subject is available to IEA members in the members area of www.ioea.org.uk This Insight will cover Question types Some general points on good practice Things to avoid

3. Question setting Whether you are setting questions for assessment in your own centre, class or tutor group or involved in setting questions for public examinations, there are still good and bad approaches to the process.

4. Question setting Compare two main characters from different works of literature from different centuries. Good question?

5. Question setting Compare two main characters from different works of literature from different centuries. “There is no evidence that Lady Macbeth liked fish; Tarka the Otter loved it.” “Whereas Tarka was despatched with a spade, it was a balcony wot done for Lady Macbeth.”

6. Question setting Poorly written questions have a negative impact on the learners who are faced with them Poorly written questions also create frustration amongst assessors

7. Types of Question Summative Assessment Multiple choice Short answer Extended answer / essay Continuous Assessment Tasks End of unit tests Extended essays

8. Types of Question Knowing which type of question to use, when, is a skill in itself

9. Multiple Choice Questions • Benefits • Objective marking – requires no interpretation from marker • Quick to answer • Test a breadth of knowledge quickly • Marking is fast and consistent • Disadvantages • Limited testing of skills • Need very tight focus • Longer to set than other question types • Setting is a specialist skill

10. Multiple Choice Questions Terminology Question = stem Correct answer = key Choices = options Incorrect answers = distractors

11. Multiple Choice Questions - Example The capital city of Spain is (STEM) A Barcelona (DISTRACTOR) B Madrid (KEY) C Valencia (DISTRACTOR) D Aragon (DISTRACTOR)

12. Multiple Choice Question Stems Stems can be written in various ways Among these are: Completing a statement The capital city of Spain is ... Logic and reasoning Which of the following statements can be justified using the information in the passage alone? Odd one out Each of the following is a cause of drought except ...

13. Negative Stems Using the negative in a Stem can be confusing There are three things wrong with this question Which of the items in the following list is not one of the major organs of the human body? A Skin B Heart C Leg D Pancreas

14. Vague and Confusing Stems Which of the items in the following list is not one of the major organs of the human body? A Skin B Heart C Leg D Pancreas Some learners will focus on the word “organ” and simply pick one out The word “major” has not been defined The answer “Leg” is too obvious

15. Vague and Confusing Stems This question stem is confusing Which of these is not a good reason for the world economy to slow down in the event of a major natural disaster?

16. Vague and Confusing Stems Which of these is not a good reason for the world economy to slow down in the event of a major natural disaster? It uses the negative The main thrust of the question is separated from the opening It is difficult to word a correct answer and even harder to write distractors

17. Carefully Worded Odd One Out Questions It is valid to test the ability to recognise which items in a list stand out in some way Using the format: Each of the following is ... except can be a good way around the confusing negative stem problem

18. True / False Questions True / False questions are not true multiple choice questions e.g. A basic principle of physics is that energy can neither be created nor be destroyed True or False?

19. True / False Questions A better approach would be to ask: A basic principle is that energy A can be created but not destroyed B can be destroyed but not created C can be destroyed and created D cannot be destroyed or created

20. True / False Questions A basic principle is that energy A can be created but not destroyed B can be destroyed but not created C can be destroyed and created D cannot be destroyed or created Even here the correct answer is the only one to contain the word “cannot” so it might need further revision

21. Multiple Choice Questions Summary • You have now seen some of the benefits of multiple choice questions. • They can: • be objective • be answered quickly • test a breadth of knowledge • be marked quickly and consistently

22. Short Answer Questions SAQs can be marked By human markers - possibly subject specialists By human markers who are not specialists On screen On line Automatically - if the list of possible outcomes is finite and easily referenced

23. Short Answer Questions Be clear in your demands Avoid ambiguity e.g. Say how glaciers move SLOWLY And the old maths classic:

24. Short Answer Questions are useful for developing points slightly further than multiple questions would allow a good way to explore a learner’s knowledge quickly easily administered and marked - but not always automatically are not Multiple Choice Questions Essay titles If the point you wish to test is ideally suited to another method then use the other method

25. Extended Answer Questions Traditional essay style questions Used to test knowledge in depth Need to be carefully worded to test what is actually required

26. Writing Extended Answer Questions e.g. Give an account of “Macbeth” This pretends to be an essay but is nothing more than a request for a candidate to write out the plot. A low order skill would actually be tested

27. Writing Extended Answer Questions e.g. Macbeth is a good man who allows himself to be tricked by the forces of evil. Discuss the above statement with reference to at least three scenes from the play. This is clearer in its demands and gives an element of structure to the task which was not present in the previous question.

28. Structuring Extended Answer Questions • Having said that ... • Don’t turn an Extended Answer Question into nothing more than a series of SAQs (Short Answer Questions) • This can be the case if the question has • too many parts • e.g. Give two reasons why Britain sent troops to France in 1914 and three reasons why the USA entered the war and two outcomes of the treaty of Versailles.

29. Deriving Extended Answer Questions The question is being asked in order for a candidate to demonstrate knowledge and/or skills The question should come naturally from a programme of study and not be tacked on artificially

30. Extended Answer Questions can assess knowledge in depth probe into skills should not lead to repetitive low order tasks be unconnected to a programme of study

31. Showing Off It might be interesting to set questions for the local pub quiz which test esoteric knowledge BUT Question setters should resist the temptation to show the world how clever they are by setting candidates ridiculously difficult tasks If there is a clear form of words then use it

32. Remember what you are testing The question is being asked for a clear reason It is in clear language It asks what it seems to ask Deciphering the question should not be the test

33. Teaching to the tests or testing what is taught? Whilst the role of assessment remains very important in school, college and workplace education and training, we should all remember that assessment is not an end in itself. Keep a sensible balance between teaching / training and the various forms of assessment. The IEA hopes to engage members in wide ranging discussion about the nature and the future of assessment. You can find out more about question setting in the members’ area of the IEA website - www.ioea.org.uk