Workshop on the Table of Specifications

1. Workshop on the Table of Specifications Presented by Dr. Chan Chang Tik

2. Contents • The Table of Specifications and the Bloom Taxonomy • Construction of Examination Questions in line with the Learning Outcomes Semester Break Training

3. Group Activity 1 Discussion How do you know whether the examination questions you set are valid? Semester Break Training

4. Valid? • Moderator’s comments: • I think is a significant problem. The students can’t answer the question properly on the basis of the information given …. I consider the question is either unfair or unanswerable. • “Why it is here?” I accept auditing is part of the learning outcomes but there seems to be too much auditing. Semester Break Training

5. Test Validity • To be valid a test must measure what it is intended to measure. • Three points to consider in test validity: • Validity is specific to some particular use • Validity is a matter of degree • Validity pertains to the results of the test Semester Break Training

6. Types of Validity • Content validity • Criterion-related validity (concurrent and predictive) • Construct validity • Which one to choose depends on the aims of your testing. Semester Break Training

7. Content Validity • The test should cover a cross section of the course. • Emphasize important topics but give less attention to less important topics. • The relative importance of topics in a test is directly proportional to the instructional emphasis given to the topics. Semester Break Training

8. Table of SpecificationsStudent’s Response Q Why? Step 1 • Number of lectures/chapters/units • Number of questions in the final exam • One table of specifications for each question Semester Break Training

9. Step 1 Cont. • 17 lectures in a course and final exam consists of 5 questions Q1 – 3 lectures Equally distributed? Q2 – 4 lectures Q3 – 3 lectures Grouping Q4 – 4 lectures Q5 – 3 lectures Discussion Semester Break Training

10. Step 2 • Learning objectives • Instruction • Assessment • Example (Mat250, Q1) • List down the number of lectures • List down the instructional objectives Semester Break Training

11. Step 3 Bloom’s taxonomy of cognitive skills: • Knowledge – remembering facts, terms, definitions, and concepts. What?, list, name, define, describe • Comprehension – understanding the meaning of material. Explain, interpret, summarize, give examples, predict, translate Semester Break Training

12. Bloom’s taxonomy • Application – selecting a concept or skill and using it to solve a problem. Compute, solve, apply, modify, construct • Analysis – breaking material down into its parts and explaining the hierarchical relations. How does … apply?, Why does … work?, How does … relate to … ? Semester Break Training

13. Bloom’s taxonomy (2) • Synthesis – producing something original after having broken the material down into its component parts. How do the data support …? How would you design an experiment that investigates … What predictions can you make based upon the data? Semester Break Training

14. Bloom’s taxonomy (3) • Evaluation – making a judgment based upon a pre-established set of criteria. What judgments can you make about …? Compare and contrast … criteria for … ? Refer to the table of specifications Semester Break Training

15. Table of Specifications Selected-Response Q • Work as a team, for example each lecturer is assigned 5 lectures to come up with 50 multiple choice questions. • Store any extra questions in the question bank. Semester Break Training

16. Why Table of Specifications • External Examiner’s Comments • Several topics covered in the lecture programme are not assessed by this exam. My analysis would indicate no assessment of: The Netherlands, France, Poland, Singapore, South Korea, Hong Kong (10 of the 28 weeks of teaching, 35.7%). • The module descriptor, the lecture programme and the exam do not tie up very well. Semester Break Training

17. Why Table of Specifications (2) • INTI-UC lecturer’s comment I just want to say a million thanks for teaching us how to do the Table of Specifications in setting exam questions. It is only now that I am able to appreciate how useful that table is, now that I have managed to create my first set of final exam questions (deadline today lah!). I use the table and I am happy with the questions I set because I could justify them if I were asked. I think this part of the skill training (the Table of Specs part …) should be made compulsory for all new lecturers. Semester Break Training

18. Table of Specifications Student’s Response Questions Consider Question 1 of MAT250. There are 4 parts in this question taken from the following lectures: Learning Outcomes Lecture 1 To define real numbers. To describe exponents and to produce the laws of exponents. To explain and simplify radicals. Lecture 2 To define the various types of equations. To relate linear equation to its applications. Lecture 3 To solve quadratic and radical equations. To discover some practical problems of equations. To explain some coordinates geometry topics. Lecture 4 To define inequalities. To solve linear, quadratic and rational inequalities. To solve equations and inequalities involving absolute values. Lecture 9 To produce partial fractions. Semester Break Training

19. Table of Specifications (2) Bloom 3 Semester Break Training

20. Table of Specifications Selected Response Questions Semester Break Training

21. Group Activity 2 • Set three questions each at different level of the Bloom’s Taxonomy using a table of specifications. Indicate clearly your learning outcomes. • Present your questions to the participants. • Time : 45 minutes Semester Break Training

22. Types of Assessment • Essay • Mathematical and technical problem items Semester Break Training

23. Essay • Use clear and concise questions • Measure effectively complex learning outcomes • Establish scope of expected response • Handling of technical terms • Content versus mechanics of expression • Use descriptive words of precise meaning as directions • No optional items Semester Break Training

24. Clear and Concise Questions • Language usage and word choice are particular important during question construction. • The language dimension is critical because it controls the comprehension level of the item for the students and also it specifies the question parameters. Example Semester Break Training

25. Clear and Concise Questions (2) • For example, “Describe the origins of World War I” • A better question is “What were the principal diplomatic events in Europe between 1890 and 1913 that contributed directly to the outbreak of World War I” • You have to narrowly specify, explicate, define, or otherwise clarify what it is that you want from the students. Semester Break Training

26. Clear and Concise Questions (3) • The question set must have an answer that “expert” could agree upon, thereby rendering it objective. For example, “Describe the world and give two examples”. Example Semester Break Training

27. Complex Learning Outcomes • Complex learning outcomes can be measured effectively with essay questions. These are the abilities to: • Explain cause-effect relationships • Describe applications of principles • Present relevant arguments • Formulate tenable hypotheses • Formulate valid conclusions Semester Break Training

28. Complex Learning Outcomes (2) • These are abilities to: • State necessary assumptions • Explain methods and procedures • Produce, organise, and express ideas • Integrate learning in different areas • Evaluate the worth of ideas Semester Break Training

29. Extended versus Restricted Response • From the instructor’s standpoint, an extensive response to a few broadly based questions allows an in-depth sampling of a student’s knowledge, thinking processes, and problem-solving behaviour relative to a particular topic. • The open-ended nature of the question posed by the instructor such as “Discuss the bond and share markets in the Malaysian context” is challenging to a student. Semester Break Training

30. Extended versus Restricted Response (2) • Extensive response question is poorly structured and its free-response essay item tends to yield a variety of responses from the students, in both content and organisation, and thus inhibit reliable grading. Example Semester Break Training

31. Extended versus Restricted Response (3) • A restricted-response question is preferable. For example, “Discuss the advantages and disadvantages of essay and short-answer question with respect to (a) reliability, (b) objectivity, (c) content validity, and (d) usability. • This question presents a better defined task which lends itself to reliable scoring, yet allows students sufficient latitude to organise and express their thoughts creatively. Semester Break Training

32. Content versus Mechanics of Expression • Factors such as expression, grammar, spelling, and the like can be evaluated in conjunction with contents of the essay. • Evaluation of such skills is legitimate provided they are included in the learning outcomes. • You may point out such mistakes in the student’s coursework. Semester Break Training

33. Directive Words • Compare • Defend • Trace • Contrast • Explain • Differences Semester Break Training

34. Non-directive Words • Review • Tell all that you know • Report your knowledge Semester Break Training

35. Classic Case Question (10 marks) Tell me all you know about Malaysia. Answer It is a beautiful country. Semester Break Training

36. Example for Objectivity of Question • External Examiner’s comment: • The students can’t answer the question properly on the basis of the information given as they don’t know how to spread the finance charges and looking at the three main alternative methods is not part of the question. I consider the question is either unfair or unanswerable. Semester Break Training

37. Example of Extensive Response • ‘The relevance and the effects of accounting practices, have been the subject of much debate’. Discuss the impact on the accounting practices. • External Examiner’s comments: • Give source. • I don’t think this is a clear question and I would be amazed if any student produced a response anything like the outline answer. Semester Break Training

38. Example of Language Usage • A business or organisation can be positioned as effective as products and services. • Internal examiner’s comment: • The sentence does not make sense. I suggest “Successful marketing of products and services often depends on effective positioning in the market place.” Semester Break Training

39. Group Activity 3 • Look back at the essay or short-answer questions you set in Group Activity 2. Do you think you need to make any adjustments? • Present your questions to the participants. • Time : 20 minutes. Semester Break Training

40. Mathematical and Technical Problem Items • Provide enough information and directions so that students clearly understand the problem. • Indicate the degree of precision expected in the response. • Clearly specify the units for reporting the response. • When using regular “everyday” words to present a problem, make sure the connection between the structured situation and the real world is genuine. Semester Break Training

41. Enough Information • During the vacation Sue and Tai agreed to paint their house. One side of the house is eight-meters long. They mark a point on the side that is four meters from each other. Each one starts at one end and paints toward the center mark, four meters from each other. Can we be sure that each painted the same area of that side of the house if each paints to the center mark? Semester Break Training

42. Degree of Precision • In the following set of exercises for calculating z-scores from the raw scores, use the mean and standard deviation values to the nearest hundredth of a raw score unit. • The proportions that you report for the ratios should be given in decimal fractions to the nearest hundredth. Semester Break Training

43. Units • Total elapsed time should be reported in minutes and seconds. • When reporting the perimeters, give your responses in meters. Semester Break Training

44. Everyday Words • What words can you make from the letters in TEA? TEA, ATE, EAT, ETA, AET, TAE • Create a permutation for the three letters in TEA. Semester Break Training

45. Group Activity 4 • Look at the mathematical and technical questions set in Group Activity 2. Are the questions ambiguous? Do you want to re-set them? • Present your questions to the participants. • Time: 20 minutes Semester Break Training

46. Thank you Semester Break Training