Measuring Vocational Competence Written Performance Assessments beyond Multiple Choice

1. Measuring Vocational Competence • Written Performance Assessments beyond Multiple Choice Katharina Stöttinger Department for Business Education and Development Karl-Franzens-Universität Graz Berlin, August 2013

2. How Students Feel About Tests • * Probably not an actual Einstein quote

3. Objectives for this Session • Gain insight into the challenges involved in developing an instrument to measure occupational skills • Get inspiration for designing competence-based learning opportunities for students of commercial subjects/students undergoing vocational training • Gain insights into ensuring a satisfactory level of consistency when it comes to the rating of open-ended test items.

4. Outline • General requirements for competence based assessments • Construction of suitable items • Construction of rating guidelines • Implication for teachers

5. General Reqirements for Competence Based Assessments Items should • represent authentic challenges grounded in real-world working situations • be open-ended • activate complex cognitive functions, requiring students to • reproduce factual knowledge • contextualize it • apply it to a specific situation • and potentially transfer it to new problems with a similar structure. • trigger responses that can be rated with a satisfactory level of consistency among various experts.

6. Case Study: A Comparison of Offers You work for FunInTheSun, a manufacturer of sporting equipment who sponsors the regional beach-volleyball tourney. Your boss Mrs. Kofler wants to use this occasion to distribute goody-bags with promotional freebies to V.I.P guests. Each goody-bag should contain a memory stick with the name of your company printed on it. Now Mrs. Kofler has asked you to compare offers from four manufacturers of promotional freebies, choose the best offer and write her a memo explaining your choice.

7. Case Study – A Comparison of Offers Consider the following context: • Today is July, 18th. The beach volleyball tourney will take place on July, 27th. Mrs. Kofler has asked you to hurry with your decision, since she doesn’t want to assemble the goody-bags at the last minute. • The guest list for the event contains 46 V.I.P guests. • Your budget for the memory sticks amounts to EUR 300,-.

8. Case Study – A Comparison of Offers Production Time Price per unit Minimum order quantity Bulk Discount Cash Discount Expiration Date

9. Case Study - Potential Solutions Cheapest option is also supplier with longest production time • Prioritize cost minimization and trust that there won’t be any delays • Choose the second cheapest supplier, who is only marginally more expensive than the cheapest option and could deliver a lot faster • Contact a third supplier, whose offer has already expired but who would be cheaper and faster than both options and ask whether the offer could be extended Answer should ideally demonstrate an awareness of trade-offs

10. Case Study - Competences

11. Case Study - Competences

12. Case Study – Common Misapprehensions • Students only compare the price per item • Minimum order requirements as a knock-out criterion • Failure to take advantage of discounts • General lack of insight into the interplay of quantity, price per item, discounts and percentages

13. Case Study – Learning Objectives • Make sure that things are compareable before you compare them. (Some prices contain VAT and costs for personalizations, some don‘t). • Price per unit is not the only factor that determines total cost. • Order quantity required is not the only factor that determines total cost. • Take advantage of discounts

14. Case Study – Learning Objectives • Know the conventions of business (eg. Bulk discounts are subtracted before cash discounts) • Be aware that certain conditions might be subject to negotiation (eg. Sometimes expired offers might be extended on request.) • Cost is not the only criterion – time and quality matter as well. • Be aware of trade-offs.

15. Case Study – Learning Objectives Transferable skills: Before making any decision • Make sure you know all the factors that influence the potential outcome. • Make sure you know how those factors influence each other (eg. the relationship between quantity and discounts) • Never look at a single factor in isolation.

16. Case Study – Learning Objectives Transferable skills When trying to convince others of your choice • Make your choice as transparent as possible • List all the factors that have influenced your choice • Point out any trade-offs involved • Explain your priorities with reference to the situation at hand. (=> Respect the complexity of a problem. Demonstrate awareness of multicausality, interdependencies, trade-offs and wider context.)

17. General Reqirements for Competence Based Assessments Items should • represent authentic challenges grounded in real-world working situations • be open-ended • activate complex cognitive functions • trigger responses that can be rated with a satisfactory level of consistency among various experts.

18. Case-Study – Rating Guideline Rating Guideline based on an analysis of • competences involved in a useful approach to the problem • common misconceptions and mistakes inherent in less competent approaches • learning objectives derived from these differences.

19. Case Study – Rating Guideline

20. Case Study – Rating Guideline

21. Personal Mission Statement as a Teacher of Commercial Subjects • I help people learn how to rationalize their choices. • I help people learn how to appreciate complexity. Guiding Principles: • There is usually more than one valid approach to anything. • Still, some approaches might make more sense than others. • The best approach is the one that makes the most sense to all the people involved in its implementation. • In order to help others make sense of your choices, make them transparent. • The goal is not objectivity, because objectivity is an illusion as soon as trade-offs are involved. • The goal is inter-subjectivity. • There is always more than one factor to consider. • Context matters.

22. Rating Guidelines as a Tool for Teaching Rating Guidelines • Require teachers to thouroughly reflect upon the challenges inherent in any given learning opportunity and students‘ potential missapprehensions in those regards • Make assessments more transparent and promote a perception of fairness • Provide a good framework for face-to-face feedback for students, promoting students‘ self reflection • Facilitate team-teaching