Measuring Human Performance

1. Measuring Human Performance

2. Introduction Kirkpatrick (1994) provides a very usable model for measurement across the four levels; Reaction, Learning, Behavior, and Results. These categories are discrete and can be measured. The goal of this presentation is to bring to light many of the topics, concerns, and issues that must be understood before carrying out the business of testing, measuring, or evaluating the success of training in the work force today.

3. What is a test? What is testing? The instrument used to collect data A process of collecting quantifiable information about the degree to which a competence or ability is present in the test taker. (Anderson, BC)

4. Reasons for Testing Prerequisite tests Entry test Diagnostic test Post test Equivalency test

5. Norm Reference Vs Criterion Reference

6. Norm Referenced Testing Test items separate test-takers one from another Normal distribution curve

7. Criterion Referenced Testing Test items based on specific objectives Mastery Curve / Skewed from Normal Distribution

8. SKA Skill Knowledge Attitude

9. Domains of Learning Cognitive Affective Psychomotor

10. Bloom’s Taxonomy for Cognitive Levels Knowledge Comprehension Application Analysis Synthesis Evaluation

11. Krathwohl’s Taxonomy for Affective Levels Receiving Responding Valuing Organization Characterization by a value or value complex

12. Simpson’s Taxonomy for Psychomotor Levels Perception Set Guided Response Mechanism Complex Overt Response Adaptation Origination

13. Test Items Related to Bloom’s Taxonomy Multiple Choice Most flexible across the Taxonomy spectrum, especially first three levels Advantages: Guessing probability low Diagnostic capabilities East to grade Statistical Analysis

14. Multiple Choice cont….. Disadvantages Difficult to write Provides keys for recall doesn’t do well for high level cognition evaluation

15. True and False Could be used at all levels but…. Advantages easy to write easy to score can to item analysis

16. T/F cont…. Disadvantages 50/50 guess factor often used when M/C seems too hard to write Reliability is so poor…..Very little evaluation value. So why do teachers often include T/F?

17. Matching Best suited for Application level….not recommended for any by me. Advantages Easy to write East to Grade Statistical Analysis

18. Matching cont… Disadvantage: Requires the two lower learning level Process of elimination diminishes probability low reliability Why would a teacher use Matching?

19. Fill in the Blank Best suited for the lower levels Advantage Recall is essential, few clues Disadvantage Single word or phrase grading beyond single word or phase is in trouble enters the realm of subjective grading..poor reliability

20. Short Answer Can get to the high order thinking Advantages Easy to write produces original responses Disadvantages Basically same as fill in….reliability

21. Essay The best for higher order Advantage high order creative ability writing ability

22. Essay cont… Disadvantage Tough to grade forget stats You’ll see this often in Master’s and Ph.d. classes

23. Validity Does the test measure what it is suppose to measure. How close to the bull’s eye did it hit.

24. Reliability How consistent is the test Is there a tight pattern of hits

26. Types of Validity Concurrent Validity Content Validity Criterion Related Validity Predictive Validity Construct Validity

27. Types of Reliability Test-Retest Reliability Inner-Rater Reliability

28. What is the real score of a test? An error factor must be considered test score + error factor

29. Ten Evaluation Instruments for Technical Training Interviews Questionnaires Group Discussion Critical Incident Work Diaries

30. Instruments cont... Performance Records Simulation Role-Play Observation Written Test Performance Test

31. Designing Tests Questions you must ask yourself Who is the test designed for? What do you want to know? How many Questions will be required? How will it be administered? How will it be scored?

32. 3 Methods of Test Construction Topic Based Statistical Based Objective Based

33. Topical Based Test Selection done by chapter Selection done by topic Selection done by the importance of the topic

34. Limitations of Topic System Procedure lacks precision Doesn’t identify test takers Not designed on learners level Doesn’t specify competencies

35. Statistical Selection Items statistically selected Standardized Norm Referenced

36. Limitations of Statistical What is measured not specific Lacks precision of CRT Difficult to select items

37. Objectives Based Test Based on defined competencies Applies to criterion referenced tests and scores

38. Testing and Kirpatrick’s Four Levels The more downward, from the performance of the company to the performance of the individuals, the more difficult to obtain. The more downward...the more usable the information

39. Four Levels REACTION LEARNING BEHAVIOR RESULTS

40. Reaction Checking individuals reaction often means, measuring “Customer Satisfaction” Happy rating sheets observations other How can you quantify the responses?

41. Learning Measurable behavior changes in the three “SKA” Dimensions

42. Behavior Behavior change due to training program. Surveys Interviews Other

43. Results Measurable by looking at changes in: production quality Safety Sales other