1 / 48

Measuring and Evaluating Training

Measuring and Evaluating Training. Session Topics. Purpose and Benefits of Evaluating Training Measurement Process Issues in Assessment Development Formative vs. Summative Evaluation. Session Topics. Kirkpatrick’s Four Levels ROI: Methodology and Calculation

koen
Download Presentation

Measuring and Evaluating Training

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring and Evaluating Training

  2. Session Topics • Purpose and Benefits of Evaluating Training • Measurement Process • Issues in Assessment Development • Formative vs. Summative Evaluation

  3. Session Topics • Kirkpatrick’s Four Levels • ROI: Methodology and Calculation • Three Measures of Central Tendency • Uses of Analysis

  4. Purpose of Evaluating Training What is the Purpose of Evaluating Training? • Determining business impact • Improving the design of the learning experience • Determining the content’s adequacy

  5. Benefits of Evaluating Training What are some benefits of evaluating training? • Secure client support and build relationships w/client • Allows practitioner to see whether the results of learning are consistent w/business analysis and needs assessment • Helps focus the training

  6. Benefits of Evaluating Training What are some benefits of evaluating training? • Validates performance gaps and learner needs • Determines where training is the solution to a gap

  7. QUESTIONS?

  8. Measurement Process List the basic steps of evaluation. • Identify evaluation goals • Develop an evaluation design and strategy • Select and construct measurement tools • Analyze data • Report data

  9. QUESTIONS?

  10. Issues in Assessment Development Validity vs. Reliability • Validity: measuring what you intended to measure • Reliability: ability of the same measurement to produce consistent results over time

  11. Issues in Assessment Development How do Validity and Reliability Relate to Each Other? • Validity: concerns what is being measured • Reliability: concerns how consistently the measure is measuring what it is intended to measure

  12. Issues in Assessment Development What are the five types of validity? • Content validity • Construct validity • Concurrent validity • Criterion validity • Predictive validity

  13. Issues in Assessment Development What is content validity? • Extent to which the items on the evaluation reflect a representative sample of that which is to be measured • Do the items on the evaluation encompass what was taught?

  14. Issues in Assessment Development What is construct validity? • Extent to which the evaluation we implemented contained the content we intended it to and the outcome we measured was what we intended to measure • Do the items on the evaluation test for what we wanted to test for?

  15. Issues in Assessment Development What is concurrent validity? • Extent to which an assessment’s results agree with the results of other assessments administered at approximately the same time to measure the same characteristics • Does the assessment bear out or conflict with known and accepted standard measures or criteria?

  16. Issues in Assessment Development What is criterion validity? • Extent to which a test predicts some criterion (measure of performance), usually in the future • Does doing well on this assessment predict that the person will do well elsewhere later in life?

  17. Issues in Assessment Development What is predictive validity? • Extent to which the results obtained by the evaluation agree with results obtained from more direct and objective measurements taken later on • Does doing well on this assessment predict that the person will do well when tested on the same material later on?

  18. Issues in Assessment Development If you don’t have a statistician at your disposal, what other source do you have for validating your evaluation? • SME

  19. QUESTIONS?

  20. Formative vs. Summative Evaluation • What is Formative Evaluation? Assessment of the effectiveness of the training program while the program is still under development. • What is Summative Evaluation? Assessment of the effectiveness of the training program after the program has been delivered.

  21. QUESTIONS?

  22. Kirkpatrick’s Four Levels What are Kirkpatrick’s 4 Levels of evaluation? • Reaction • Learning • Behavior • Results

  23. Kirkpatrick’s Four Levels List some of the tools you could use to conduct a Level 1 evaluation. • Reaction sheets • Word-of-mouth feedback

  24. Kirkpatrick’s Four Levels Which level assesses the overall impact of the program on the organization? • Level 4

  25. QUESTIONS?

  26. ROI: Methodology and Calculation • What are some ways to assign monetary value to program improvement? • Benchmarking • Expert input • Estimates • Control groups • Customer input

  27. ROI: Methodology and Calculation What is the formula for deriving ROI? ROI = Total Benefits – Program Costs X 100 Program Costs

  28. ROI Calculation Exercise

  29. QUESTIONS?

  30. Three Measures of Central Tendency • What is another word for “measures of central tendency?” • Averages • What are the three “measures of central tendency” we are concerned with today? • Mean • Median • Mode

  31. Three Measures of Central Tendency • Define Mean • The sum of all numbers divided by the number of values that make up the sum • 64 + 73 + 82 + 98 = 317 (sum of values in list; aka Σx) • 317 ÷ 4 = 79.25 (mean; Σx ÷ n, where n is the number of values in the list)

  32. Three Measures of Central Tendency • Define Median • The number separating the higher half of a sample from the lower half. The median of a finite list of numbers can be found by arranging all the observations from lowest value to highest value and picking the middle one. If there is an even number of observations, the median is not unique, so take the average of the two middle values.

  33. Three Measures of Central Tendency • Define Median • Less sensitive to extreme scores than the mean is, so the median is a better measure than the mean for highly skewed distributions.

  34. Three Measures of Central Tendency • Define Median • 64, 73, 82, 98 (scores from previous slide) • 73 + 82 = 155 (even number of scores, so add the two in the middle and… • 155 ÷ 2 = 77.5 …divide the result in half to get the median)

  35. Three Measures of Central Tendency • Define Mode • The most frequently occurring score in a distribution. • Greatly affected by sample fluctuations, so not recommended for use as the only measure of central tendency. • Distributions can be multi-modal; contain more than one mode.

  36. Three Measures of Central Tendency • Define Mode • 3 + 8 + 5 + 3 + 9 + 2 + 6 + 3

  37. Three Measures of Central Tendency • Define Mode • 3 + 8 + 5 + 3 + 9 + 2 + 6 + 3 • Mode = 3

  38. Central Tendency Exercise

  39. QUESTIONS?

  40. Uses of Analysis • What are some of the uses for ROI evaluation, cost-benefits, utilities, and forecasting? • To measure a program’s economic contribution • To measure the degree of change in retention • To isolate the effects of training

  41. Uses of Analysis Case Problem Activity • An organization has been running the same new hire orientation program for over a year. • The HR director wants to determine the program’s effectiveness, with a desire to improve it. Also, HR staff want to know why they seem to be answering the same questions repeatedly from new hires.

  42. Uses of Analysis Case Problem Activity • Your boss asks you, the primary trainer, to identify the strengths and weaknesses of the program. • Company management has indicated they would like to offer the new hire orientation program online, in connection with links to the other helpful information that HR provides.

  43. Uses of Analysis Program Evaluation Activity As a program evaluation consultant, you have been hired to evaluate a training program at Horizons, Inc. Complete the handout to perform your evaluation.

  44. QUESTIONS?

  45. Session Review • Purpose and Benefits of Evaluating Training • Measurement Process • Issues in Assessment Development • Formative vs. Summative Evaluation

  46. Session Review • Kirkpatrick’s Four Levels • ROI: Methodology and Calculation • Three Measures of Central Tendency • Uses of Analysis

  47. Next Session • Selecting and Planning Your Work Product

  48. Questions?

More Related