1 / 31

Moving the Needle Forward: The Role of Research Design and Analysis in Training Evaluation

This presentation discusses the importance of research design and analysis in training evaluation, focusing on reliability in measurement, survey development, and the use of rating scales. It also addresses the potential impact of unreliable measurement tools on decision-making.

maryf
Download Presentation

Moving the Needle Forward: The Role of Research Design and Analysis in Training Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Moving the Needle Forward: The Role of Research Design and Analysis in Training EvaluationFED Evalu-Con: September 26, 2016JonePapinchock, Ph.D.Corporate UniversityFederal Deposit Insurance Corporation The views expressed in this presentation are those of the presenter and do not necessarily reflect the views of the FDIC.

  2. Overview of Presentation General Concept of Reliability in Measurement General Concept of Validity in Measurement Reliability of Rating Scales Guidance on Survey Development (Avoiding “Oops”) Survey Response Rate Other Topics: Over-reliance on Surveys; End-of-Course Tests (Beware of Misuse or Misinterpretation of Scores)

  3. The Big Question How did you get here?

  4. How did you get here? • “My jobs have always been focused on some aspect of measurement.” • “I am an Instructional System Designer so evaluation is part of my position description.” • “I was assigned the role of evaluation with little background in the topic.” • “I do not know.” • “I drew the short straw.” • “I am interested in measurement.”

  5. Why does it matter? Moving the needle forward in terms of developing our skills and improving measurement in the training function will vary for each of us (and our programs) because of where we are starting. Let’s get started with terms.

  6. What is Training Evaluation? The use of measurement instruments to gather reliableand valid information that can be interpreted for making judgments and decisionsregarding learning and development interventions.

  7. What is a Measurement Instrument? • Any method of collecting information in a standardized manner for a given purpose (e.g., answer a research or business question). • Examples: Surveys, Interviews, End-of-Course Tests, Focus Groups, Observation of Performance Back On the Job.

  8. How Does Reliability Apply to Evaluation? • Reliability Refers to Consistency in Measurement. • No Measurement is Perfectly Consistent. • There Is Always Some Form of Error.

  9. Reliability What Role Does Reliability Play in Evaluation?

  10. Reliability What Role Does Reliability Play in Evaluation?

  11. Reliability What Role Does Reliability Play in Evaluation?

  12. Reliability Activity: You will only get one instruction. Interpret the instruction the best you can without help or discussion with others. Measure the blue object on this paper using the measuring instrument you received.

  13. Reliability Activity: Measure the blue object on this paper using the measuring instrument you received. What is the measurement?

  14. Reliable, But Not Valid Not Reliable Reliability

  15. Reliable and Valid Validity

  16. Reliability • What are the tools we use in training or program evaluation? • Are we using tools that lack reliability but making decisions as if they are reliable? • Are we thinking about the reliability of the tools?

  17. Reliability of Rating Scales • Rating Scale Reliability • Are the rating scales we use capturing participants’ reactions reliably? • Do all participants have a common understanding of the values assigned to the rating scale? • Are our ratings scales ensuring a positive outcome? That is, are they biased? • Are our rating scale results interpretable?

  18. Reliability of Rating Scales Activity: We are trying to develop a 10-point rating scale and want to assign adjectives to each rating scale point. A value of 1 is the lowest value or least positive scale value. A value of 10 is the highest value or most positive scale value. Which word would you choose for a value of 1? Which word would you choose for a value of 5? Which word would you choose for a value of 10?

  19. Reliability of Rating Scales Activity: Forming a group with others around you, please choose the word or phrase that best describes a value of 1? Awful Exceptionally Poor Extremely Poor Horrible Terrible

  20. Reliability of Rating Scales Activity: Forming a group with others around you, please choose the word or phrase that best describes a value of 5? Alright Could be improved OK So-So The least that can be expected

  21. Reliability of Rating Scales Activity: Forming a group with others around you, please choose the word that best describes a value of 10? Awesome Fantastic Superb Superior Tremendous

  22. Reliability of Rating Scales We have a choice of two rating scales: • Terrible to Superior • Very Bad to Very Good What impact are these rating scales likely to have on survey respondents?

  23. Reliability of Rating Scales We have a choice of three rating scales: • Very Poor-Poor-Average-Good-Very Good • Poor-Average-Good-Very Good-Excellent • Awful-Very Poor-Poor-Average-Good What impact are these rating scales likely to have on survey outcomes?

  24. Reliability of Surveys Are there any problems with these survey items? Please indicate your agreement with each of the following statements on the scale: Strongly Disagree (1), Disagree (2), Neither Agree Nor Disagree (3), Agree (4), Strongly Agree (5). • The information in the presentation was interesting and will be applicable to my job. • I prefer presentations with a lot of activities. • I find holidays fun but stressful.

  25. Developing Surveys • Identify the purpose of the survey. • Only ask questions that support the purpose. • Write questions with care: • Avoid double barrel questions/statements. • Make sure the respondents have the knowledge to answer accurately. • Ensure there is only one interpretation of the question. • Choose the rating scale with care. • Determine sample size (if everyone will not be included). • Evaluate the impact of the response rate.

  26. Survey Response Rate Activity: • We presented training to 100 people and asked all of them to complete an end-of-course feedback survey. • 25 people responded. • The average ratings on the survey items were all between 4.75 and 5 on a 5-point scale. What can we say about the reaction to the training? (Choose all that apply and discuss with others) • The training was successful. • The people who responded had positive feelings about the training. • We do not know the overall impact of the training. • The people who did not respond are likely to share the opinions of those who responded.

  27. Survey Response Rate Activity: • We presented training to 100 people and asked all of them to complete an end-of-course feedback survey. • 25 people responded. • The average ratings on the 3 survey items were 5, 4.76, and 4.76, on a 5-point scale. We follow up with the 75 people who did not respond and 60 of them provide responses. These 60 people were not as positive as the first 25. One-half of them gave ratings of 2 and the other half gave ratings of 3. Now the average ratings on the 3 survey items are 3.24, 3.16, and 3.16.

  28. Survey Response Rate Activity: • We presented training to 100 people and asked all of them to complete an end-of-course feedback survey. 25 people responded. • The average ratings on the 3 survey items are 5, 4.76, and 4.76, on a 5-point scale for 25 people. • The average ratings on the 3 survey items are 3.24, 3.16, and 3.16 after adding responses from 60 more of the people. Are we safe to assume that the remaining 15 people could not change the average by much? No. If the remaining 15 people were very dissatisfied with the course and all gave ratings of 1, the averages would be 2.9, 2.84, and 2.84.

  29. Survey Results Do not overstate what the survey results indicate: • Think about the limitations in our ability to reliably measure some concepts (e.g., training participant opinions). • Be aware of problems with survey questions or rating scales that could have impacted the interpretation of the survey. • Consider the impact of a low response rate, if one occurs, on our confidence in whether the results represent the majority of the course participants or not. • Remember that self-report methods have limitations; and survey respondents may have different motivations when answering a survey.

  30. Other Topics • Do participants understand what Level 3 surveys are asking? One Experience. • In End-of-Course tests – have we achieved sufficient reliability and validity? How do we know? Could test scores be misused? Are we controlling access to numerical scores.

  31. QUESTIONS?

More Related