Evaluation of Training and Education Activities

1. Evaluation of Training and Education Activities

2. Objectives By the end of this presentation, participants will be able to • List reasons why evaluation is needed • Describe different types of evaluation • Discuss how to use different types of evaluation for training and education activities

3. Evaluation (1) As trainers, what do we want to evaluate?

4. Evaluation (2) • What have we accomplished? • What was the training? Where? How many people were trained? Cost? • How well have we done? • Did the trainings go as planned? • Did the participants like the training? Was the training relevant? • What were the strengths and weaknesses?

5. Evaluation (3) • How effective is the training? • To what extent did the training achieve the objectives? • Did the participants increase their knowledge and skills? • Did they change their job performance? Did health status improve? • How can we improve on it? • Are the training objectives appropriate, given the available resources?

6. Systematic Health Education Process NeedsAssessment (Formative Research) Assess Effectiveness Development (Outcome and Impact Evaluation) Evaluation Pilot-testing Implementation (Formative Evaluation) (Process Evaluation)

7. When to Begin Evaluation? • Consider evaluation up front (in planning phase of training project) • Allocate time • Allocate resources • Evaluation actually occurs at every stage of development • Determine the type of evaluation you need

8. Levels of Evaluation • Formative (before) • Identifies strengths and weaknesses of training/product before implementation • Process (during) • Describes the implementation of the training or distribution/use of the product • Summative/Outcome/Impact (after) • Assesses the short-term and long-term effects of the training/product

9. Four Types of Evaluation Impact Summative Evaluation Outcome Program implementation ends here. Process Program implementation occurs here. Formative

10. Goals and Objectives • Goal • Describes the overall mission/purpose of the program • Broad general statement that clearly states the problem that will be addressed • Objectives • State the expected result • Specific, measurable, attainable, realistic, time-framed • Partial accomplishment of a goal • Activities • Specific actions or procedures expected to occur to achieve objectives

11. Formative Evaluation Impact Summative Evaluation Outcome Program implementation ends here. Process Program implementation occurs here. Formative

12. Formative Evaluation(Pilot Testing) • Conducted with the target audience • Answers questions about whether trainings/products are • Understandable, relevant, attractive, credible, acceptable, effective • Identifies strong and weak points • Identifies confusing terms or concepts • Allows for necessary revision prior to implementation • Prevents production of ineffective products

13. Formative Evaluation Methods • Focus groups • With various types of participants (e.g., experienced, non-experienced) • Interviews (key informant/audience) • Informal discussion • Open-ended questions • Understanding • Appeal • Utility • Test the curriculum in the “real” environment • Readability testing (literacy and comprehension)

14. Formative Evaluation Measures • Ease of use or intuitiveness • Readability • Reading level • Tone • Fonts, colors, spacing • Comprehension • Terminology and illustrations • Concepts

15. TB: The Hollywood Movie

16. Formative Evaluation • Screen tests • Movie screenings • Focus groups TB

17. Process Evaluation Impact Summative Evaluation Outcome Program implementation ends here. Process Program implementation occurs here. Formative

18. Process Evaluation(Monitoring of Training/Curriculum) • Examines the procedures and tasks involved in implementing a training/product • Assesses if the training/product is working as planned • Answers questions such as: • Are the activities being implemented as planned? • Is the material being used as intended? • How many participants attended the program? • Were the participants satisfied with the educational materials?

19. Process Evaluation Measures • Product inventory • Hard copies distributed • Web pages viewed or downloaded • Attendance • Checklist and timetable review • Budget, $$$$ spent • Audience feedback on quality of program/material during implementation (participant reactions)

20. Participant Reactions • Do audience/participants think that: • Objectives are relevant to participants' work • Course content is realistic • Training methods are suitable • Facilitators are effective • Training environment (classrooms, meals, accommodation) contributes to learning

21. Participant ReactionsMethods • Written feedback – participants write down their reactions • Ensures that all participants provide comments, not just the outspoken ones • Both open- and closed-ended questions are asked • Oral feedback and observations • Observations by facilitators • Facilitators meet at the end of each day to discuss their opinions and participant comments

22. Participant Reaction Form

23. Process Evaluation • Ticket sales • Critical reviews • Rotten tomatoes TB

24. Outcome Evaluation Impact Summative Evaluation Outcome Program implementation ends here. Process Program implementation occurs here. Formative

25. Outcome and Impact Evaluation Answers the questions… • What have you accomplished? • How well have you done it? • How much have you done? • How effective have you been?

26. Outcome Evaluation • Based on learning objectives • Evaluation of learning & performance of skills • Assesses immediate or short-term effects • End of course or immediately after distribution • Helps draw conclusions about effectiveness • Answers questions such as: • Were learning objectives achieved? • What knowledge and skills were gained as a result of the training/product?

27. Outcome Evaluation Effectiveness of Self-Study Modules on Tuberculosis Knowledge Mean Scores Pre-test Post-test Print 74% 91% Satellite 81% 92% Web 76% 92% (p<.01)

28. Impact Evaluation Impact Summative Evaluation Outcome Program implementation ends here. Process Program implementation occurs here. Formative

29. Impact Evaluation • Gathers information about the long-range results of the training/product and improvements in health status • 3 months, 6 months, one year following implementation • Answers questions such as: • Did the user’s job performance improve? • What are the changes in job practice of user or participant? • What changes were implemented in the program? • Costly, time consuming, but important

30. Should You Measure Impact? • Ask yourself the following questions: • Do you have the time and resources? • Do you have support from management and partners? • Are you able to reach the audience at this later date? • What will motivate the audience to participate?

31. Impact EvaluationMethods • Methods • Surveys • Key-informant interviews • Direct observation/performance checklists • Document/record reviews/treatment registers • Reviewing of program indicators • Timeline • Depends on the goal of the training; however, generally 1 to 6 months after training

32. Summative Evaluation • Domestic and international box office revenues • DVD sales • Franchise opportunities TB

33. Evaluation Reports

34. Evaluation Reports • Justify program, course, or material to stakeholders • Allow for identification of areas for improvement • Provide evidence for additional needs • Share lessons learned

35. Most Importantly… Plan for evaluation at the beginning and conduct throughout development process