Chapter 14 Evaluation in Healthcare Education

1. Chapter 14Evaluation in Healthcare Education

2. An Evaluation Is: • The final components of; • Education process • Nursing process • Decision-making process • Because these process are cyclic ,so evaluation serves as a bridge at the end of one process that guides direction of the next process

3. Definition of Evaluation Gathering, summarizing, interpreting, and using data to determine the extent to which an action was successful A systematic process by which the worth or value of something-in this case teaching and learning –is judged

4. Evaluation • Evaluations are not intended to be generalizable, but are conducted to determine effectiveness of a specific intervention in a specific setting with an identified individual or group.

5. What is the relationship between Evaluation, Evidence-Based Practice EBP and Practice-based evidence PBE

6. Evidence-Based Practice EBP • EBP has evolved and expanded over decades and can be defined as the conscientious use of current best evidence in making decisions about patient care (Melnyk and Fineout-Overholt,2005,p.6) • It includes results of systematically conducted evaluation from researches

7. Practice-based evidence PBE • PBE is just beginning to be defined and include results of systematically conducted evaluation from practice and clinical experience rather than from research

8. The Difference between Assessment and Evaluation Assessment = Input Evaluation = Output

9. The Difference between Assessment and Evaluation Assessment and evaluation are two concepts that are highly interrelated and are often used interchangeably as terms, but they are not synonymous. Assessment: a process to gather, summarize, interpret, and use data to decide a direction for action. Evaluation: a process to gather, summarize, interpret, and use data to determine the extent to which an action was successful.

10. Formative and summative assessment • Formative assessment • A set of formal and informal assessment methods undertaken by the teachers at the time of the learning process is known as Formative Assessment. It is a  part of the instructional process, which is undertaken by the teachers, with an objective of enhancing the student’s understanding and competency, by modifying teaching and learning methods.

11. Formative Assessment attempts to provide direct and detailed feedback to both teachers and students, regarding the performance and learning of the student. It is a continuous process, that observes student’s needs and progress, in the learning process. • The goal of formative assessment is to monitor student learning to provide ongoing feedback that can be used by instructors to improve their teaching and by students to improve their learning.

12. More specifically, formative assessments: • help students identify their strengths and weaknesses and target areas that need work • help faculty recognize where students are struggling and address problems immediately

13. Formative assessments are generally low stakes, which means that they have low or no point value. Examples of formative assessments include asking students to: • draw a concept map in class to represent their understanding of a topic • submit one or two sentences identifying the main point of a lecture • turn in a research proposal for early feedback

14. Summative assessment • refers to the evaluation of students; that focuses on the result. It is a part of the grading process which is given periodically to the participants, usually at the conclusion of the course, term or unit. The purpose is to check the knowledge of the students, i.e. to what extent they have learned the material, taught to them.

15. Summative Assessment, seeks to evaluate the effectiveness of the course or program, checks the learning progress, etc. Scores, grades or percentage obtained to act as an indicator that shows the quality of the curriculum and forms a basis for rankings in schools.

16. The goal of summative assessment is to evaluate student learning at the end of an instructional unit by comparing it against some standard or benchmark. • Summative assessments are often high stakes, which means that they have a high point value. Examples of summative assessments include: • a midterm exam • a final project • a paper • a senior recital

17. Information from summative assessments can be used formatively when students or faculty use it to guide their efforts and activities in subsequent courses.

18. Basic difference between Formative and summative assessment

19. Steps in Evaluation • Focus of evaluation • Designing evaluation • Conducting evaluation • Analyze and interpret data • Reporting and using result of data

20. STEP ONE Focus of evaluation

21. RSA Evaluation Model • RSA model places five basic types of evaluation in relation to one another based on purpose ,related question, scope and resources components of evaluation focus (Figure 14-1) • These 5 types leading from the simple to complex

22. RSA Evaluation Model high low frequency time & cost Impact Outcome Content Process low Total Program high

23. Process (Formative) Evaluation • Purpose: to make adjustments as soon as needed during education process • Scope: limited to specific learning experience; frequent; concurrent with learning

24. Content Evaluation • Purpose: to determine whether learners have acquired knowledge/skills just taught • Scope: limited to specific learning experience and objectives; immediately after education completed (short-term)

25. Outcome (Summative) Evaluation • Purpose: to determine effects of teaching • Scope: broader scope, more long term and less frequent than content evaluation

26. Impact Evaluation • Purpose: to determine relative effects of education on institution or community • Scope: broad, complex, sophisticated, long-term; occurs infrequently

27. Total Program Evaluation • Purpose: to determine extent to which total program meets/exceeds long-term goals • Scope: broad, long-term/strategic; lengthy, therefore conducted infrequently

28. STEP TWODesigning the evaluation

29. Designing the evaluationAn important question to be answered in designing an evaluation is “How rigorous should the evaluation be”All evaluation should be systematic and and carefully planned and structured before they are conducted.Evaluation design could be structured from a research perspective

30. Essential questions to be asked when designing evaluation tool • What types of data will be collected? • Complete (people, program, environment) • From whom or what will data be collected? • From participants, surrogates, documents, and/or preexisting databases • Include population or sample

31. How, when, and where will data be collected? • By observation, interview, questionnaire, test, record review, secondary analysis • Consistent with type of evaluation • Consistent with questions to be answered • By whom will data be collected? • By learner, educator, evaluator, and/or trained data collector • Select to minimize bias

32. Evaluation Barriers • Lack of clarity • Resolve by clearly describing five evaluation components. • Specify and operationally define terms. • Lack of ability • Resolve by making necessary resources available. • Solicit support from experts.

33. Fear of punishment or loss of self-esteem • Resolve by being aware of existence of fear among those being evaluated. • Focus on data and results without personalizing or blaming. • Point out achievements. • Encourage ongoing effort. • COMMUNICATE!!!

34. Selecting an Evaluation Instrument • Identify existing instruments through literature search, review of similar evaluations conducted in the past. • Critique potential instruments for: • Fit with definitions of factors to be measured • Evidence of reliability and validity, especially with a similar population • Appropriateness for those being evaluated • Affordability, feasibility

35. STEP THREE Conducting an evaluation

36. When conducting an evaluation: • Conduct a pilot test first. • Assess feasibility of conducting the full evaluation as planned. • Assess reliability, validity of instruments. • Include extra time. • Be prepared for unexpected delays. • Keep a sense of humor!

37. STEP FOUR Data Analysis and Interpretation

38. Data Analysis and Interpretation The purpose for conducting data analysis is two-fold: • To organize data so that they can provide meaningful information, such as through the use of tables and graphs, and 2. To provide answers to evaluation questions. Data can be quantitative and/or qualitative in nature.

39. STEP FIVE Reporting and using result of data

40. Reporting and using Evaluation Results • Be audience focused. • Begin with a one-page executive summary. • Use format and language clear to the audience. • Present results in person and in writing. • Provide specific recommendations. • Stick to the evaluation purpose. • Directly answer questions asked.

41. Use data as intended. • Maintain consistency between results and interpretation of results. • Identify limitations.

42. Summary of Evaluation Process • The process of evaluation in healthcare education is to gather, summarize, interpret, and use data to determine the extent to which an educational activity is efficient, effective, and useful to learners, teachers, and sponsors. • Each aspect of the evaluation process is important, but all of them are meaningless unless the results of evaluation are used to guide future action in planning and carrying out interventions.