Closing the Educational Loop: Feedback and Assessment as Learning Tools

1. Closing the Educational Loop: Feedback and Assessment as Learning Tools Teresita McCarty MD B.A.T.C.A.V.E. University of New Mexico School of Medicine Surgery HPSN 2008 First let�s find our perspective.First let�s find our perspective.

2. Formative tidal volume question A healthy adult, Ursula is rapidly transported from sea level to the top of a high mountain. Ursula soon observes that she is breathing faster. What is Ursula most likely observing about the depth of her breathing? Depth unchanged Less deeply More deeply Michael, JA. Adv in Physiology Education 19:1:S90-98,1998 40% of 393 undergraduates correctly predicted that the depth of respiration would increase on exposure to high altitude. Most students predicted tidal volume would decrease. Michael, JA. Students� misconceptions about perceived physiological responses. Adv in Physiology Education 19:1:S90-98, 199840% of 393 undergraduates correctly predicted that the depth of respiration would increase on exposure to high altitude. Most students predicted tidal volume would decrease. Michael, JA. Students� misconceptions about perceived physiological responses. Adv in Physiology Education 19:1:S90-98, 1998

3. Why a question? Answer is expected Engaged, active Information �chunked,� feels manageable Social context Little at stake Return to novice state Illustrates larger issues Potential misconception Formative assessment Feedback to me Answer is expected Engaged, active Information �chunked,� feels manageable Social context Little at stake Return to novice state Respond to cues Illustrates larger issues Potential misconception Formative assessment Feedback to me Answer is expected Engaged, active Information �chunked,� feels manageable Social context Little at stake Return to novice state Respond to cues Illustrates larger issues Potential misconception Formative assessment Feedback to me

4. Objectives Contrast formative and summative assessment Describe learning impact of formative assessment Compare formative assessment and deliberate practice Illustrate how assessments can be used to: Facilitate learning Make learner progress decisions Inform curricular development and revision Fulfill institutional tracking and reporting responsibilities To emphasize professional and institutional values To guide instructor teaching efforts As a basis for making learner progress decisions To inform curricular development and revision To fulfill institutional tracking and reporting responsibilitiesTo emphasize professional and institutional values To guide instructor teaching efforts As a basis for making learner progress decisions To inform curricular development and revision To fulfill institutional tracking and reporting responsibilities

5. Assessment types Summative Certify mastery Assign grades Formative Frequent Feedback to teacher and learner Focus on learning Finds misinformation/misconceptions Changes the form of future teaching Less formal Most of you are very familiar with summative tests. These are tests like a course midterm and final. The MCAT, the GRE, the SAT and USMLE Steps 1, 2 & 3 as well as your specialty boards and your recertification examinations are all summative tests. Unfortunately, some of you may be less familiar with formative tests. The reason that is unfortunate is because formative tests have been demonstrated to result in larger improvements in learning than most other interventions. Toothpick to test if the cake is done. So we are going to demonstrate a couple of techniques.Most of you are very familiar with summative tests. These are tests like a course midterm and final. The MCAT, the GRE, the SAT and USMLE Steps 1, 2 & 3 as well as your specialty boards and your recertification examinations are all summative tests. Unfortunately, some of you may be less familiar with formative tests. The reason that is unfortunate is because formative tests have been demonstrated to result in larger improvements in learning than most other interventions. Toothpick to test if the cake is done. So we are going to demonstrate a couple of techniques.

6. Formative assessment research Literature review of 250 studies �Innovations that include strengthening formative assessment produce significant and often substantial learning gains.� Effect sizes larger than most educational interventions Helps low achievers most Black, Wiliam. Assessment in Education 5:7-74,1998 Always involves enhanced feedback Requires actively involved learners Attention to motivation important Always involves enhanced feedback Requires actively involved learners Attention to motivation important

7. Formative assessment framework Establish criteria Learning objectives Mastery-based standards Foster learner self-efficacy Success is achievable and likely Provide systematic feedback, measures Blueprint Offer opportunities to try again Deliberate practice Evaluate learning via self-reflection Life-long learning Develop faculty knowledge and skill

8. Deliberate Practice Practice with the primary goal of improving some aspect of performance Well defined task Appropriate level of difficulty At edge of proficiency Detailed, immediate feedback Opportunities for repetition & improvement

9. Improve formative assessment in clinical training Clear learning objectives Set standards Blueprint assessment Educate educators Formative methods Sustain educators Share methods Emphasize improvement over outcomes Build feedback loops Analysis of learning needs

10. Educate educators Ask questions Wait for a response Allow thinking, dialogue, time to investigate Encourage deep thinking rather than quick response Give feedback Frequent, specific, connect the dots for novices Combine measurement with feedback Why was the right response the right response? Learn to respond with a fact Learn to not respond and faculty will answer Chart Stimulated Recall What, How documented but Why? Sniffing student Review books Learn to respond with a fact Learn to not respond and faculty will answer Chart Stimulated Recall What, How documented but Why? Sniffing student Review books

11. Summary � Formative Assessment Fundamentally simple Identify the desired goal Provide evidence of present position Guide to close the gap Complex in execution Powerful in impact

12. Best assessment practices Determine what you want to assess. Design a simulation that evokes this performance. Observe/record the performance. Analyze the performance using some type of rubric: checklist, GRS, etc. Debrief/give feedback about performance Debrief/receive feedback about teaching Repeat The OSCE based tool used by the ABEM has shown a similar level of validity when applied to the simulator environment. Gordon 2003, Clyne 2004.The OSCE based tool used by the ABEM has shown a similar level of validity when applied to the simulator environment. Gordon 2003, Clyne 2004.