Using Multiple Mini Interviews in the Admissions Process

1. Using Multiple Mini Interviews in the Admissions Process • Rick Edwards MHS, PA-C • Director of Admissions • ECU Physician Assistant Program

2. So How Do We Get and Keep The Best Students?

3. Introduction • This presentation focuses primarily on processes having to do with current evidence relating to Graduate level Medical Education • Most competitive candidates are highly qualified academically, BUT • Prior academic performance sheds little information on the professionalism, integrity or emotional intelligence of candidates

4. Why does attrition matter? • Educating gradual level medical professional students represents a significant cost to students and universities • Significant investment in educational equipment, salaries, and teaching space are required • Whether at a private or public institution, loss of tuition paying students can significantly impact costs of education • But most importantly, dismissed students represent a significant lost opportunity to provide badly needed medical providers to the communities that most need them.

5. Traditional Methods of Interviewing • Traditionally, applicant interviews have been conducted in a variety of ways: • A panel of faculty members (often 4-5) conducted an interview with an individual candidate and reached a collective recommendation • Individual faculty members met a candidate and asked “general questions” of a candidate. Often applicants could be trained to effectively answer “predictable” questions • There was often significant variability in how applicants were rated by different interviewers

6. Applicant written statements and personal letters of reference have tended to have very little value in predicting students with: • Resilience • Integrity • Compassion and Empathy • Critical Thinking and Communication Skills • Moral and ethical attitudes • Supplementary Information to Application

7. Intimidating? Group Interview

8. Development of the MMI • This mode of interview was developed at McMaster Hospital Medical School approximately 14 years ago • Two problems were widely recognized at the time • Traditional interviews had poor predictive value of which students could successfully complete medical training • Patient complains about physicians most often concerned non-cognitive personality traits, including communication skills and moral/ethical concerns that were often not discovered during traditional interviews

9. Development of the MMI • Rather than traditional interview formats candidates had multiple shorter interviews with several interviewers, typically 8 minutes or less • Questions addressed more “soft topics” including communications skills, ethical concerns, empathy and team work • Pilot studies conducted by McMaster showed significant correlation between applicant scores on MMI and subsequent performance as students • Studies have shown that the optimal number of stations is 7-9 • Further study demonstrated that both applicants and interviewers found the process to be fairer and more efficient than traditional interview formats

10. Development of the MMI • Further study demonstrated that MMI format interviews were free of gender, socioeconomic and cultural bias • Significantly, results also demonstrated that prior “interview coaching” did not improve applicant performance • While MMI format interviews require more preparation and a increased number of interviews, they remain cost effective

11. ECU’s Experience with Traditional Interviewing Techniques • We traditionally interviewed utilizing a one on one faculty member to candidate format with no specified questions. • Each candidate was rated on a 10 point scale with additional comments • A six person admissions committee then met to make admissions decisions • It was a SLOW and laborious process • We sometimes picked candidates late in the admissions season only to learn they had already matriculated elsewhere • Our process did not adequately identify students who would experience academic or professional behavior issues after admission

12. Revamping the Interview Process At ECU • An MMI interviewing process was adopted in 2015 • Focused 8 minute interviews were developed with written scenarios • There were a total of eight stations • Every attempt was made to ensure that the same rater performed the same station in every interview for inter-rater reliability • The admissions committee was reduced to the two co-directors of admissions • Faculty members were encouraged to decide at the time of the interview what their preliminary decision would be

13. Application of Process to the ECU Program • MMI’s with consistent stations and questions were developed • Program Director • Two Group Teamwork Stations • A written station • Individual stations focusing on • Maturity • Critical Thinking • Professionalism • Ethics/Character • Communication Skills/Collaboration

14. Our New Process • Each Station has a written scenario for the student to read prior to the room • The faculty reviewer receives a score sheet that contains specific scoring guidelines • The score for each question is averaged to achieve a score of 0-7 • There are 3 additional components to the scoring • A wholistic score • A firm admissions recommendation • A faculty review box for concerns

15. The Wholistic Score • The wholistic score was added to get the overall impression of the candidate’s suitability for the program • It was scored on a 0-3 scale as follows: • 0: Extremely Poor • 1.5: Poor • 2.0: Adequate • 2.5: Good • 3.0: Ideal • This score is added to the average from the specific questions

16. The Faculty Recommendation • Our previous interviewing process did not provide a real opinion on the interviewers impression of the suitability of a candidate • It did infer an opinion based on scores, but made the process somewhat more complicated when attempting to make admissions decisions • The new recommendation process incorporated a firm opinion by the interviewer • They were: • Accept • Hold for Further Consideration • Reject • This provided us the opportunity to use a scoring model of A/H/R for decisions

17. Faculty Review • This box was developed as a fallback method for concerns noted by the interviewer • Any applicant with this box checked received further discussion with the interviewer and a larger group of faculty • It provided an opportunity to broaden the discussion when there was no apparent problem, but “something just doesn’t feel right”

18. A Sample set of Scenarios • Applicant Prompt: • During the first year, a struggling classmate that you have befriended confides in you. You learn that your classmate used a forged diploma to claim credit for prerequisite classes they did not actually take. The next morning before classes, you have a routine meeting with your faculty advisor who states that you seem troubled and asks you what is on your mind. How do you respond to this question?

19. Scoring Sheet SCORING SHEET for MMI Station 4: First year study-buddy APPLICANT NAME: FACILITATOR NAME: Rick Edwards Date: September 4, 2018 SCORING: Circle the appropriate score per line, then AVERAGE and write in the average score below. Circle the appropriate score per line, then AVERAGE and write in the average score below. • The candidate appears to understand the ethical implications of lying on the application? N/A 1 2 3 4 5 6 7 2. Does the candidate shows empathy for the student relationship? N/A 1 2 3 4 5 6 7 3. The candidate expresses a willingness to facilitate the student turning himself/herself in? N/A 1 2 3 4 5 6 7 4. Does the candidate accept responsibility for reporting the ethical violation? N/A 1 2 3 4 5 6 7 Facilitator: Please average the scores given Please choose one of the below categories as an overall choice for the whole station Extremely Poor Poor Adequate Good Ideal 0 1.5 2 2.5 3 Please add the Average to the score above (maximum score available per station is 10 points)

20. Team Scenario • Applicant Prompt: • You are a passenger on a yacht that was wrecked on an uninhabited island far from any other known islands or land. You have enough fresh water and food for only 3 days. Compile a ranked list of the 10 greatest priorities to improve your chances of survival and rescue.

21. Scoring Sheet Group Scenario

22. Team Scenario • The team scenario allows the interviewer to observe how candidates work in a team environment • Do they contribute meaningfully to attempting to solve the problem • Do they provide leadership without trying to “take over” the group • Do they seem to provide suggestions that reflect careful thought • Are they respectful of fellow team members in their interaction • The team exercised is also video taped for subsequent review when necessary

23. Altruism/Empathy Station • Applicant Prompt: If you personally had the ability to make a single decision that would change the world, what would it be? • Facilitator prompts: • Do you think you have ever changed someone’s life? • Do you think you could change a community? • How do your actions affect others?

24. Scoring Sheet • SCORING: • Circle the appropriate score per line, then AVERAGE and write in the average score below. • 1. How creative a response does the candidate come up with? • N/A 1 2 3 4 5 6 7 • 2. Does the response show a concern for others (altruism)? • N/A 1 2 3 4 5 6 7 • 3. Is the response a solution to a significant world issue? • N/A 1 2 3 4 5 6 7 • 4. Does the choice of responses reflect maturity in the candidate? • N/A 1 2 3 4 5 6 7 • 5. Does the choice of responses reflect the mission of underserved populations/ECU’s mission? • N/A 1 2 3 4 5 6 7

25. Our Experience with Results of the New Format • Academic attrition dropped significantly: • Class of 2014 2 • Class of 2015 3 • Class of 2016 5 • Class of 2017 1 • Class of 2018 0 • Class of 2019 0 • Class of 2020 0 (so far)

26. Our Experience with the New Format • Professionalism Probation • Class of 2014 1 • Class of 2015 2 • Class of 2016 0 • Class of 2017 0 • Class of 2018 0 • Class of 2019 0 • Class of 2020 0 • These numbers represent concerns significant enough to merit probation with the possibility of dismissal

27. Our Experience with the New Format • Professionalism concern • Class of 2014 1 • Class of 2015 2 • Class of 2016 2 • Class of 2017 3 • Class of 2018 2 • Class of 2019 4 • Class of 2020 0 • These numbers represent students who committed “soft” violations that merited discussion, but not serious enough to merit probation

28. What kinds of Negative Professionalism Behaviors Were We Experiencing? • A student who attempted to pretend throughout a clinical rotation that he was from Great Britain and spoke with an English accent • A student who stood up in class and berated an instructor for their perception of the effectiveness of their teaching • A student who argued with an onboarding coordinator for a local hospital and ultimately submitted a falsified document to demonstrate compliance with chickenpox immunity • And too many more!

29. The “Diamonds in the Rough” Program • With the widespread competitiveness of the application process, admissions decisions have become particularly “numbers” driven • It is actually harder to get into many Physician Assistant programs than Medical Schools • However, there are some candidates who would never be considered for admission but have significant life or professional experience that would broaden a class and contribute maturity • We save two spots in our class of 36 to admit students with those experiences • Our research is ongoing into the ability of those students to perform academically • Our hope is that we can broaden the diversity of our program overall

30. “Diamonds in the Rough” • These students typically have a less competitive GPA from early coursework • They have demonstrated an ability to mature and grow and perform satisfactorily academically • Some examples • The Green Beret Medic • The Homeless Student • The Student who ultimately earned a PhD in Anatomy

31. Conclusions • The MMI interview process has been successful at ECU in decreasing attrition and professionalism issues • It does not unfairly favor any socioeconomic group, gender, or ethnic group • Post interview surveys of our applicants and faculty indicate that: • Applicants enjoy the process • Faculty interviewers feel it is more revealing and consistent • Both groups feel the interview process is fairer

32. Rick Edwards MHS, PA-C 252-744-6273 Edwardscl14@ecu.edu Thank You.