Item Dependency in an Objective Structured Clinical Examination

1. Item Dependency in an Objective Structured Clinical Examination Cherdsak Iramaneerat Carol M. Myford Rachel Yudkowsky

2. Objective Structured Clinical Examination Objective structured clinical examination (OSCE) An assessment approach used in medical education in which the clinical competence of residents is evaluated using multiple stations of standardized clinical tasks Standardized patients (SP) Lay persons trained to portray a scripted patient presentation in a standardized fashion

3. Conditional Item Independence A basic assumption of the Rasch model After accounting for the latent trait, item responses on a test are independent.

4. Items in OSCE In each OSCE station, items are linked to the same clinical task and are rated by the same SP. A resident�s level of performance on one item may be dependent on his/her level of performance on other items in the same OSCE station.

5. Purposes To check for the existence of item dependency in an OSCE To outline an alternative approach for analyzing rating data using a MFRM model to ameliorate the problem of item dependency To compare reliability estimates, parameter estimates, and fit statistics obtained from MFRM analyses when local dependence is present/absent

6. Participants 79 residents from one Midwestern medical school 68 internal medicine residents 66% Male 34% Female 11 family medicine residents 45% Male 55% Female

7. Tasks (OSCE Stations) A communication skills assessment Six OSCE stations of simulated clinical scenarios Patient education Informed consent Treatment refusal Elderly abuse Giving bad news Physical examination

8. Rating Scale A modification of the communication skills rating form of the American Board of Internal Medicine 18 items asking for agreement ratings Five-point Likert Scale 1 (Strongly disagree) to 5 (Strongly agree)

9. Items 1. You greeted me warmly... 2. You were friendly... 3. You treated me like we were on the same level... 4. You let me tell my story without interruption... 5. You were truthful... 6. You never ignored what I had to say... 7. You discussed options with me... 8. You made sure that I understood the options... 9. You allowed me to make my own decision... 10. You encouraged me to ask questions... 11. You were patient... 12. You never avoided my questions... 13. You clearly explained the problem... 14. You clearly explained what should be expected... 15. You used plain language, not medical jargon... 16. You were careful in approaching sensitive issues... 17. You displayed a positive attitude... 18. I will choose this physician as my personal physician.

10. Analyses Pnijk Probability of resident n receiving a rating of k on item i in station j Pnij(k-1) Probability of resident n receiving a rating of k-1 on item i in station j Bn Level of communication competence of resident n Di Difficulty of item i Cj Difficulty of OSCE station j Fik Difficulty of receiving a rating of k relative to k-1 for item i

11. Local Independence Yen�s Q3 statistic (Yen, 1984, 1993) the correlation of the residuals for a pair of items after partialling out the latent trait estimate Fisher�s Z approach (Shen, 1996) A modification of Yen�s Q3 statistic Adjusting residuals by the accuracy of the resident communication competence measure Establishing a practical significance level

12. Fisher�s Z Index Calculate the standardized residuals for each rating of resident n on item i dni = (observed rating � expected rating)/SEn Correlate the standardized residuals for all pairs of item i, j in each OSCE station Compute Fisher�s Z statistic

13. Alternative Approach Treating each OSCE station as a scoring unit Average the ratings from all items in one station and multiply by 10 to produce a station score. Integers ranging from 10 (poor performance) to 50 (excellent performance)

14. Alternative Analysis Pnjk Probability of resident n receiving a station score of k in station j Pnj(k-1) Probability of resident n receiving a station score of k-1 in station j Bn Level of communication competence of resident n Cj Difficulty of OSCE station j Fjk Difficulty of receiving a station score of k relative to k-1 for station j

15. Item Dependency

16. Resident Separation Reliability Using items as scoring units A resident separation reliability = 0.94 Using stations as scoring units A resident separation reliability = 0.74

17. Resident Communication Competence Measures

18. Resident Communication Competence Measures

19. Misfitting Residents

20. Station Difficulty Measures

21. Misfitting Stations

22. Item Dependency in MFRM Analyses A violation of a basic assumption of the model Results Overestimation of separation reliability estimates Poorer fit of resident communication competency measures (according to standardized fit statistics) Poorer fit of station difficulty measures (according to both standardized and unstandardized fit statistics)

23. Suggestions When conducting a MFRM analysis of a data set that has items linked to the same task or raters Check for the violation of local independence assumption If item dependency is a problem: combine ratings from multiple items into a station score Alleviate item dependency problem Loss of information and decrease in resident separation reliability

24. Questions and Comments Cherdsak Iramaneerat Department of Educational Psychology College of Education University of Illinois at Chicago cirama1@uic.edu