Plagiarism in Residency Application Essays

1. Plagiarism in Residency Application Essays Brian J. Gelfand, M.D., F.A.C.S. Scott Segal, M.D., M.H.C.M. Brigham and Women’s Hospital Harvard Medical School

2. Truth

3. “Journal retracts article by Harvard doctor.” January 30, 2008 • Lee Simon, M.D. resigns after discovery that greater than 50% of his published paper has been plagiarized from another author’s publication.

4. Why? How? • Lapses in professionalism in medical school and residency training can be predictive of future disciplinary action by state medical boards. Papadakis MA, et al. Acad Med. Mar 2004;79(3):244-249 Papadakis MA, et al. Ann Intern Med. Jun 2008;148(11):869-876

5. The Harvard Crimson Ex-Harvard Student, Adam Wheeler, Pleads Not Guilty to Charges of Fabricating Academic History May 2010 Falsified transcripts Falsified standardized exam scores Falsified letters of recommendation Falsified publications Plagiarized academic work $ 45,000 (£ 30,420) grants and awards

6. Academic Competition?

8. Unique Population • Highly educated • All have Doctoral degrees • Many with multiple post-baccalaureate degrees • These applicants often are educators and hold teaching positions at respected universities

9. Education vs. (Lack of) Understanding Scientific American February 2009

10. Patient Care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health • Medical Knowledge about established and evolving biomedical, clinical, and cognate (e.g. epidemiological and social-behavioral) sciences and the application of this knowledge to patient care • Practice-Based Learning and Improvement that involves investigation and evaluation of their own patient care, appraisal and assimilation of scientific evidence, and improvements in patient care • Interpersonal and Communication Skills that result in effective information exchange and teaming with patients, their families, and other health professionals • Professionalism, as manifested through a commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population • Systems-Based Practice, as manifested by actions that demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value

11. Applying for US Postgraduate Education • Electronic Residency Application Service (ERAS) • A service of the Association of American Medical Colleges (AAMC) to facilitate the National Resident Matching Program (NRMP) • Computerized algorithm that matches over 35,000 applicants to available residency positions each year.

12. Primary Source material • Official academic transcripts • Standardized examination scores (USMLE, COMLEX) • Medical Student Performance Evaluation (MSPE/Dean’s Letter) • Letters of Recommendation

13. Applicant supplied information • Research Activities • Publications • Volunteerism • Personal Statement

14. Personal Statement • Free-form format • Content is not specified • Limited only by length • Approximately 8 pages

15. Personal Statement • Information not covered elsewhere in the ERAS application • Further elaboration on already submitted information • Opportunity for creative expression of the applicant’s individuality Jean-Michel Labat Ancient Writing

16. Personal Statement • Articulate reasoning and aspiration for training in the specific specialty • Express a desire to train at a particular program

17. Personal Statement • Above all permits the applicant to “speak” to the admissions committee and uniquely present themselves in an attempt to secure an interview

18. ERAS Instructions Clearly warns that “any substantiated findings of plagiarism may result in the reporting of such findings to the programs to which (they) apply now and in the future.”

19. ERAS • Before electronically submitting, applicants must “certify that work is accurate and original before an ERAS application is complete.” Disney 1945

20. Plagiarism in applications for US Postgraduate Medical Education • 10 – 30% prevalence of misleading publication citations • Falsified authorship • Non-existent articles, journals Baker DR, Jackson VP. ActaRadiol. Sep 2000;7(9):727-729 Cohen-Gadol AA, et al. Surg Neurol. Oct 2003;60(4):280-283 Dale JA, Schmitt CM, Crosby LA. J Bone Joint Surg Am. Dec 1999;81(12):1679-1681 Yang GY, et al.. J Am CollRadiol. Apr 2006;3(4):259-264

21. Plagiarism in Graduate Medical EducationCole AF, Fam Med Jun 2007;39(6):436-438 • Geriatric Medicine Fellowship • 3/26 application personal statements over 2 years • Verbatim from single website sample essay • “…failure in personal integrity.” • “…lack of professionalism…” ACGME Core Competencies

22. Plagiarism in residency application essays • Originally an assessment of Anesthesiology residency applications (n=692) • Expanded to the 5 largest training programs at our institution (BWH) • Anesthesiology, Internal Medicine, Emergency Medicine, Obstetrics and Gynecology, General Surgery (n=5010)

23. Study Design • Essays exported from individual training program’s ERAS database • De-identified by name and NRMP application number • Assigned randomly generated 6 symbol alphanumeric code e.g. A12345 or IM67890 • IRB waiver for informed consent

24. Study Design • Other data exported to include: • Gender, age, medical school location, volunteer work, research, work experience, Alpha Omega Alpha membership, USMLE scores, citizenship • All matched with computer generated codes and de-identified

25. Plagiarism Detection Software • Algorithmic comparison between submitted essay to maintained database • Web pages, printed resources, previously submitted essays • Similarity score between 0-100% • Percentage suspect text

27. Applicant Pool

28. Study Population • Confined to a single institution • Sample represents 28% to 45% of total applicants in studied specialties • Study ratios of US and international applicants consistent with total NRMP applicant pool

30. Essay Cross-matching Protocol A A* Set to 0 Set to lesser of 2 scores A A* A B As reported

31. Distribution <10% • Similarity Score 0-4% minimal chance true plagiarism (Proper names, individuals, institutions, or common short phrases) • 4-10%: mixture of “innocent” matches and others judged likely to be plagiarized

32. Study not designed to determine best cutoff value of similarity score • ≥10 Value for dichotomous cutoff for unquestionable plagiarism • Individual review of ≥10 subset (n=338) revealed no instances felt to be false positives

33. Only a formal sensitivity analysis of varying cutoff values could determine a “best” value • Would require an agreed upon “gold standard” for plagiarism

34. Distribution of Similarity Scores

37. Results • 4975/5010 essays screened • 35 no essays or unable to be extracted for technical reasons • Similarity Score ≥10 in 5.2% essays overall

38. Plagiarism Prevalence All Applicants • US citizen 65/3561 (1.8% ) • Permanent resident 48/353 (13.6%) • Foreign national 143/1028 (13.9%)

39. Underestimation? • Similarity Score <10% threshold not incorporated • Incomplete Turnitin database • Applicants plagiarizing from other applicants’ essays not in the public domain or previously submitted are missed • Close paraphrasing involving careful word substitution may not be detected

40. WHO? • Applicants with AOA membership, research experience, higher USMLE Step 1 scores were less likely to demonstrate plagiarism (p=0.0001)

41. Plagiarism Prevalence Other Characteristics

42. Perceived competitive edge?

43. WHO? • International applicants, whether defined by citizenship, medical school type/location were significantly more likely to demonstrate plagiarism (13.9% p=0.0001)

44. Cultural Sensitivity • Increasing number of international applicants • Varying interpretations defining plagiarism and attitudes towards it • Cultural perceptions of copying work may not be as negative as seen by Western academics

45. Response • Public scrutiny of physicians’ ethical behavior • Demands to enforce strict rules of conduct • Begin with the application process

46. Punitive action for individuals in this study? • All essays were de-identified by using an automated computer algorithm • Some involved applicants undoubtedly were invited for interview • Some candidates may have even been selected • Authors had agreed to maintain the anonymity of the essays and not unmask the identity of the applicants

47. Now What? • Action must be taken to discourage and detect plagiarism • Centralized database at the national level (ERAS) • Process unavoidable by applicants • Eliminate difficulties of multiple programs using software simultaneously • Essay cross-matching

48. Unique Issues • Unlike College level course assignments • A single application essay may be sent to multiple programs or specialties • May be used in subsequent years for reapplication by same candidate

49. Summary • Significant evidence of plagiarism in the application system among 5 specialties at a major academic training hospital • More common among international applicants but also among those with significant academic honors

50. Summary • Unique population • Application essays are not term papers • Challenges for detection • Increasing international applicant pool • Address cultural variation