NIH LOAN REPAYMENT PROGRAM EVALUATION

1. NIH LOAN REPAYMENT PROGRAM EVALUATION NIH Loan Repayment Programs Extramural LRPs FY 2003-2009 Milton J. Hernández, Ph.D. Director Division of Loan Repayment OEP, OER, OD National Institutes of Health Bethesda, MD Laurel Haak, Ph.D. Chief Science Officer Discovery Logic, Inc. Rockville, MD

2. NIH LRP Evaluation Questions • Question 1: Who is applying to the NIH LRP? • Question 2: Does participation in the LRP affect career outcomes? • Question 3: How does the LRP compare to other NIH career development award programs? NIH Loan Repayment Programs

3. New and Renewal LRP Applications and Awards Trends Analysis, FY03-FY07 RENEWAL NEW The renewal program has decreased the success rate for new applications but has not affected the number of new applications submitted. The success rate for new applications from FY05-FY07 is 40% and for renewals is 70%. NIH Loan Repayment Programs

4. Gender Distribution of New LRP Applicants Trends Analysis, FY03-FY07 • The ratio of women to men LRP applicants is reversed from the ratio in recent graduating MD and PhD classes. • Success rates are not different. NIH Loan Repayment Programs

5. Gender Distribution of New LRP Applicants, by Degree Trends Analysis, FY03-FY07 About the same number of men and women applicants are MDs, but 2.5 times more women applicants are PhDs, and twice as many men applicants hold dual degrees. p<0.001 p<0.001 NIH Loan Repayment Programs

6. Race/Ethnicity of New LRP Applicants Trends Analysis, FY03-FY07 • More Blacks and fewer Asians applied as compared to recent MD and PhD graduating classes. • Success rates did not differ from each other or from the overall success rate. NIH Loan Repayment Programs

7. Educational Debt, by Degree Type Trends Analysis, FY03-FY07 Among LRP applicants, academic doctorates have a higher debt load than recent PhD graduates, and physician doctorates have a lower debt load than recent MD graduates. *Source: Survey of Graduates, American Association of Medical Colleges (2003-2007). †Source: Survey of Earned Doctorates, National Science Foundation (2003). Prepared by NIH OER/OEP/DLR

8. Years Since Degree Trends Analysis, FY03-FY07 Up to 3 Years (35%) 4-6 Years (37%) 10+ Years (10%) 7-9 Years (18%) • LRP attracts early career researchers • 72% of new applicants apply within 6 years of receiving their degree • 90% of new applicants apply within 9 years of receiving their degree • Academic doctorates and physician doctorates apply at different times • 67% of academic doctorates apply within 3 years of their degree • 68% of physician doctorates apply 4-6 years after receiving their degree NIH Loan Repayment Programs

9. NIH LRP Evaluation: Outcome Analysis • Question 1: Who is applying to the NIH LRP? • Question 2: Does participation in the LRP affect career outcomes? • Question 3: How does the LRP compare to other NIH career development award programs? NIH Loan Repayment Programs

10. Outcome Analysis Methodology • Outcome analysis: Regression Discontinuity Analysis. • We compared two nearly-identical applicant groups. LRP applicants with very similar priority scores were identified, about half of whom were funded and half of whom were not funded. Priority Score “Bubble” Funded Not Funded NIH Loan Repayment Programs

11. Identifying the Comparison Cohort LRP Applicant Cohort, FY03-FY06 Funded Priority Score range, e.g. 190-214 Identify bubble applicants foreach IC and each fiscal year Not Funded F259 Combine individual bubbles NF254 Outcome Analysis Subsequent publications and NIH grant participation F8 F9 F8 F11 NF9 NF8 NF9 NF14 IC-FY Bubble513 NIMH, FY03 NIDDK, FY04 NICHD, FY04 NIAID, FY05 A 50-50 mix of funded and not funded applicants with closely-matched priority scores NIH Loan Repayment Programs

12. IC-FY Bubble, Demographic Distribution Outcome Analysis, LRP Applicant Cohort, FY03-FY06 The demographics of the funded and not funded IC-FY Bubble cohorts are indistinguishable, hence they can be used as comparison groups. There are some notable differences compared to the total new applicant group.

13. NIH Grant Participation: IC-FY Bubble Overall p=0.0102 13

14. NIH Grant Participation: IC-FY Bubble, by Gender Outcome Analysis, LRP Applicant Cohort, FY03-FY06 Male Female p=0.0474 p=0.0147 Women LRP awardees are less likely than men to have applied for an NIH award or to have a subsequent observable role in the NIH extramural program. NIH Loan Repayment Programs 14

15. NIH Grant Participation: IC-FY Bubble, by Degree Outcome Analysis, LRP Applicant Cohort, FY03-FY06 Physician Doctorate Academic Doctorate p=0.0324 p=0.0106 p=0.0470 Participation in the LRP program increased the likelihood that a physician doctorate applied for a subsequent NIH award and was retained in the NIH extramural workforce. It also increased the likelihood that an academic doctorate was awarded an NIH grant. NIH Loan Repayment Programs 15

16. Question 3: Comparison with Other NIH Programs Identification of Comparison Cohorts K Cohort K01, K08, K23 LRP Cohort LRP+K Cohort Funded Type I awardees, FY03-FY06 Top 10% priority score, each FY/IC K awards only Funded Type I awardees, FY03-FY06 Top 10% priority score, each FY/IC L and K awards* Funded Type I awardees, FY03-FY06 Top 10% priority score, each FY/IC L awards only LRP+K Top-10% n = 96 LRP Top-10% n = 215 K Top-10% n = 218 Demographicparameters Researchaccomplishments Publications Outcome analysis Subsequent NIHgrant participation *L award between FY03-FY06 anda K01/K08/K23 award at any time. Awardees were in the top 10% of priority scores for both award mechanisms. NIH Loan Repayment Programs

17. NIH Grant Participation: K and L Programs Outcome Analysis, 0-6 Years Since Degree, Top-10% Comparison p=0.0003 p=0.0176 p=0.0003 Even after balancing for years since degree, the LRP recipients show less likelihood of applying for subsequent NIH funding than K or LRP+K awardees. LRP awardees are also less likely than K awardees to have a subsequent role on NIH awards. NIH Loan Repayment Programs 17