KimOanh Nguyen-Lam, Ph.D. Advanced Training & Research Division Director

1. Measuring the Impact of IFLE Grant Programs Evidence-Based Grant Design & Evaluation U.S. Department of Education, International & Foreign Language Education Program Andy Gluck Technical Director ICF International KimOanh Nguyen-Lam, Ph.D. Advanced Training & Research Division Director

4. Evaluation Proposed Priority

5. accomplished

6. FLAS Survey on Post-Graduate Outcomes • Mandated by the Higher Education Opportunity Act (HEOA) of 2008, Section 601: “The Secretary shall assist grantees in developing a survey to administer to students who have completed programs under this title to determine postgraduate employment, education, or training. All grantees, where applicable, shall administer such survey once every two years and report survey results to the Secretary.” • To be administered every other year for 8 years (QUALTRICS) • Comparison groups (national data sets: American Community Survey, the NSF Survey of Earned Doctorates, the Survey of Doctorate Recipients, or the Baccalaureate & Beyond Longitudinal Study)

7. IFLE Evaluation Contracts2011-2013 • EPI: Education Policy International • To assist IFLE staff in developing a comprehensive plan for analysis and formative assessment of the UISFL, BIE and NRC programs. • Data sources: IRIS database, Project Directors Surveys and Interviews • BPA: Berkeley Policy Associates IMPAQ • To Improve the effectiveness of IFLE’s programs and Increase transparency and accountability • Tasks: • Analyze existing performance data/measures (Logic Model) • Assist the development of outcome measures (GPRA), • Develop evaluation materials for grantees (Evaluation Guide)

8. OMB May 18, 2012 MOU

9. Evidence-Based Grant Program Designs Evidence-Based Program Design • Build evidence: use strong evaluation designs to test and find effective practices • Act on evidence:increase the share of funds that support evidence-based practices • Support knowledge diffusion and learning among grantees: Programs should disseminatefindings about what works and what does not to current and potential grantees • Build grantee capacity and use grantee input: Use training, technical assistance, collateral materials, grantee learning networks, and information systems that help grantees implement effective programs. • Return data to its suppliers with value added through analysis: Agencies that gather data from grantees should return the data collected to grantees in ways that the grantees find useful. • Harness available high-quality data: Programs should look for available high-quality data sets that can inform decisions and lower the cost of performance measurement, data analysis, and evaluation. • Engage both program and research offices: The most effective programs enable researchers to inform program design and implementation, and program staff to inform the research agenda.

10. A key recommendation from the 2011 grants improvement initiative was to revise EDGAR to reflect policy goals of the department (ED). As a result, ED developed a notice of proposed rulemaking (NPRM), published in the federal register (77 FR 74392) in December 2012. The amendments promote objectives related to evidence of effectiveness, project evaluation, simplified procurement procedures, sustainability, productivity, and strategy to scale in parts 75 and 77 of EDGAR. The notice of final rulemaking (NFRM) was published on August 13 and the changes became effective 30 days later – September 13, 2013.

15. Performance Measures GPRA Measures GPRA of 1993 - GPRA Modernization Act of 2010 requires federal agencies to develop and report to Congress measures that are: • quantifiable annual and long-term • ambitious, yet achievable targets from baseline data • based on program regulations and goals Project Specific Measures • Include both quantitative & qualitative • Support program goals • Link directly to project’s goals and objectives • Align with institutional goals • Have clearly defined outcomes

16. Performance Measures • Objective-linked • Responsibility-linked • Organizationally acceptable • Comprehensive • Credible • Compatible • Comparable with other data (useful in making comparisons, for example, performance can be compared from period to period, with peers, with target groups, etc.) • Easy to interpret Additionally, we have to be able to demonstrate that NRC program and project performance is: • Sustainable • Reasonable • Measurable • Replicable

17. Project Performance Measures • Student Outcomes: advanced proficiency (knowledge/skills), degree completion, employability • Faculty Outcomes: improved course curricula & delivery, published & disseminated new knowledge • Infrastructure Outcomes: courses, certificate, degree programs, signed agreements/collaboration, sustainability Other Research Study Questions: • Why should my institution care about international education? • Why should it be a priority in higher education or undergraduate program? • What are my institution’s overarching goal and priorities? • How can I align international education with them? • How do I set up an evaluation to produce concrete data to show the values of international abroad education? Education Abroad & academic performance (Kuh, 2009, McKeown 2010) Education Abroad & college persistence and completion (GLOSSARY 2001-09, U of Minnesota 1999-2009, Indiana2009, UC San Diego 2008-09)

18. IFLE PERFORMANCE MEASURE FORM (GPRA Measures)

19. IFLE PERFORMANCE MEASURE FORM (Project-Specific Measures)

20. IFLE NEW GPRA MEASURES FY 2014NRC, FLAS, CIBE, UISFL, AND DDRA

22. LRC GPRA MEASURESLanguage Resource Centers (LRCs)Program provide grants for establishing, strengthening, and operating centers that serve as resources for improving the nation's capacity for teaching and learning foreign languages through teacher training, research, materials development, and dissemination projects. • Percentage of LRC products or activities judged to be successful by LRC customers with respect to quality, usefulness, and relevance. • Percentage of LRCs products judged to be successful by an independent expert review panel with respect to quality, relevance, and usefulness. (In addition, experts could survey end users of LRC products.)

23. NRC GPRA MEASURESNational Resource Centers (NRC) Program provides grants to institutions of higher education or consortia of institutions of higher education to establish, strengthen, and operate comprehensive and undergraduate Centers that will be national resources for teaching of any modern foreign language. • Percentage of National Resource Centers that targeted and taught priority languages, as defined by the Secretary of Education. • Percentage of National Resource Centers that increased the number of intermediate or advanced level language courses in the priority languages and/or LCTLs during the course of the grant period. (Long-term).   • Percentage of National Resource Centers that taught intermediate or advanced courses in any of the priority languages as defined by the Secretary of Education. • Percentage of National Resource Centers that increased the number of certificate, minor, or major degree programs in the priority and/or LCTLs, area studies, or international studies during the course of the four-year grant period. • Efficiency measure: Cost per NRC that increased the number of intermediate or advanced level language courses in the priority and/or LCTL s during the course of the grant period.

24. FLAS GPRA MEASURES The Foreign Language and Area Studies (FLAS) Fellowship Program provides academic year and summer fellowships to institutions of higher education to assist undergraduate and graduate students in foreign language and either area or international studies. • Percentage of Foreign Language and Area Studies (FLAS) academic-year fellows who increased their foreign language reading, writing, and/or listening/ speaking scores by at least one ILR equivalent proficiency level. • Percentage of Foreign Language and Area Studies (FLAS) graduated fellows who secured employment that utilizes their foreign language and area studies skills within 8 years after graduation based on FLAS tracking survey. • Percentage of Foreign Language and Area Studies (FLAS) master’s and doctoral graduates who studied priority languages as defined by the Secretary of Education. • Efficiency measure: cost per Foreign Language and Area Studies (FLAS) Fellowship Program academic year fellow who increased his/her reading, writing, and/or listening/speaking language score by at least one proficiency level

25. UISFL GPRA MEASURESThe Undergraduate International Studies and Foreign Language (UISFL) Program provides funds to institutions of higher education, a combination of such institutions, or partnerships between nonprofit educational organizations and institutions of higher education to plan, develop, and carry out programs to strengthen and improve undergraduate instruction in international studies and foreign languages. • Percentage of UISFL projects that added or enhanced courses in international studies in critical world areas and priority foreign languages as reflected in the course syllabi.(Short-Term). • Percentage of UISFL consortium projects that established certificates and/or undergraduate degree programs in international or foreign language studies.(Long-Tem)

26. DDRA GPRA MEASURESThe Fulbright-Hays Doctoral Dissertation Research Abroad (DDRA) Program provides grants to colleges and universities to fund individual doctoral students to conduct research in other countries in modern foreign languages and area studies for periods of 6 to 12 months. • Percentage of DDRA fellows who increased their foreign language scores in speaking, reading, and/or writing by at least one ILR equivalent proficiency level. • Percentage of Doctoral Dissertation Research Abroad (DDRA) fellows who complete their degree in their program of study within 8 years of receipt of the fellowship • Percentage of DDRA fellows who found employment that utilized their language and area skills within 8 years of receiving their award • Efficiency Measure: The cost per DDRA fellow who found employment that utilized their language and area studies skills within 8 years.