Changes That Occur Abroad: Measuring the Impact of Our Programs

1. Changes That Occur Abroad: Measuring the Impact of Our Programs Richard C. Sutton, Ph.D. University System of Georgia Board of Regents Michael Vande Berg, Ph.D. Council for International Educational Exchange

2. Not quite the session described in your conference program… • Information on the Texas A&M research project will not be presented today • Georgia GLOSSARI project overview/update • Special guest presentation on related research projects

3. The GLOSSARI Project: Assessing Learning Outcomes of Study Abroad Richard C. Sutton University System of Georgia Board of Regents NAFSA Annual Conference Minneapolis, MN 30 May 2007

4. Significance of Study Abroad in Higher Education • 200,000+ U.S. students currently go abroad each year • About 8% of all UG degree recipients have part of their education abroad • Students can earn up to one-fourth of their academic degree from overseas study • Study abroad participation among all U.S. students increased 20% since 2001. • Georgia participation will triple within 10 years (1997-2007) • New federal funding initiatives to increase study abroad

5. Growth of USG Study Abroad Participation 15% average annual growth rate

6. Assessment Efforts in Study Abroad • Strong research efforts to assess second language acquisition learning outcomes from study abroad • Some very good studies on documenting gains in intercultural sensitivity and personal growth • Limited attention paid to knowledge and skills acquired abroad • Increased public scrutiny of SA investment requires rigorous investigation of learning outcomes attributable to intentional design of programs

7. GLOSSARI • GEORGIA • LEARNING • OUTCOMESOF • STUDENTS • STUDYING • ABROAD • RESEARCH • INITIATIVE

8. University System of Georgia • Large, diverse public higher education system • 260,000 students at 35 institutions • 4 research universities, 15 comprehensives (including 3 HBCUs), 16 two-year/four-year schools • 350+ study abroad programs of all types (exchange/immersion, faculty-led, short-term, etc.) • System-level International Education Office • System-level Institutional Research (IR) Office • Regents strategic mandate to increase study abroad since 1995 • Began detailed data collection on USG students abroad in 1999 • Began GLOSSARI research project in 2001 Pretty good laboratory for conducting research on student learning & study abroad

9. Primary Objectives of GLOSSARI • Identify cognitive learning outcomes attributable to • diverse study abroad experiences • for students at a wide variety of public institutions • Identify impact on academic performance indicators • Identify impact of study abroad on core liberal arts aspirations (critical thinking, leadership, adaptability, etc.) • Identify program characteristics that optimize learning outcomes to guide future program development • Identify student characteristics that predict • likely participants • successful participants • Refine, replicate, and disseminate methods for assessing the impact of study abroad on student learning outcomes.

10. New Federal Grant to Accelerate the GLOSSARI Research Project • GLOSSARI project began in 2001 with modest internal funding • U.S. Department of Education: International Research & Studies Program Grant for 2006-2009 • GLOSSARI research team headed by Prof. Don Rubin (University of Georgia) • Six components examine discipline-specific and cross-disciplinary learning outcomes

11. The Six Phases of GLOSSARI Phase I: Learning Outcomes of SA Participants & Non-Participants Phase II: Pre- and Post-participation Learning Outcomes with multiple measures Phase III: Teaching the same course content abroad & at home Phase IV: Academic performance measures among SA participants and non-participants Phase V: Program design features that make a difference Phase VI: Impact 2- to 5-year post-graduation

12. What we’ve learned so far…. • Better navigational skills • Improved academic performance upon return • Much higher persistence and graduation rates

13. Phase I: Learning Outcomes of SA Participants & Non-Participants a) self-reported learning outcomes (ILO) • Initially conducted with 250 SA students and 250 non-SA students (random sampling) • Now being conducted with 500 SA students and 500 non-SA students (representative, proportional) • More rigorous control of test timing and other variables to reduce internal & external threats to validity • skills and knowledge testing component added for second cohort (ILO + IST) • Not just what they say they know, but what they actually know

14. Learning Outcomes Factor 1Knowledge of Verbal Resources • Knows how to adjust to new situations • Knows when to take risks • Knows how to talk one’s way out of difficult situations • Knows how to lead discussions

15. Comparison between study-abroad participants and nonparticipants on knowledge of verbal resources(not statistically significant; p>.05)

16. Learning Outcomes Factor 2 Sensitivity to Cultural Context • Knows how to interact in different culture • Knows the importance of withholding judgment • Sensitive to language and culture differences • Sensitive to one’s own reactions to others

17. Comparison between study-abroad participants and nonparticipants on sensitivity to cultural context(p<.001)

18. Leaning Outcomes Factor 3 Knowledge of Self as Cultural Being • Understands how settings affect one’s own style • Understands how one’s self is viewed by others • Knows how to define the term “culture”

19. Comparison between study-abroad participants and nonparticipants on knowledge of the self as a cultural being (p<.001)

20. Learning Outcomes Factor 4Functional Knowledge of Cultural Practices • Knows how to compare and contrast culture • Knows norms and taboos • Knows how to locate information in a newspaper • Knows how to buy toothpaste and can opener • Knows how to give coherent directions

21. Learning Outcomes Factor 4Functional Knowledge of Cultural Practices (continued) • Knows different ways to express ideas • Knows what’s funny • Knows how to take a train or a bus in a foreign country • Knows how to use a public telephone • Can locate safe clubs or bars • Knows how to pacify an angry person

22. Comparison between study-abroad participants and nonparticipants on functional knowledge of cultural practices (p<.001)

23. Learning Outcomes Factor 5 Knowledge of World Geography • Can name six countries in Africa • Can name four South American capitals • Can name four rivers in Europe

24. Comparison between study-abroad participants and nonparticipants on knowledge on world geography(p<.001)

25. Learning Outcomes Factor 6 Knowledge of Global Interdependence • Understands how freedoms in U.S. compare with other nations • Can explain presence of U.S.troops abroad • Can explain aspects of U.S. foreign policy • Understands how foreign markets might affect one’s own career • Understands how foreign manufacturing affects pricing in U.S.

26. Comparison between study-abroad participants and nonparticipants on knowledge of global interdependence (p<.005)

27. Phase I Replication/Refinement: • Preliminary analysis of replication data (still in process of collection) shows no surprises or significant changes from previous results: • Change in sample population (from random to scientifically selected) improves methodology, but doesn’t appear to change results • Apparent correlation between self-reported and independently tested knowledge (i.e., students are pretty honest in answering survey questions)

28. Stage II: Pre- and Post-Participation Learning Outcomes • with modified ILO survey • 100 study abroad students • Minor changes in survey questions • correlated with other testing instruments • 200 study abroad students are taking 5 concurrent assessments (both pre-test and post-test) • Intercultural Learning Outcomes (ILO) • Independent Skills Test (IST) • Intercultural Development Inventory (IDI) • Cross-Cultural Adaptability Inventory (CCAI) • California Critical Thinking Skills Test (CCTST)

29. Expectations for Phase II: • How do different measures commonly used in study abroad assessment correlate with each other? • How do sub-measures within each instrument correlate with each other? • How do SA assessment instruments (IDI, CCAI, ILO) correlate to other instruments (IST, CCTST) that seek to measure related skills/knowledge/attitudes?

30. Phase III: Teaching the same course content abroad & at home • 10-20 courses taught on-campus and abroad (faculty-led programs) • 600 students (half SA, half domestic) • Analyze student learning artifacts (exams, papers, projects, journals, etc.) from both environments • Qualitative complement to quantitative data—what is the value added to student learning by teaching a course abroad vs. at home?

31. Phase IV: Academic performance measures (graduation and persistence rates) OIE Databases Provides: • 25,000 individual study abroad records (location, duration, class level & major at time of SA, etc.) from 35 USG institutions • Program catalog database USG Databases provide: • Age, gender, race, etc. • Matriculation/graduation • High school GPA, SAT • USG semester GPA • Transfers w/in USG Takes advantage of USG’s unique ability to merge OIE study abroad databases with System-wide student records databases

32. Persistence Toward Graduation at Test Sample of Three USG State Universities Blue column includes 2000-2001 study abroad students from three universities who had graduated or were still enrolled in Spring 2003. Red column represents average graduation rate at the three institutions.

33. U of Minn Cohort Study of 1999 entering class showed Improved Graduation Rates… University of Minnesota study of 1999 entering freshman class.

34. …especially among lower-aptitude students (ACT/SAT + HSGPA index)… University of Minnesota study of 1999 entering freshman class (cohorts with low admissions index; AAR = 100-119)

35. …and among students of color University of Minnesota study of 1999 entering freshman class. University of Minnesota study of 1999 entering freshman class.

36. USG Initial Findings (& the perils of collaborative research!) • Phase IV designed to conduct factor analysis of multiple variables affecting graduation rates (including SA)—does SA have statistical significance in graduation and persistence of undergraduate students? • If so, do different elements of program design (duration, location, etc.) have greater/lesser impact? • Extraction of relevant info from USG databases was delayed, tautological • Will re-execute data request this summer to construct valid analysis (in the meantime working with the numbers you have, not the numbers you wish you had) • Still, have found some evidence that corroborates U of Minn correlations

37. USG Graduation Data, SAT, & SA Among students who studied abroad and graduated, some SAT variance in 4-year rates but almost none in 6-year rates. SAT VerbalIn 4 yrsIn 6 yrs 200-550 61.8% 95.9% 551-650 71.4% 98.6% 651-800 75.8% 99.3% Virtually identical results for SAT Math cohorts.

38. 4-Year Graduation, SAT, & Program Duration Among students who studied abroad and graduated in 4 years, higher rates for those on short-term programs (<8 wks) consistently across SAT cohorts. SAT VerbalTotal <8 wks>8 wks 200-550 61.8% 66.0% 56.8% 551-650 71.4% 77.5% 65.8% 651-800 75.8% 83.6% 69.0% Could suggest: • spending a semester+ abroad complicates credit accumulation and delays graduation; OR • shorter (mainly summer) programs don’t interrupt other studies and don’t interfere with time to degree; OR • both; OR • other

39. 4-Year Graduation, SAT, & Degree Year of SA Among students who studied abroad and graduated in 4 years, rates are significantly higher for those who studied abroad in freshmen and sophomore years. SAT VerbalTotalFrSophJuniorSenior 200-550 61.8% 73.7% 71.8% 45.0% 52.5% 551-650 71.4% 83.5% 78.1% 53.0% 61.0% 651-800 75.8% 84.8% 84.4% 52.4% 64.2% Could suggest: • students who wait until Junior year to study abroad don’t catch up with remaining academic courses/graduation requirements; OR • students who study abroad early in their college careers become more focused and disciplined in completing remainder of their studies; OR • both; OR • other

40. Graduation + SA x Gender/ethnicity • 4-year rates significantly higher for women than for men (77% vs. 62%); no gender variance in 6-yr rates. • 4-year rates slightly higher for students of color (73%) than for white students (70%); no ethnic/racial variance in 6-yr rates

41. Phase V: Program design features that make a difference • Identify high-performance SA programs correlated to academic performance measures and learning outcomes • Trying to retrofit Engle & Engle taxonomy of program design elements into OIE database • Conduct case studies of exemplary programs

42. Phase VI: Impact 2- to 5-year post-graduation • Since GLOSSARI began, two major studies have been launched to look at medium- and long-term impact of study abroad on career development, life choices • U of Minnesota alumni study (20 years back) • CIEE longitudinal study (20 years forward) • USG will likely re-direct grant resources into the previous five phases

43. Next Steps, Questions, & Challenges: • Still a lot of work to do in each phase • Invite New State/Institutional Partnerships to Replicate Assessments

44. For Further Information: • On-going reports on this project will be posted periodically on our website: www.usg.edu/oie • Or contact us at: Office of International Education University System of Georgia Board of Regents 270 Washington Street SW Atlanta, GA 30334 Tel: 404/651-2950 Richard.sutton@usg.edu or Reliang.Tsang@usg.edu