1 / 42

Assessing Impact: Study Abroad at Two Institutions

Assessing Impact: Study Abroad at Two Institutions. Elaine Meyer-Lee Saint Mary’s College Carol Warfield Auburn University 2006 AIEA Annual Conference February, 2006. Why Assess Outcomes?. 1,022 US institutions reported information on their study abroad programs

adele-puckett
Download Presentation

Assessing Impact: Study Abroad at Two Institutions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Assessing Impact: Study Abroad at Two Institutions Elaine Meyer-Lee Saint Mary’s College Carol Warfield Auburn University 2006 AIEA Annual Conference February, 2006

  2. Why Assess Outcomes? • 1,022 US institutions reported information on their study abroad programs • 174,629 U.S. students studied abroad Data from Open Doors Report on International Educational Exchange 2003-2004

  3. Inform continuous quality improvement • Advocate for international education • Satisfy regional accrediting bodies • Add to basic understanding of student growth in this area

  4. Young, wide-open field • Much need for better methods and more data on many different dimensions • This presentation is not a step-by-step overview, but two examples (See Forum’s upcoming Guide to Outcomes Assessment in Education Abroad)

  5. Multi-method Longitudinal Assessment of Intercultural Engagement, IC Sensitivity, Identity, and Goals: Preliminary Results Elaine Meyer-Lee Director of International & Intercultural Learning Joy Evans Research Analyst-Center for Women’s InterCultural Leadership Saint Mary’s College Notre Dame, Indiana 206 AIEA Annual Conference February 25, 2006

  6. Presentation Overview • Context • Assessment Methods • Review of Instruments • Preliminary Results • Contributions to Study Abroad Assessment

  7. Saint Mary’s College • Catholic Women’s Liberal Art • Established 1844 • About 1600 enrollment • 98% full-time • 82% on-campus • 85% Catholic • 9% students of color 20% of students study abroad

  8. Mission of the Center for Women’s InterCultural Leadership Saint Mary's College's mission includes preparing women to make a difference in an increasingly complex and interdependent world. The role of the Center for Women's InterCultural Leadership (CWIL) within this mission is to foster the intercultural knowledge and competence critical to educating the next generation of women leaders.

  9. CWIL at Saint Mary’s • Encompasses much more than study abroad (see brochures) • Is adopting Byram’s definition of intercultural competence • Is launching a portfolio-based certificate program in Women’s Intercultural Leadership

  10. Year/ Semester Ireland (Year only) India (Semester only) Rome Spain France Australia Austria South Africa Study Abroad Programs

  11. 4-2 Weeks European Studies (4) Internships in Europe (4+) Honduras (3) Ecuador (3-2) Greece (2 ½) S. Korea (2) 1 Week Prague/Poland Jamaica Mexico Study Abroad Programs

  12. Assessing Study Abroad… Going Beyond Student Satisfaction

  13. Evaluation & Assessment Goals • Evaluate programs’ effectiveness • Measure Students’: • Changes in sensitivity to cultural differences • American identity development • Own sense of growth toward goals • Explore correlations of change with: • Intercultural engagement while abroad (interaction and reflection) • Program or demographic characteristics

  14. Assessment Methods/Design • Longitudinal Research Design • Pre-test/Post-test (some grad follow-up) • Procedures – 70% response rate so far • Control Group • 30 First-year students • Focus Group • Study Abroad Re-entry Course • Written reflection and Kolb’s LSI

  15. Study Abroad Participants Intercultural Development Inventory (IDI) American Identity Questionnaire Goals/motivations from Saint Mary’s College Study Abroad Survey Control Group Intercultural Development Inventory (IDI) American Identity Questionnaire Assessment Instruments: Pre-Test

  16. Study Abroad Participants Intercultural Development Inventory (IDI) American Identity Questionnaire (AIQ) SMC Study Abroad Survey Program Impact & Measure of Intercultural Engagement (MIE) Control Group Intercultural Development Inventory (IDI) American Identity Questionnaire Assessment Instruments: Post-Test

  17. Data Collection Timeline

  18. Review of Instruments

  19. Intercultural Development Inventory (IDI) • 50-item Self-Assessment • Valid & Reliable Psychometric Measure • Culture general in focus • Quantifies the subjective experience of cultural difference • Most widely used currently • Based on Milton Bennett’s Developmental Model of Intercultural Sensitivity (constructivist stage model)

  20. Slide Available Online:http://www.coe.int/T/E/North-South_Centre/Programmes/3_Global_Education/c_GENE/GENE%20Web-page%20-%20(3)%20Hesse%20article.asp

  21. IDI Measurement Scales • Denial/Defense • Reversal • Minimization • Acceptance/Adaptation • Encapsulated Marginality

  22. Overall Profile

  23. American Identity Questionnaire • 10-item Likert-style Self Assessment • Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good track record) • Based on Eriksonian identity development theory. Measures 2 Factors: • identity search (e.g. I have often talked to other people about what it means to be an American.) • affirmation, belonging, and commitment (e.g. Being American plays an important part in my life.) • One goal of study abroad is more complexity in understanding own culture, and students in pilot focus group felt it captured an important dimension • Currently refining adaptation’s psychometric properties

  24. Semester/Year-Long Programs Core Survey 111 Items 4 open-ended questions Customized Addendum Summer/Short-term Programs Core Survey 58 Items 1 open-ended questions Customized Addendum Saint Mary’s College Study Abroad Survey

  25. Semester/Year-Long Programs Question Sections Include: Demographics/Background Goals Pre-Departure Academic Program Cultural Tours & Excursions On-Site Student Services Intercultural Engagement Overview of Program Program Impact (Goals) Volunteer Interests Summer/Short-term Programs Question Sections Include: Demographics Pre-Departure Living Arrangements Intercultural Engagement Overview of Program Volunteer Interests Saint Mary’s College Study Abroad Survey

  26. Self-reported Growth in: • Understanding of different cultures • Different perspective on American culture • Greater understanding of the US in world affairs and history • Different perspective on your ethnic heritage • Understanding of my faith • Self-awareness • Maturity and self-confidence • Interpersonal skills and the ability to adapt • Intellectual development in the liberal arts • Ability to communicate in a foreign language

  27. Measure of Intercultural Engagement Asks students to quantify the frequency of their engagement through: • Interaction with cultural resources • Interaction with local people • Explicit reflection on these interactions

  28. Interaction with cultural resources • Theater/opera/symphony/concerts • Museums/ historical sites • Forests/parks/beaches • Traveling outside of the host country • Traveling within the host country

  29. Interaction with local people • Shopping/visiting local markets • Movies/entertainment catering to locals • Restaurants/pubs/cafes catering to locals • Visiting local families/friends in their homes • Participating in worship services • On-campus informal conversations • Events sponsored by the study abroad program • Extra-curricular activities (student clubs, intramurals, etc.) • Volunteering, service-learning, internships, shadowing • Reading or listening to local news

  30. Explicit reflection on these interactions • Journaling/Blogging • Photography • Other artistic expression (painting, drawing, collages, etc.) • Class discussions • Informal conversations with locals • Informal conversations with students studying abroad • Calling/e-mailing family/friends/faculty in America about experience

  31. Pre-test ResultsIDI Worldview Profile: Pre-Abroad

  32. IDI Worldview Profile “Pre”: Controls

  33. Pre-test Comparison to Control: American Identity Questionnaire • Affirmation, belonging and commitment  no significant difference • Identity search  statistical significance (at the .05 level)

  34. Summary of Pre-test Findings • IDI: Students’ scores do not support developmental model • IDI: Low Minimization Score • Catholic beliefs ….We’re all God’s children • Control on both: Heritage Seeker Factor • Are study abroad participants a different population than other SMC students?

  35. Preliminary Analysis of Change Data for Matched Set • No significant change in means for IDI or AIQ yet (on paired T-test on pre-post mean scores, n=42 and 27) • BUT, high self-reported growth (1.7 when 1 is high) and wide range in change scores and program means

  36. What about those who do change? • Correlation of AIQ with IDI scales (bivariate non-parametric correlations) • Correlations with Intercultural Engagement: • Cultural resource scale with AIQ change • Overall and reflection with program impact (especially interpersonal items) • 50% did grow on at least one scale (defined as 2 standard deviations of positive change, n=21)

  37. How do the “growers” compare? Characteristic Growers Non-Growers From N.America 91% 95% White 67% 86% Latina 24% 4.5% Never been abroad 53% 63% Abroad < 3 months 24% 26% Whole year program 10% 46% Semester program 76% 46% Summer program 14% 8% Rome program 24% 32% Seville program 24% 5% Mexico 14% 9% France 14% 9% Ireland 10% 46%

  38. Methodological Findings • Importance and Challenges of Control Group • Procedural effects (e.g. on-line vs. paper) • Signs of internal consistency in measures (inter-item correlations)

  39. Intercultural Development Inventory: Standardized instrument Ability to compare study abroad program outcomes with domestic intercultural programs Intercultural Development Inventory: Requires qualifying training to administer Highly rigorous and proprietary Labor- and cost-intensive for large scale use Benefits to Study Abroad Assessment & Challenges to Institutional Research

  40. Program Impact & MEIM Comparable & program specific data on students’ intercultural interactions AIQ Examine identity dimension, connect to literature on this Program Impact & MEIM Labor-intensive development and administration AIQ Reliability and Validity of adaptation Needs refining Benefits to Study Abroad Assessment & Challenges to Institutional Research

  41. Next Steps • Analysis • More on change and changers with full dataset • More on correlations with intercultural engagement, program variables (including Engle & Engle’s category), and student variables (including original motivation) • Qualitative • Psychometrics on AIQ, Program Impact, MIE • Data collection • Add 1 year follow-up • Draw on Women’s Intercultural Leadership Portfolios? • Applications • Many improvements to (and rationales for) programs already. Overhauling re-entry course.

  42. For more handouts or bibliography on assessing intercultural outcomes of study abroad programs, go to http://www.saintmarys.edu/~cwil/php/intercultural.learning/IILOutcomes.php or email me:meyerlee@saintmarys.edu

More Related