Assessment Toolbox for International Educators

1. Assessment Toolbox for International Educators Presenters: David Comp, Darla K. Deardorff, Elaine Meyer-Lee, Lee Sternberg, Victor Savicki

2. Session Overview • Assessment Overview and Context • Highlights of Specific Assessment Tools/Methods • Tools/Methods Exploration through “Roundtable” discussions

3. Assessing International Education: An Overview Dr. Darla K. Deardorff Duke University d.deardorff@duke.edu

4. Why assess international education? • Inform continuous quality improvement • Advocate for international education • Satisfy regional accrediting bodies • Add to basic understanding of student growth in this area • Provide feedback to students on their personal growth

5. Why Assess? Measure success Moving beyond OUTPUTS to OUTCOMES What are meaningful outcomes (results) of internationalization efforts?

6. ____________________________________________________________________________________________________________________________________________________ INTERNATIONALIZATION at institutions of higher education Inputs/Resources needed for implementation of components of internationalization | Activities/Components of Internationalization (college leadership, faculty involvement, curriculum, study abroad, international students/scholars/faculty, international co-curricular units) | Outputs of Internationalization (i.e., number of international students, number of study abroad programs, number of students studying foreign languages, etc.) | Outcomes of Internationalization Intercultural competence – what is it? How do higher education administrators define it? intercultural experts? How can it be assessed? = Long-Term Impact of Internationalization _______________________________________________________________ Figure 2. General program logic model applied to internationalization. (Deardorff, 2004, Page 58)

7. Ways of approaching assessment Some key points…

8. ASSESSING OUTCOMES Starting point: Mission Statement into Goals into Measurable Objectives

9. Assessment-Contexts • “At Home” – curriculum, extra-curricular activities, community involvement/impact, domestic/internat’l student interaction, policy, financial, campus climate • Abroad(Cross-border) – involves cross-border delivery of education through exchange, distance and e-learning, branch campuses, partnerships, host community impact

10. Assessment Cycle • Define outcomes (based on mission/goals) and establish measurable criteria • Identify appropriate assessment methods • Collect data • Analyze data • Reflect on needed changes • Design and apply changes

11. Assessment – Key Points • Multi-method, multi-perspective • Ongoing • Integrated

12. Student Involvement in Assessment Student outcome assessment is not a measurement of learning but an integral part of learning. Can include language/disciplinary assessment, personal growth, intercultural competence, etc. Involve students in negotiating outcomes

13. Assessment Methods • Methods determined by clearly articulated assessment questions – what do we want to measure? Include: • DIRECT METHODS – authentic assessment, portfolios, observation • INDIRECT METHODS – surveys, focus groups, interviews • COMBINATION OF QUANTITATIVE AND QUALITATIVE

14. DIRECT METHODS • Embedded course assessment • Portfolios • Performance • Testing • Papers/projects • Capstone

15. Indirect Methods • Surveys (inc. self-assessments) • Interviews • Focus groups • Curriculum/transcript analysis • Documented data

16. ICC Assessment Tools (85+!) • Intercultural Development Inventory • Cross Cultural Adaptability Inventory • Intercultural Conflict Styles Inventory • Language Strategies Survey • Strategies Inventory for Culture Learning • Beliefs, Events, Values Inventory • GAP Test • Assessment of Intercultural Competence

17. Questions to consider… • What are the specific goals and objectives? • What does the tool/method specifically measure? • Limitations and cultural biases of tools/methods?

18. More questions to consider… • How will assessment data be utilized? How will data be used to provide feedback to students? To improve the program/curriculum, etc? • Is assessment multi-method, multi-perspective, ongoing, intentional, integrated, part of strategic plan?

19. OVERVIEW OF ASSESSMENT TOOLS GAP, American Identity, ICAPS, BEVI

20. Assessment Toolbox for International Educators: GAP David Comp The University of Chicago dcomp@uchicago.edu NAFSA: Association of International Educators Annual Conference Minneapolis, Minnesota ~ June 1, 2007

21. Global Awareness Profile (GAP) Developed by J. Nathan Corbitt in 1998. • J. Nathan Corbitt is Professor of Cross-Cultural Studies at Eastern University in St. Davids, Pennsylvania and President of BuildaBridge International, a cross-cultural non-profit arts education organization. J.N. Corbitt, Global Awareness Profile. (Yarmouth, ME: Intercultural Press, 1998).

22. Global Awareness Profile (GAP) • Definition of Global Awareness • Global Awareness involves a recognition and appreciation of the size, complexity, and diversity of the earth conceived as a single entity. It is literally the recognition of a worldview. It enables us to perceive the vastness of the world, its dynamic complexity, and the diversity of its peoples, cultures, and environment. - J.N. Corbitt, http://www.globalawarenessprofile.com/

23. Global Awareness Profile (GAP) • Structure of the Global Awareness Profile (GAP) • The GAP test is a self-scoring inventory that gives participants a graphic representation of their global awareness. It presents 120 questions based on common knowledge in six geographic regions (Asia, Africa, North America, South America, the Middle East and Europe) and six subject areas (environment, politics, geography, religion, socioeconomics and culture), along with twelve questions about broad global issues. After scoring their tests, participants transfer their scores to profiles that allow them to see the areas in which they are strongest and weakest. http://www.globalawarenessprofile.com/

24. Global Awareness Profile (GAP) • Who uses the GAP test? • Universities measuring global awareness outcomes • Cross-cultural trainers preparing trainees for overseas service of multi-cultural contexts • Educators teaching complex thinking about world issues and general wariness of the world • Business persons who wish to develop their global competence or intelligence • Individuals for personal self-awareness http://www.globalawarenessprofile.com/

25. Global Awareness Profile (GAP) • Validity and Reliability • University professors and international students were used in researching the GAP test by asking a single question: What should a person in your discipline or country know as common knowledge. Designed from hundreds of questions, and updated in 2005 by graduate students from St. Joseph’s University, the inventory, was tested using face and content evaluation, and retesting to validate the instrument and check for reliability. • Testing is ongoing in a number of high schools and universities with both undergraduate and graduate students to establish median scores. http://www.globalawarenessprofile.com/ • According to Paige & Stallman, “one study by Corbit reports a satisfactory test-retest reliability coefficient (.83). This study provides evidence of the instrument’s validity by showing that the GAP was able to discriminate between individuals who had and had not studied abroad.” A Guide to Outcomes Assessment in Education Abroad. (2007). Edited by Mell Bolen. The Forum on Education Abroad

26. New Tools for Intercultural Learning Outcomes Assessment Elaine Meyer-Lee Director Joy Evans Assistant Director for Research and Scholarship Saint Mary’s College Center for Women’s InterCultural Leadership Notre Dame, Indiana

27. For five years we conducted a comprehensive study of our intercultural and international learning on and off campus. For study abroad we used a multi-method longitudinal assessment of intercultural engagement, IC sensitivity, identity, and goals

28. Evaluation & Assessment Goals • Evaluate programs’ effectiveness • Measure Students’: • Changes in sensitivity to cultural differences • American identity development • Own sense of growth toward goals • Explore correlations of change with: • Intercultural engagement while abroad (interaction and reflection) • Program or demographic characteristics

29. American Identity Measure • Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good track record), and her much newer American Identity Questionnaire developed for underrepresented groups • Based on Eriksonian identity development theory. Measures 2 Factors: • identity search (e.g. I have often talked to other people about what it means to be an American.) • affirmation, belonging, and commitment (e.g. Being American plays an important part in my life.)

30. American Identity Measure • Adapted it (with Phinney’s blessing) for study abroad students because one goal is more complexity in understanding one’s own culture • Students in returnee courses felt this national identity version captured an important dimension • 10-item Likert-style Self Assessment (5 for each) • Currently refining adaptation’s psychometric properties, but alpha coefficients good so far (.79+)

31. Sources • Phinney, J. (1992). The Multigroup Ethnic Identity Measure: A new scale for use with adolescents and young adults from diverse groups. Journal of Adolescent Research, 7, 156-176. • Phinney, J., & Devich-Navarro, M. (1997). Variations in bicultural identification among African American and Mexican American adolescents. Journal of Research on Adolescence, 7, 3-32.

32. Measure of Intercultural Engagement Asks students to quantify the frequency of their engagement through: • Interaction with cultural resources • Interaction with local people • Explicit reflection on these interactions

33. MEI Comparable & program specific data on students’ intercultural interactions AIQ Examine identity dimension, connect to literature on this MEI Labor-intensive development and administration AIQ Reliability and Validity of adaptation still being established Needs refining Benefits to Study Abroad Assessment & Challenges to Institutional Research

34. Intercultural Leadership Certificate Program • Capture Saint Mary’s strengths and CWIL’s unique location at the intersection of intercultural (local and global) and leadership education in the context of a women’s college • Guide students through a developmental process into a deeper capacity for intercultural leadership • Bring coherence and provide an organizing framework for every effort (curricular and co-curricular) being undertaken at Saint Mary’s College to encourage intercultural leadership • Portfolio assessment of students’ proficiency as an intercultural leader in 6 proficiency areas • Recognize the Leader Within • Articulate Your Ethical/Spiritual Center • Engage With & Value Diversity • Dialogue on Power & Privilege • Create Inclusive & Equitable Community • Make Your Difference in the World

35. For more handouts or bibliography on assessing intercultural outcomes of study abroad programs, go to http://www.saintmarys.edu/~cwil/php/intercultural.learning/IILOutcomes.php or email me:meyerlee@saintmarys.edu

36. Assessment Toolbox for International Educators: ICAPS Victor Savicki Western Oregon University

37. Intercultural Adjustment Potential Scale (ICAPS) • Developed by David Matsumoto and associates to predict adjustment to a foreign culture • Matsumoto, D., Leroux, J. A., Ratzlaff, C., Tatani, H., Uchida, H., Kim, C., & Araki, S. (2001). Development and validation of a measure of intercultural adjustment potential in Japanese sojourners: The Intercultural Adjustment Potential Scale (ICAPS). International Journal of Intercultural Relations, 1-28. • Many subsequent supporting studies

38. Intercultural adjustment potential definition • “the ability to adapt successfully to life in a cultural environment different than that which one is accustomed to” • “The ICAPS-55 is the only scale available today that can reliably and validly predict the degree to which a person will successfully adjust to living, working, and playing effectively in a new and different cultural environment.” • Applied to both foreign nationals coming to the U.S and U.S. nationals going abroad

39. ICAPS Scales • ICAPS Total: overall potential for successful intercultural adjustment • Emotion Regulation: The ability to monitor and manage one’s emotional experiences and expressions, and to channel their energies in constructive ways • Openness: The ability to encounter new experiences, emotions • Flexibility: The ability to assimilate new experiences, schemas, and ways of thinking into one’s own • Critical Thinking: The ability to think outside the box in creative and autonomous ways

40. ICAPS Structure • 55 items • Responses given on a scale ranging from 1 (Strongly Disagree) to 7 (Strongly Agree) • Takes about 10-15 minutes to complete • Scores reported as T-scores (mean=50, SD=10) • Scale and scoring through David Matsumoto • www.davidmatsumoto.info/Research_Tools/icaps.htm

41. Advantages and Disadvantages • Predicts intercultural adjustment over months • Gives several scores to interpret • Highlights individual’s strengths and weaknesses • Is sensitive to training and education • Negligible face validity (I like haiku poems.) • Is sensitive to timing of administration • No self-scoring yet..

42. The Beliefs, Events, and Values Inventory (BEVI) Developed by Craig N. Shealy, Ph.D. James Madison University shealycn@jmu.edu http://www.jmu.edu/ibavi

43. The Beliefs, Events, and Values Inventory (BEVI) What does the BEVI measure? The BEVI…is designed to assess a number of processes relevant to international experiences including (but not limited to): basic openness; receptivity to different cultures, religions, and social practices; the tendency (or not) to stereotype in particular ways; self and emotional awareness; and preferred but implicit strategies for making sense of why ‘other’ people and cultures ‘do what they do’ (from Shealy, 2005)

44. The BEVI: • Has been in development since the 1990s • Is derived from belief-value statements • Is comprised of three validity and ten process scales • Does not appraise “correctness” of individual items but an overall response pattern • Is valid and reliable over time

45. The BEVI: • • Has 54 demographic items including age, gender, ethnic background, educational attainment, political orientation, and religious affiliation • • Has 391 items across a wide set of beliefs and values rated on a four-point Likert scale • • Takes 45-55 minutes to complete • • Is available online

46. BEVI items include: I am never sad. I have felt jealousy toward someone I loved. There is no such thing as destiny. You can't argue with fate. We should do more to help minority groups in our culture. Big corporations often harm the earth. I know that evil people go to hell when they die. Hinduism and Buddhism have much to teach the modern world.

47. The BEVI: • • Is effective in showing how people’s beliefs and values change as a result of having an international experience, including the way people feel about their own cultures, the value of learning about other cultures, the importance of learning a second language, the value in learning about world events, and other attitudes. • • Measures complex phenomena and is not face-valid • • Administration time can be long for some

49. Your Turn…! Join us for “Roundtable Discussions” on the specific tools you’ve just heard about… ask more questions, learn more in detail about how and when to use these instruments