Power up your Program Evaluations

1. Power up your Program Evaluations Marketing, Development, & Assessment

2. Traditional Approach • Evaluation Form (paper or online) • Responses reviewed by SAO • Responses shared with faculty • If paper, some reviews placed in the resource center for prospective students to review • If online (Zoomerang, SurveyMonkey, etc.) reviews may be downloaded & stored. • Other actions taken on a case by case basis.

3. What is evaluation? • Definition: A systematic exploration to discover and judge something that we think has value. • Evaluation is “Ex Valore” in Latin, which means to “strengthen” or “empower” • Ex – “out of, from” • Valor – “strength, value, valor” • It was originally a “means” rather than an “end” • A means to “empowerment” synonymous to positive growth and development • A means that involves people, processes, and results associated with what we do

4. We can evaluate… • Practices - Offices that manage the process, third-party subcontracts, host institution services, information sessions, student advising, pre-departure, policies, procedures, etc. • Programs -Direct enroll, faculty-led, third-party, consortium, student teaching, internships, etc. (their administration, academics, housing, social, expenses, etc.) • Projects -Freshmen orientation, study abroad fair, marketing workshop for faculty, study abroad ambassadors project, welcome back reception, and other special events. to… • Prove (show) something is working or needed • Improve the practice, program, or project • Discover something new for development

5. Power up your evaluations… • Coordinate evaluation process for multidimensional goals • Student Satisfaction Data (regarding Administration, Academics, Housing, Social, Expenses) helps us in marketing and development • Learning outcomes, intercultural development, progress towards global citizenship guides our questions regarding assessment • Shrewdly and efficiently share information with stakeholders.

6. AbroadScout 2.0 Mission: to empower students, schools, and the education abroad industry with quality information and Web 2.0 tools for marketing, development, and assessment.

7. AS 2.0 Marketing, Development, Assessment Tools

8. AS2.0 Funding Model • FREE – All Marketing-Development-Assessment tools. • PRO ACCOUNT – Allows you to upload a logo. Cost $300/yr. • Marketing - It gives your programs more visibility, as they are randomly sorted among all pro-account programs in the first tier of any search (in the main directory and customized program finders). Also, more information is included in the initial search results. • Added Customization - In your Program Finder and Course Finder, it gives your pages a more personalized feel with the logo from your institution on top. • ADVERTISINGPOLICY –To keep information unbiased, and avoid conflicts of interest, the directories (programs, scholarships, and reviews) are completely free of advertising, aside from the pro-account option we offer to all program sponsors equally. • FUNDING MODEL – We fund the site through pro-accounts in the directories and banners-links in continent/country sections of the website. These sections contain articles and news, relevant to student interests, many of which are written by students.

9. Study Abroad @Abilene Christian University • Private Master’s Level University • 4000 Undergraduate and 700 Graduate • 3 Permanent sites, 2 Temp sites, 8 consortium sites • Semester and Summer Programs • Oxford, England • Montevideo, Uruguay • Leipzig, Germany

10. Study Abroad @ Abilene Christian University • Approx. 220 students abroad split evenly between semester-long and summer programs • General Education during semesters • Discipline specific during summers • ACU On-site directors, ACU visiting faculty, local adjuncts and partner universities

11. University Mission and Vision • Increase Student Learning • Increase participation to reach 50% by 2020 (currently 28%) • Expectation that we balance our budget

12. Institutional Drivers of Marketing, Development and AssessmentAbilene Christian University • Accountability to constituents • Students, faculty, parents, donors, board of trustees, home and host communities • Regional Accrediting Body – SACS

13. From Assessment to Marketing • What, How and Why • Global Perspective Inventory • Pre-departure and post completion • Embedded in an orientation course • Course evaluations • Paper and pencil • Online

14. From Assessment to Marketing • Online student program evaluation • Yes/No and open ended-text box questions obtained at the close of the program • Reflection papers embedded in a course • Faculty debriefing

15. Getting from A (Assessment) to M (Marketing) • Internal • Global Perspectives Inventory (GPI) and online program evaluation results to Program Directors, Senior University Administrators, PR/marketing and Board of Trustees (BOT)

16. Getting from Assessment to Marketing – Internal (continued) • Course Evaluations – faculty, Dept. Chairs and College Deans, PR/marketing offices • Student Reflection papers –faculty, PR/marketing offices • Student numbers and demographics - Senior Administrators, BOT – sign leases, purchase property etc…

17. Getting from A to M (continued) • External • Quotes incorporated into Web page articles, ads etc… • Information relayed to current and prospective students and their families via promotional and recruiting materials • Host community is provided feedback – i.e. volunteer organizations, Property/facility owners, neighbors, partner institutions

18. C *e*l*e*b*r*a*t*i*o*n*s • Increased student learning and self-efficacy • Value Added • Holistic education • Increased student enrollment • opportunities for growth • Paying the bills • Increased funding to improve and expand services and sites

19. C-h-a-l-l-e-n-g-e-s • Making every one happy vs. creating a culture of quality enhancement and improvement • Closing the Loop – Follow up assessment to maximize continual improvement • Biting off more than we can chew – trying to do too much

20. Study Abroad at Univ. Delaware • 4-1-4 calendar (5-week January term), abroad and on-campus courses offered • About 1,400 UD students abroad in 2010/11; 75% on January programs (1060 in 2011); 90% on short-term programs (summer and winter programs) • Over 40% of undergraduates (population 15,000) study abroad • About 70 UD programs annually, 50 in January, all faculty led, representing 75% of academic departments • About 20 semester/exchange programs (about 130 students total)

21. Where We Were • Paper evaluation (satisfaction survey) • unwieldy to administer • data hard to retrieve/summarize/distribute • Office of Educational Assessment created 2005 • institutional interest in assessing undergraduate learning • campus resource for expertise • Discovery Learning Experience required for all undergrads as of 2006 • good reason for mandatory assessment • study abroad counts as DLE!

22. Thoughts on How to Move Forward Given that: • the number of students and programs is very large; • students are enrolled in a wide variety of courses; • most are not enrolled in a foreign language course (though many are); • programs are located across the globe in both rural and urban settings; • the range of program conditions is great (housing, mobility, interaction with host culture); and • faculty directors have different program goals: What can we measure that applies to all students? and How can we measure it?

23. What We Do – Data Collection • Look for general impacts (changes from pre to post) in areas of knowledge, skills, and attitudes (applicable to all programs) • Design a short instrument (31 Likert scale items, 2 short answer) • Incorporate into required online pre-departure orientation • Institute a required post-program assessment (linked to University-wide Discovery Learning Experience • Include standard satisfaction survey (subjective feedback on program quality) • Use existing online course evaluation system (via registration for UNIV 370, pass/fail, 0- credit, • study abroad marker)

24. What We Do – Data Organization • Download satisfaction survey feedback; send as pdf to faculty after grades are submitted • Download course feedback; send as pdf to dept. chairs • Program coordinators prepare summaries of each program, addressing budget, faculty organization and performance, incidents abroad, student feedback (numerical ratings, selected quotations), and suggestions for next time • Summaries sent with course syllabi to dept. chair and dean • Assessment results stored and analyzed internally

25. What We Get and Use: Marketing • Largely responsibility of faculty for short-term programs • Responses to short-answer questions can be used in publications

26. What We Get: Development • Masses of data: 98% return rate • Electronic data collection, retrieval, storage using UD’s course evaluation system • Student satisfaction survey results helpful for faculty and program administrators • Used in program summary sent to faculty, chairs, deans • Documentation of problem cases

27. What We Get: Assessment • Quantitative and qualitative pre- and post-sojourn data from about 1400 students on 70 programs annually • 98% return rate • Electronic data collection, retrieval, storage, using UD’s existing online course evaluation system • Ongoing data collection • Improve academic programming and student learning • Friends in Assessment Office • Positive contribution to institutional re-accreditation

28. Limitations/Disadvantages • assessment is self-reported • on-campus control group difficult to maintain • in-house instrument (untested beyond UD) • data backlog (lack of time for analysis) • summaries very time-consuming and labor-intensive to compose and send • not completely objective • impact of summaries unknown (Do chairs/ deans read them?)

29. Questions/Discussion • What kind of assessment or evaluation of education abroad programs is done at your institution? • Who sees the results? • How are the results used? • What prevents results from being used for marketing purposes, for implementing programmatic or procedural changes, or for better understanding student learning?

30. How to Find Us Wendy Williamson Director Office of Study Abroad Eastern Illinois University Charleston, Illinois USA 217-581-3390 wswillliamson@eiu.edu www.eiu.edu/~edabroad Kevin Kehl, Ed.D. Executive Director Center for International Education Abilene Christian University Abilene, Texas, USA 325-674-2710 kehlk@acu.edu www.acu.edu/academics/studyabroad Lisa Chieffo, Ed.D. Associate Director Institute for Global Studies University of Delaware Newark, Delaware, USA 302-831-2852 lchieffo@udel.edu www.udel.edu/international