Evaluation NRC, FLAS, and LRC Project Directors’ Meeting September 14, 2010 Jonathan Plucker, Ph.D. Indiana Universit

1. Evaluation NRC, FLAS, and LRC Project Directors’ Meeting September 14, 2010 Jonathan Plucker, Ph.D. Indiana University

2. Presentation overview • Why Evaluation? • Characteristics of High Quality Evaluations • Future Directions for NRC, FLAS, and LRC Evaluations

3. Why Evaluation? • High quality evaluation … • makes it easier for you to measure your progress • allows you to report progress easily and quantitatively • allows ED staff to gather evidence of program effectiveness • it just makes your life a heck of a lot easier (see the following slides).

4. Characteristics of HighQuality Evaluations

5. An effective evaluation has… • … a good logic model • … good objectives and performance measures • … an efficient structure • … its place in a larger campus evaluation context

6. Logic Models They really are important.

7. What is a Logic Model? • Simplified picture of a program, initiative, or intervention. • Shows logical relationships among the resources that are invested, the activities that take place, and the benefits or changes that result. • (program theory or the program's theory of action) • It is a "plausible, sensible model of how a program is supposed to work" (Bickman, 1987).

8. Sample Logic Model INPUTS OUTPUTS OUTCOMES Program Investments Activities Participation Short Term Intermediate Long Term (Impacts) What is invested What we do Who we reach Learning Action / Performance Conditions • Inputs - the resources invested that allow us to achieve the desired outputs. • Outputs - activities conducted or products created that reach targeted participants or populations. Outputs lead to outcomes. • Outcomes - changes or benefits for individuals, families, groups, businesses, organizations, and communities.

9. Harvard Committee on African Studies

10. Harvard Asia Center

11. Project Objectives What are you trying to accomplish?

12. Goals – Objectives – Measures PROGRAM GOAL Project Objectives:What your project is doing to support the overall program goal (approved in application – cannot be changed) Performance Measures:How you measure your progress toward meeting your objectives (GPRA, Program, Project - Modifiable)

13. High Quality Project Objectives • Relevance • Applicability • Focus • Measurement

14. Relevance of Performance Measures

15. Components of Performance Measures In general terms, a performance measure is a measurable indicator used to determine how well objectives are being met. The following four components are necessary to ensure good performance measures. • What will change • How much change you expect • Who will achieve the change • When the change will take place

16. Performance Measures The Center will increase number of current students, alumni, and FLAS recipients (who) who report that they are proficient in Middle Eastern languages (what) including modern Hebrew, Persian, and Turkish by 5% (how much) annually (when).

17. Performance Measures 50% (how much) of K-12 teachers (who) attending CMES workshops will respond that they had implemented knowledge gained from the activities (what) one year later (when).

18. Performance Measures: Problems No Action Verb “Number of teachers attending workshops.” “Percentage of students achieving language proficiency.”

19. Performance Measures: Problems Not Measurable “Habits of mind: teachers embed habits of mind and intellectual inquiry that span all disciplines.” “Evaluation will gauge content proficiency and project effectiveness.” “Collaborative connections will be maintained.”

20. Activities Are NOT Performance Measures • Activities ≠ Performance Measures • If the best response is “Yes, we did that,” it is almost certainly an activity. • Legacy of the “checklist evaluations” that emerged from a compliance mentality.

21. Efficient Evaluation Designs Are you eliminating redundancy and maximizing “bang for your buck?”

22. What is the Common Survey? • An annual online survey that is administered to all current and past students affiliated with the Area Studies Center including: • Current undergraduate and graduate students • Alumni • FLAS recipients • Study abroad students

23. Purposes of the Common Survey • To measure the impact the center has had on its respondents’ current/future studies &career • Moves the evaluation away from simply measuring satisfaction (formative) to more meaningful, long-term outcome (summative) • To maintain a database & gather data for the center’s IEPS reports including participants’ • current status • employment • use of cultural knowledge/language in their current studies/employment • To measure the aggregate impact that all of the centers located at a university have had • Shows collaboration which is a current invitational priority & strategy to reduce costs

24. The common survey can answer: • To what extent do the current students, alumni, study abroad students, & FLAS recipients report that they • are proficient in [the center’s] languages including priority languages? • are utilizing [the center’s] language(s) (including priority languages) in their current employment or studies? • are using their cultural knowledge & area studies skills in their current employment or studies? • rate that cultural knowledge & area studies skills are important for their current employment or studies?

25. Example of Survey Content • Skip logic allows questions to appear based on respondent’s previous response (i.e., all FLAS recipients will answer questions about FLAS) • Demographics (e.g., how they learned of center, residence, gender, languages studied, employment status, occupation, title) • Please indicate your level of agreement with the following statements (Strongly disagree, Disagree, Moderately disagree, Moderately agree, Agree, Strongly agree). • It was easy to get information about the FLAS Fellowships. • My application was processed and I was told about my FLAS award in a timely fashion. • I have been satisfied by the equal access and treatment of students in the FLAS program. • Participating in the FLAS has better prepared me for my career.

26. Please rate each of the following items:

28. Yet More Examples! • What would you say was your biggest take-away from majoring/minoring and/or taking courses in Area Studies Center languages or studies? • Have you used your knowledge from your language classes in your career beyond your current job? • Please respond with your level of agreement to the following statements (Strongly disagree, Disagree, Agree, Strongly Agree, Not Applicable): • My study abroad experience influenced my choice of major/minor. • My study abroad experience influenced my career choice. • My study abroad experience led me to a specific geographical location after graduation. • My study abroad experience met the expectations I had at the start.

29. Indiana University’s 2009 aggregate common survey results

30. Current Residence

31. Respondent by FLAS Year

32. Current Employment Status • 63% (17) Higher Education • 19% (5) Teach in area studies, • international studies, or foreign language • “French Language Coordinator”  • “Dean for International Programs” • 11% (3) Full-Time Private Sector for profit • “Sr. Financial Analyst - Intel Corporation” • “Producer, WBUR-FM (Boston\'s NPR affiliate)”  • 11% (3) US Governmental Agency • “Budget Analyst, U.S. Marshals Service”  • “Political/Economic Officer, U.S. Embassy Georgetown, Guyana” • 7% (2) Full-time Secondary Education • “Teacher of first and second-year Russian at an urban high school in Chicago.”  • 7% (2) International Position • “Associate Professor, Tokyo, Japan” 

33. Area Studies Impact Respondents were asked if they have used or expect to use their area studies knowledge and skills in any jobs or internships.

34. FLAS Impact Employment and Advanced Degrees 90% (119) of all survey respondents are either currently employed or in full-time graduate or undergraduate programs, compared to 96% (106) of respondents who are also FLAS recipients. FLAS Versus Total Employment /Graduate School

35. Use in Current Job

36. FLAS Outcomes Percent of respondents indicating they currently use these skills gained through FLAS.

37. You Can Still Do Satisfaction!

38. Placing the Evaluation in the Larger Campus Context What other campus resources can be brought to bear?

39. Future Directions • We need a database of best practices in NRC, FLAS, and LRC evaluations. • It would be super cool to have a database of best practices for ALL types of international education, cultural, and language education evaluations. • In the meantime, parsimonious, efficient data collection strategies are probably our best bet to provide meaningful evaluation information to stakeholders.