Lessons Learned from Years of Administering a Multi-Institution Online Alumni Survey

1. Lessons Learned from Years of Administering a Multi-Institution Online Alumni Survey American College Personnel Association March 2013 Amber D. Lambert, Ph.D. Angie L. Miller, Ph.D. Center for Postsecondary Research, Indiana University

2. Presentation Outline • Literature Review: Importance of alumni assessment and survey issues • Lessons from the Strategic National Arts Alumni Project (SNAAP) • Survey administration challenges • How schools are implementing survey results

3. Literature Review • As funding to higher education institutions continues to be cut, colleges and universities are often required to show measures of their effectiveness (Kuh & Ewell, 2010) • Surveys are used in many areas of higher education (Kuh & Ikenberry, 2009; Porter, 2004) • Alumni surveys can provide valuable information on student satisfaction, acquired skills, strengths and weaknesses of the institution, and current career attainment

4. Literature Review • A major concern with all surveys, and alumni surveys in particular, is low response rates • Over the last decade survey response rates have been falling (Atrostic, Bates, Burt, & Silberstein, 2001; Porter, 2004) • Alumni surveys often have lower response rates than other types of surveys (Smith & Bers, 1987) due to: • Bad contact information • Suspicion of money solicitation • Decreased loyalty after graduation

5. Lessons Learned from the Strategic National Arts Alumni Project (SNAAP)

6. SNAAP • As an example, we will discuss some best practices for survey administration and share results from the Strategic National Arts Alumni Project (SNAAP) • What is SNAAP? • On-line annual survey designed to assess and improve various aspects of arts-school education • Investigates the educational experiences and career paths of arts graduates nationally • Findings are provided to educators, policymakers, and philanthropic organizations to improve arts training, inform cultural policy, and support artists

7. Who does SNAAP survey? • Participants drawn from: • Arts high schools • Independent arts colleges • Arts schools, departments, or programs in comprehensive colleges/universities • Over 5 years, SNAAP has been administered at nearly 300 institutions of various focuses, sizes, and other institutional characteristics • Cohort Year Sampling • 2008 and 2009 Field Tests: 5, 10, 15, & 20 years out • 2010 Field Test: 1-5, 10, 15, & 20 years out • 2011 and forward: all years to generate the most comprehensive data possible

8. Increasing Numbers… • 2010 Field Test • Over 13,000 respondents • 154 Institutions • 2011 Administration • More than 36,000 respondents • 66 institutions • 2012 Administration • More than 33,000 respondents • 70 institutions • Now able to combine 2011 and 2012 respondents to create a “SNAAP Database” with over 68,000 respondents

9. Questionnaire Topics • Formal education and degrees • Institutional experience and satisfaction • Postgraduate resources for artists • Career • Arts engagement • Income and debt • Demographics

10. Survey Administration Challenges

11. Survey Administration Challenges: Locating the Lost • Important that contact information is accurate and up-to-date • Encourage proactive efforts • Newsletters • Websites, social networking • Alumni tracking • Contracted with Harris Connect, a direct marketing firm

12. Survey Administration Challenges: Response Rates • Response rates are directly related to the accuracy of contact information • Incentives: only minimally effective • “Open enrollment” features can increase number of responses • Social networking sites • Need to verify respondents

13. Survey Administration Challenges: Response Rates • Email invitations to participate in the survey • Is it better to have HTML or plain text? • For the 2011 administration, we created visually appealing email invitations in HTML format

14. Survey Administration Challenges: Response Rates • For the 2012 administration, we systematically compared the effectiveness of HTML invites to plain text invites across the 5 email contacts sent to participants • Results of this experiment suggested that a combination of message types gets the highest response rates • Plain text was more effective for the initial contact • HTML was more effective for follow-up contacts • Potential reasons: plain text may reach larger numbers, but HTML may give the project legitimacy

15. Survey Administration Challenges: Response Rates • Long-term strategies can also influence the tendency of alumni to respond • Consider building recognition of the alumni survey presence while they are still students, so there is familiarity with the project once contacted to participate as alumni • Sharing data on campus • Involving students in campus or curricular changes made based on survey results • Connecting alumni surveys with senior exit surveys (SNAAP plans to test this model with select institutions in the 2014 administration)

16. Implementing Survey Results

17. Using SNAAP for Curricular Assessment

18. Using SNAAP for Curricular Assessment (cont.)

19. Using SNAAP for Curricular Assessment (cont.) • Most important skills: • Creative thinking and problem solving • Listening and revising • Interpersonal relations and working collaboratively • Broad knowledge and education • Critical thinking and analysis of arguments and information • Recommend that faculty: • Incorporate open-ended projects (top skill #1) and group projects (top skill #3) • Require analysis of theories or reviews/critiques (top skill #5) and provide opportunities for feedback and revision (top skill #2) • Ensure curricula include a firm knowledge foundation in a wide variety of areas (top skill #4)

20. Using SNAAP for Curricular Assessment (cont.)

21. Using SNAAP for Curricular Assessment (cont.)

22. Using SNAAP for Curricular Assessment (cont.) • Can identify strengths: • What skills and competencies have the highest percentages of alumni reporting the institution helped them develop “very much” or “quite a bit”? • Can identify areas for improvement: • What skills and competencies have the highest percentages of alumni reporting the institution helped them develop “very little” or “not at all”? • Peer group information provides context: • Do other institutions have similar strengths and weaknesses?

23. Examples From 2011 Aggregate Findings

24. Using SNAAP for Curricular Assessment (cont.) • Alumni receive strong training in learning artistic techniques • Discrepancies between those who say a skill is important for their work and those who say the institution helped them develop that skill suggest some improvements that could be made, such as: • Requiring business and financial classes, or incorporating these elements into existing courses • Include classes looking at the “nontraditional” career paths of arts graduates

25. Using SNAAP for Program Assessment

26. Using SNAAP for Program Assessment (cont.)

27. Using SNAAP for Program Assessment (cont.) • Programs and services with low satisfaction may need to be revised • Career advising had 59% report either “very dissatisfied” or “somewhat dissatisfied” • Additional resources could be devoted to developing new components of career advising such as: • Alumni career panel presentations • Résumé or portfolio building sessions • Networking opportunities for graduating students

28. Examples: Sharing on Campus

34. Examples: Alumni & Donor Outreach

43. Examples: Recruitment

47. Conclusions • Assessing alumni can provide important information on institutional effectiveness, but alumni surveys can pose several obstacles • When administering alumni surveys, some steps can be taken to update contact information and increase response rates • The results from alumni surveys can be useful in multiple areas, including curricular and program assessment, campus information sharing, alumni and donor outreach, and recruitment

48. Questions or Comments? • Contact Information: • Amber D. Lambert adlamber@indiana.edu • Angie L. Miller anglmill@indiana.edu Strategic National Arts Alumni Project (SNAAP) www.snaap.indiana.edu (812) 856-5824 snaap@indiana.edu

49. References Atrostic, B. K., Bates, N., Burt, G., & Silberstein, A. (2001). Nonresponse in U.S. government household surveys: Consistent measure, recent trends, and new insights. Journal of Official Statistics, 17(2), 209- 226. Kuh, G. D. & Ewell, P. T. (2010). The state of learning outcomes assessment in the United States. Higher Education Management and Policy, 22(1), 1-20. Kuh, G. D. & Ikenberry, S. O. (2009). More than you think, less than we need: Learning outcomes assessment in American higher education, Urbana, IL: University of Illinois and Indiana University, National Institute of Learning Outcomes Assessment. Porter, S.R. (2004). Raising response rates: What works? New Directions for Institutional Research, 121, 5-21. Smith, K., & Bers, T. (1987). Improving alumni survey response rates: An experiment and cost-benefit analysis. Research in Higher Education, 27(3), 218-225.