Our goal for this presentation

1. The FroYo Effect: Using small incentives to achieve a 51% university-wide response rateOffice of Institutional Research and AnalysisDirector, Janel Sutkus, Ph.D.Research Designer and Analyst, Hyun Kyoung Ro, Ph.D.

2. Our goal for this presentation • To describe – in detail – how we administered • a small incentive program for our spring census survey that resulted in a 51% response rate • (a response rate we had not achieved since Spring 2005)

3. Outline • Study background • Response rate history and goal • Survey schedule by cohorts • The big RQ • Survey populations (on-campus and off-campus) • Marketing plan • Administration • Calendar • Invitation and incentive messages • Outcomes • Response rates and odds of responding • Redemption of small incentives • Questions and discussion

4. Study background • The Carnegie Mellon University-Student Experience Study (CMU-SES) is a university-wide study designed to understand: • how well, if at all, we are meeting our educational objectives for students • which experiences, resources, and activities affect those educational outcomes • when they do affect the outcomes, in what direction and magnitude • The CMU-SES was designed to replace the End of Year Survey which was administered to undergraduates in 2000, 2001, 2004, 2005, and 2009. • It will be administered to all CMU students on a three-year cycle, with the next administrations in the spring semesters of 2015 and 2018.

5. The core planning team began meeting on June 1, 2011 with these working principles: • this will not be a revision of past EOY’s, but an entirely new study • the development process will be research question-driven, rather than survey • item-driven, and the instrument will be created hand-in-hand with the analysis • plan (we will hypothesize that certain predictors affect particular outcomes, • and will test those hypotheses using regression methods) • satisfaction is not one of our primary educational objective, so this will not be a satisfaction survey • we cannot study everything we want to know, so we will set our priorities • and study a few things in a thorough manner • We spent ten months creating and testing this survey.

6. Survey and assessment schedule 11-12 to 18-19

7. Response rate history for spring census surveys UG pop randomly split in two

8. The big goal and associated RQ • By using a series of small incentives, can we return our spring census response rate to the levels we had in the mid-2000’s? • We hedged our bets by also providing a large “raffle-type” incentive, but our plan was to do analysis at the end to determine whether small incentives could replace large incentives in the future. • RQ: Are there differences in the odds of responding during a particular small incentive window for students who are eligible for small incentives compared to students who are not eligible for small incentives?

9. Survey population • ON CAMPUS: 5517 students Were eligible for a drawing • for one of two iPad3’s (or $500 at Bookstore) AND • received a small incentive for completing the survey • OFF CAMPUS: 117 students on study abroad were eligible • for a drawing for one of two iPad3’s (or $500 at Bookstore) • LIMITATION AND CAVEAT: Other than the big difference of being off-campus and having study abroad experience, the two groups were much the same in terms of demographics and ultimate survey responses • Off campus students were not aware what they were missing

10. Marketing plan • “Who says? CMU-SES!” branding campaign • Banner in University Center • Table tents at dining locations • Flyers • “Tabling” on the Cut with balloons and branded candy • Core team had CMU-SES t-shirts with iron-on transfers

11. Who says? CMU-SES Student Experience Study

12. Administration calendar

13. Invitation Dear Andrew,We are writing to invite your participation in the Carnegie Mellon University Student Experience Study (CMU-SES). Every three years the university conducts a study of the entire student body to gain insight into the educational experience at CMU. Your input is extremely valuable and should take fewer than 10 minutes using the link below.Students who complete the survey will be placed into a drawing for one of two iPad3’s (the recipients may choose to substitute $500 of merchandise at the University Stores). In addition, all students who complete the survey between now and 10:00 PM on Monday, April 2, will be eligible to pick up a certificate for a free FroYo at Skibo Café.Please click here to participate in the survey: <<link>> Or copy and paste the URL below into your internet browser: <<URL>>This study is sponsored by the Vice Provost for Education and the Dean of Student Affairs and is administered by the Office of Institutional Research and Analysis. Your individual responses will be kept confidential and will only be reported in the aggregate. No information that could identify your responses will be reported.The survey will be open until Friday, April 27, at 2:00 PM.Thank you for your participation.  If you have any questions about the study, please contact Janel Sutkus, Director of Institutional Research and Analysis, at jsutkus@cmu.edu.Sincerely,Amy Burkert, Vice Provost for EducationGina Casalegno, Dean of Student Affairs Not included for students studying abroad

14. End of survey messages • On campus: • Thank you for completing the CMU-SES!Watch your e-mail for details on how to claim your thank you gift and for your invitation to the closing party on April 27 where we will announce the winners of the two iPad3's (you do NOT need to attend to win). • Studying abroad: • Thank you for completing the CMU-SES!You have been entered into the drawing for one of two iPad3's.

15. E-mail upon completion Dear Andrew,Thank you for participating in the CMU-Student Experience Study. This e-mail is your ticket for TWO things:1) You have won a certificate for a free FroYo at the Skibo Café. Please pick up your certificate at:The University Center, Wean Commons on Tuesday, April 3, from 3:00 to 6:00 PM.You will need your student ID and this e-mail to receive your certificate. You can either print this e-mail or show it on your smartphone. The certificate will be valid until May 11.2) You are invited to the CMU-SES closing party at:The University Center, Connan Room on Friday, April 27, from 5:00 to 7:00 PM.The party will include refreshments and music, and students who attend will receive tickets for drawings for CMU gear.We will also announce the winners of the two iPad3's (you do not need to attend to win).You will need your student ID and this e-mail to attend. You can either print this e-mail or show it on your smartphone.We look forward to seeing you there.Janel Sutkus and Hyunny Ro Amy Burkert Gina Casalegno and Holly HippensteelOffice of Institutional Research and Analysis Office of the Vice Provost for Education Office of the Dean of Student Affairs

16. Certificates Each certificate was hand-numbered in green ink and samples were given to each vendor.

17. Responses by date

18. Responses by incentive

19. Incentive redemption

20. Total costs Compared to the most recent census survey (April 2011 Undergraduate Advising), we paid $143 per additional response percentage point or $2.61 per additional student

21. Future plans • CMU-SES in Spring 2015: • GOAL: Maintain response rate in the low 50’s • Marketing and branding continues • No large incentives • Keep small incentives (replace fitness class and coffee) • Anticipated budget - $1700 • New “health behaviors” survey in Fall 2013: • GOAL: Response rate in low 50’s to match with • FY’s and sophs from CMU-SES in Spring 2013 • Create marketing and branding campaign • Guaranteed small incentives (FroYo, other tasty but healthy treats) • Raffle every two days for Fitbugs

22. In the time since. . . One large incentive: iPad Mini UG pop randomly split in two

24. For further information • Janel • jsutkus@cmu.edu • Hyunny • hro@cmu.edu