Five Innovative Technologies for Student Affairs Assessment

1. Five Innovative Technologies for Student Affairs Assessment Dr. Nathan Lindsay, Director of Student Life Assessment at UNCW

2. Session Objectives • To gain insights into how to integrate the five technologies into your institution’s assessment initiatives. • To become familiar with the pros and cons of the technologies, as well as the associated costs. • To become aware of the types of data gathered through these five methods, with an example project shown from each technology.

3. Testing the Clickers: When I think about Using technology in assessment, I… • Have bad flashbacks to experiences when the technology failed me • Wish that I would never get another web survey for the rest of my life • Am amazed by how much easier assessment is than it used to be 10-20 years ago • Get excited because I like trying new things • Other…

4. Technology is used effectively to Conduct Assessment at our institution. • Strongly Agree • Agree • Neither Agree nor Disagree • Disagree • Strongly Disagree • Don’t Know/Not Applicable

5. Five Technologies • Clicker Technology • Web-based survey instruments • PDA’s • OMR Scanner • Digital Voice Recorders

6. Context: UNCW and the Division of Student Affairs • UNCW • Regional, comprehensive university • 2008-2009 Enrollment = 12,195 • 58% Female; 85% White • Division of Student Affairs • Approx. 150 staff in 12 departments • Extensive recent growth in programs, services, and buildings • Long history of a Divisional Assessment Coordinator/Director; Assessment Committee established in 2004-2005

7. Student Response Systems (aka…Clickers) • Several companies provide technology and services • Turning Point Technologies gaining market share due to its seamless interaction with PowerPoint • Cost: $30 for the individual clickers; $100 for each receiver

8. Clickers ADVANTAGES DISADVANTAGES Initial cost of acquiring clickers/receivers Replacement cost of individual units that may disappear over time Can take longer to collect data than paper and pencil surveys • Enhances learning experience in workshops/classrooms • Provides opportunities for immediate feedback from audiences during lecture or presentations • Ability to save data after sessions

9. Examples of Clicker Use at UNCW • Used in student-based programs • Community Standards Presentation at Orientation • Student staff training in the Recreation Center • Used for division-wide staff meetings and trainings (the age limit is unspecified ) • Used in academic and non-academic settings

10. Web-Based Survey Instruments • Send an email invitation with a link • Post a survey on an existing website or computer • Data is available in real-time once a survey is submitted

11. Web-Based Survey Instruments: StudentVoice ADVANTAGES DISADVANTAGES • “Toolkits” provide a starting point in survey design • Security and anonymity controlled through usernames and passwords • Can offer incentives at end of survey • Reduces number crunching; cut and paste tables and graphs • Cross-tabbing function • Costs • Self-selection bias • Lower response rates

12. Gains from using Web-based assessment at UNCW • Easy to use—accessible for all levels of expertise • Staff ownership; level of engagement with assessment has increased • Enhanced culture of assessment and research

13. One Example: A Commuter Survey • Administered online by Student Voice in the Spring of 2008 • 232 randomly sampled UNCW students responded to this survey • The purpose of the survey was to assess the needs and satisfaction levels of UNCW students related to a wide variety of commuter services.

14. Online Commuter Survey (cont’d) “Of which of the following resources for non-traditional students are you aware?” • 55.3% None of the above • 29.0% Non-Traditional Student Organization (NTSO) • 18.4% “Seahawk Perch” Website • 17.1% The Finish Line Newsletter • 14.5% Non-Traditional Week • 9.2% Listserv • 6.6% Babysitter list

15. Online Commuter Survey (cont’d) • In response to the question, “Are you affiliated with the military?”, the results were as follows… • 0.5% - Yes, I am active duty. • 5% - Yes, I am a veteran. • 2% - Yes, I am a dependent. • 5% - Yes, as a spouse. • 5% - Yes, as a son or daughter • 2% - Yes, in some other capacity • 82% - No, I am not affiliated with the military

16. Online Commuter Survey (cont’d) The 39 participants who were affiliated with the military responded to the following question… “Of which of the following resources for students who are affiliated with the military are you aware?” • 59.0% Financial Aid and Veterans Services • 33.3% Student Veterans Organization • 5.1% Military Task Force • 35.9% None of the above

17. Online Commuter Survey (cont’d) • When students affiliated with the military were asked what resources have been most helpful, the most popular response was Financial Aid and Veteran's Services (11): • “Of course, I need money to go to school. So, the financial aid is the most helpful. She gets my paperwork in on time and voila! I get help through the year.” • “Financial Aid and Veterans Services have been an outstanding aid to my collegiate career. I'm a member of the SVO.” • When students affiliated with the military were then asked how these services could be improved, they responded that more information would be helpful (7): • “Knowledge about the other resources because I have no idea what they are outside of financial aid.” • “Be pro-active and contact X military to let them know what is offered. Older military students are isolated by campus life.”

18. Our Institution has Increased Its level of support for Student Veterans in the last few Years. • Strongly Agree • Agree • Neither Agree nor Disagree • Disagree • Strongly Disagree • Don’t Know/Not Applicable

19. Palm Pilots (PDA’s) • Randomly intercept students at busy campus locations • Position PDAs at points of service • Use multiple PDAs after a campus event or program • Access data immediately after PDA is hot synced

20. Palm Pilots Advantages Disadvantages Some computer expertise needed Small screen size Battery life (if not fully charged) Limits qualitative data entry • Inexpensive ($70-300) • Portable • Easy access and entry of information • Easy compatibility for downloading survey information

21. Channel 77 PDA Survey • Channel 77 is a closed circuit cable channel on the UNCW cable network. This channel broadcasts student media productions, motion pictures, educational programming, and university events. This survey was intended to determine the effectiveness of this channel and how it can better meet the needs of UNCW students. • Administered in spring 2009 through PDA’s; survey took students 3–5 minutes to complete • Surveys were randomly administered to students in residence halls, on-campus apartments, UNCW cafeteria, UNCW Student Union, and off-campus apartments; 268 students participated

22. Channel 77 PDA Survey(cont’d) The 80% of participants who were aware of UNCW’s Channel 77 indicated that they spend the following amounts of time watching it. • 0.5% Over 5 hours a day • 0.9% Between 3 and 5 hours a day • 6.6% Between 1 and 3 hours a day • 17.8% Less than 1 hour each day • 21.5% A few hours each week • 19.2% A few hours each month • 33.6% Less than a few hours each month

23. Channel 77 PDA Survey(cont’d) • The survey results indicated the following viewing preferences: *In addition to UNCW sporting events, respondents also indicated that they would like to watch other sporting events and more movies on Channel 77.

24. Channel 77 PDA Survey(cont’d) When asked how informative they find Channel 77 in helping them learn about UNCW’s campus culture and events, respondents indicated the following… • 2.3% Extremely informative • 26.2% Very informative • 45.3% Moderately informative • 19.6% Slightly informative • 6.5% Not at all informative

25. News Flash: Upgrade to iPhone and iPod • Students think it’s hip • No more tethered syncing • Can use finger instead of stylus • Enhanced options to capture open-ended comments

26. OMR Scanner • Optical mark read (OMR) technology • Auto-feeds paper-based surveys • Single or dual sided, ink or pencil read heads • Price varies; we paid approx. $5K for scanner and software

27. OMR Scanner Advantages Disadvantages Hardware/software costs; OMR forms cost Customization limitations unless additional OMR software is purchased (Design Expert) Large amounts of text from open-ended questions must be hand entered Paper/scanning issues, as well as challenges with missing data, stray marks, misalignment of sheets • Fast • Accurate • Eliminates data entry time and errors

28. Recent Usage at UNCW • Tracking health center visitors each month • Limits data entry for thousands of forms • Survey about student substance abuse at UNCW • Collaboration between academic and student affairs • Survey administration conducted in classroom, leading to a better response rate • 600 surveys were scanned • Data confirmed findings from other studies conducted by CROSSROADS, our students affairs department focusing on substance abuse

29. Digital Voice Recorders • Cost: $50-$200 • Can download the recording directly to your computer; digital format provides many options. • Adjustable speed on some models allows faster playback of a recording to finish it in half the time and still understand it. • Some models can set up index marks, which act like bookmarks and help you return to a specific point in the recording.

30. Digital Voice Recorder Advantages Disadvantages Data may need to be converted to text at some point for reports; transcribing requires extensive time Voice quality/clarity could result in missed data • Ability to concentrate on interview/focus group without having to worry about recording all comments. • Direct quotes for reports/presentations/articles can be captured more easily

31. Focus groups have been useful In our Assessment efforts. • Strongly Agree • Agree • Neither Agree nor Disagree • Disagree • Strongly Disagree • Not applicable/Don’t Know

32. PROJECT EXAMPLE:Mixed-Method Assessment • There are currently 400 active military or veteran students at UNCW • Focus group conducted with Student Veterans Organization, which built upon the prior Commuter Survey. The veterans’ feedback included the following: • Please realize that we are our own group—we are not just non-trads or adult students • “Veterans talk to veterans”—marketing and information sharing is more effective coming from within the group • SVO would like to make Veterans Day a more significant event on campus (perhaps with a full day of events)

33. Applying the Focus Group feedback • More proactive in educating faculty and staff on student veterans’ issues and resources. • Increased academic and personal support resources for student veterans. • Special session at Orientation created for veterans. • We would like to make more mentoring programs, social gatherings, and leadership opportunities available to the student veterans organization. (Amy Hector, 2009)

34. Concluding thoughts • Using various technologies helps us get out of the rut of always doing the same types of assessments • To facilitate greater technology use, staff training can be done individually or collectively • Funding for technology can be solicited from across the division/university and through grants

35. Which of the following technologies was it most helpful to learn about today? • Clickers • Web-based survey tools • Palm pilots • OMR scanners • Digital voice recorders • Not sure/Other

36. Sharing Best Practices • How have you used technology in assessment at your institution? • How have you overcome technological/logistical challenges in using such tools? • Are there additional technologies that you are considering for your assessment initiatives?

37. OTHER QUESTIONS/COMMENTS?