Assessing with Surveys - Which Ones to Use

1. Assessing with Surveys - Which Ones to Use Ross A. Griffith, Director of Institutional Research and Academic Administration Nancy Whelchel, Coordinator of Survey Research University Planning and Analysis Presented at the North Carolina Association for Institutional Research, March 19, 2002, Greensboro, NC available on line at http://www2.acs.ncsu.edu/UPA/assmt/ncair02_natlsurveys.pdf

2. Overview of Presentation • Entering Undergraduates • Enrolled Undergraduates • Graduate Students • Alumni • Faculty • UNC-OP/NC State Surveys • Summary – NC Publics and Privates • Conclusions

3. Entering Undergraduates • Cooperative Institutional Research Program (CIRP) • Freshman Survey (FY) • Entering Student Survey (ESS) • College Student Expectations Questionnaire (CSXQ) • Admitted Student Questionnaire (ASQ) and Admitted Student Questionnaire Plus (ASQ Plus)

4. CIRP Freshman Survey (FY) & Entering Student Survey (ESS) Administrator Higher Education Research Institute (HERI) and American Council on Education (ACE) heri@ucla.edu http://www.gseis.ucla.edu/heri/cirp.htm Purpose • Collect demographic and attitudinal characteristics on incoming students • Baseline measures for longitudinal follow-up (CSS)

5. CIRP Freshman Survey (FY) & Entering Student Survey (ESS) Information Collected • demographic characteristics • expectations of college experience • secondary school experience • degree goals & career plans • college finances • attitudes, values, & life goals • reasons for attending college • up to 21 local items Cost • $400 participation fee • $1 per returned survey

6. College Student Expectations Questionnaire (CSXQ) Administrator Center for Postsecondary Research and Planning (CPRP) at Indiana University cseq@indiana.edu http://www.indiana.edu/~cseq Purpose • Assess new students’ expectations upon matriculation • Baseline measure for longitudinal follow-up (CSEQ)

7. College Student Expectations Questionnaire (CSXQ) Information Collected • demographic characteristics • expectations for involvement in college activities • predicted satisfaction with college • expected nature of college learning environment • up to 20 local questions available Cost PaperWeb Participation fee: $125.00 $495.00 Per instrument: 0.75 na Per completed survey: 1.50 2.25 Additional questions: na 250.00 Special analysis (per hour): 100.00 100.00

8. Admitted Student Questionnaire (ASQ) & Admitted Student Questionnaire Plus (ASQ Plus) Administrator The College Board, Educational Testing Service info@aes.collegeboard.org www.collegeboard.com Purpose • Assess admitted students’ perceptions of institution and admissions process (all types of institutions) • facilitate competitor and overlap comparisons

9. Admitted Student Questionnaire (ASQ) & Admitted Student Questionnaire Plus (ASQ Plus) Information Collected • demographic characteristics • assessment of programs, admissions procedures, literature, institutional image • financial aid package • common acceptances • comparative evaluations • specific institutional comparisons (ASQ Plus) • limited local questions available (fee-based)

10. Admitted Student Questionnaire (ASQ) & Admitted Student Questionnaire Plus (ASQ Plus) Cost • ASQ • $600 participation fee • 0.55 per printed survey (minimum $235) • $225 for local questions • ASQ Plus • $925 participation fee • 0.60 per printed survey (minimum $260) • $225 for local questions

11. Enrolled Undergraduates • College Student Survey (CSS) • Your First College Year (YFCY) • College Student Experiences Questionnaire (CSEQ) • Community College Student Experiences Questionnaire (CCSEQ) • National Survey of Student Engagement (NSSE) • Adult Student Priorities Survey (ASPS) • Higher Education Data Sharing (HEDS) Consortium (Seniors)

12. College Student Survey (CSS) Administrator Higher Education Research Institute (HERI) heri@ucla.edu www.gseis.ucla.edu/heri/cirp.htm Purpose • Evaluate students’ experiences and satisfaction to assess how students have changed since entering college • Longitudinal follow-up for CIRP FY

13. College Student Survey (CSS) Information Collected • demographic characteristics • satisfaction with college experience • student involvement • cognitive and affective development • degree goals & career plans • attitudes, values, & life goals • use of technology • up to 20 additional local questions Cost • $450 participation fee • $1 per returned survey

14. CSS vs. CIRP - Events Occurring During College vs. Expected Events

15. Activities in College Senior Year vs. High School Senior Year

16. Student “Agrees Strongly” or “Somewhat”

17. College Student Experiences Questionnaire (CSEQ) Administrator Center for Postsecondary Research and Planning (CPRP) at Indiana University cseq@indiana.edu www.indiana.edu/~cseq Purpose • Measure student progress and quality of students’ experiences in and out of the classroom (4-year institutions) • longitudinal follow-up measures (CSXQ)

18. College Student Experiences Questionnaire (CSEQ) Information Collected • demographic characteristics • student engagement in learning activities • student ratings of college environment • student progress toward learning goals • student satisfaction with college • up to 20 local questions available

19. College Student Experiences Questionnaire (CSEQ) Cost PaperWeb Participation fee: $125.00 $495.00 Per instrument: 0.75 na Per completed survey: 1.50 2.25 Additional questions: na 250.00 Special analysis (per hour): 100.00 100.00

20. Wake Forest Items for 2000 CSEQ with Significantly Higher Mean Scores than 1998 CSEQ Time spent: Used e-mail to communicate with instructor/other students Searched the WWW or Internet for course material Used computer to retrieve material from outside library Used computer to produce visual displays Used computer to analyze data Developed a Web page/multimedia presentation

21. CSEQ - Quality of Effort: Computer and Information Technology Scale

22. HEDS Senior Survey • Location: http://www.heds.fandm.edu/ • Cost: $425 plus $.30 per survey plus $.40 per scanned survey – web based option is less • Purpose: Evaluate students’ experiences, enhancements, satisfaction and plans for the future

23. HEDS Senior SurveyUse of Technology - Enhancement

24. Senior Survey Mean Scores

25. National Survey of Student Engagement Administrator Center for Postsecondary Research and Planning (CPRP) at Indiana University, with Indiana University Center for Survey Research nsse@indiana.edu www.indiana.edu/~nsse Purpose • Gather outcomes assessment, undergraduate quality, and accountability data • Measure students’ engagement in effective educational practices

26. National Survey of Student Engagement Information Collected • demographic characteristics • assessment of academic challenge • engagement in learning activities • interaction with faculty • educational experiences • ratings of campus environment • progress toward learning goals • satisfaction with college • up to 20 local questions available for consortium members only

27. National Survey of Student Engagement Cost (Paper or Web) Participation fee:……………………………………...$275.00 Per-student fee: <4,000 undergrad enrolled*………………..2,250.00 4,000-15,000 “ ………………..3,350.00 >15,000 “ ………………..5,000.00 Optional oversampling paper ………………………………………...5.00 each web: 1st 1,000 ………………………………………...2.00 each # over 1,000 …….………………………………….1.50 each Local administration ……………………………………….1.50 each Consortium fee …………………………………150.00 each * $5 per student if less than minimum sample size

28. NSSE at NC State Why: UNC-OP volunteered UNC system When: Spring 2001 Who: random sample of 996 undergraduates • 499 first year students (10.6%) • # respondents = 217 (43.5% response rate) • 497 seniors (9.0%) • # respondents = 195 (39.2% response rate) Format: Web and paper

29. NSSE at NC State Cost* Participation fee:………………… $ 275 (UNC) Per-student fee: ………………… $5,000 (UNC) Peer** comparison report…. $ 250 (NCSU) Peer benchmark report ……. $ 250 (NCSU) Letterhead paper (@2000) $ 160 (NCSU) Total …………………………………….. $5,935 * no oversampling, no consortium ** 10 peer institutions selected by NC State (maximum number allowed)

30. NSSE at NC State Presentation of results: Highlights Report (prepared by Ephraim Schechter, University Planning and Analysis) • Focuses on noteworthy differences between NC State (all, first year, senior) and: • NSSE national norms • Doctoral/Research-Extensive Institution NSSE participants • Peer Group (combined) • NC State expected results • on web (http://www2.acs.ncsu.edu/UPA/assmt/nsse/nsse01.htm ) • presented at Assessment Brown Bag Seminar • to be presented to deans, associate deans, department heads

31. NSSE at NC State

32. NSSE at NC State Possible uses of comparative data: NC State • Public relations & recruitment • seniors give relatively high ratings for active and collaborative benchmark • first-year students give relatively high ratings for student interactions with faculty members • all students give high ratings to educational experiences contributing to their ability to use computers and information technology • Improvement • 1st-year students give relatively low ratings to level of academic challenge • seniors report relatively fewer assigned readings

33. NSSE at NC State Possible Uses: NC State Programs • Tutorial Center • possible evidence that center is having positive impact in certain skill areas • New Student Orientation • possible evidence that NSO is having positive impact in certain areas • possible evidence that NSO could enhance/expand some programs to improve some areas with low ratings • increased usefulness of NSSE and NSO own assessment instruments if NSO revises survey items and response categories to mirror NSSE

34. NSSE at NC State Limitations • Comparison groups • only to NSSE participants • only to certain groups of institutions • Small sample size • unable to do needed sub-group analyses (college, department, program participants, etc.) • Cost • prohibitively expensive ($25,000+) to administer to all 2nd term freshman and seniors • Redundancy • collect similar information on UNC/NC State Sophomore and Graduating Senior Surveys

35. The Future of NSSEat NC State? Maybe…. • Use in alternate years (non-UNC-OP survey years) • Oversample for more useful comparisons • Develop a consortium for more useful comparisons • Include some NSSE-type items in UNC/NC State surveys

36. HEDS Graduate Student Survey • Location: http://www.heds.fandm.edu/ • Costs: $250 – web based only over institution’s server • Purpose: Determine primarily students’ satisfaction with “graduate” school

37. Alumni Comprehensive Alumni Assessment Survey (CAAS) College Results Survey (CRS) Higher Education Data Sharing (HEDS) Alumni Survey: • Location: http://www.heds.fandm.edu/ • Cost: $450 plus $.29 per survey plus $.45 per scanned survey • Purpose: Determine five years out present activities of alumni and how well alma mater prepared them

38. Faculty/Staff/Institution • Institutional Performance Survey (IPS) • Institutional Priorities Survey (IPS) • Higher Education Research Institute (HERI) Faculty • WFU Faculty Survey • http://www.wfu.edu/Administrative-offices/Institutional-Research/Faculty-Survey/results01.pdf

39. HERI Faculty Survey • Location: http://www.gseis.ucla.edu/heri/cirp.htm • Cost:$325 plus $3.25 per returned survey • Purpose: Demographics, activities and priorities of faculty

40. 1998 HERI Faculty SurveyUse of Computers

41. 1998 HERI Faculty SurveySources of Stress

42. UNC-OP/ NC State Surveys State of North Carolina requires UNC-OP survey data from all 16 institutions in the system • Surveys include Entering First Year Students, 2nd Semester Sophomores, Graduating Seniors, Alumni (1 year out) • NCSU builds on OP surveys • more often • more questions • own alumni

43. UNC-OP/ NC State Surveys Census Improved ability for sub-analyses = more useful data Survey reports include results: • campus-wide • campus-wide by gender & race/ethnicity • by college • by department within college • College/Departmental insert-specific items (Graduating Senior & Alumni surveys)

44. UNC-OP/NC State Surveys Items are repeated over time and between surveys Reports and presentations (NC State and professional meetings) • trend analyses • cohort analyses • longitudinal analyses • retention studies • benchmark with other UNC institutions

45. UNC-OP/NC State Surveys Results used by • UNC Board of Governors and NC legislature for planning programs and budgets for the university system • Students, prospective students, parents, and NC public • NC State/colleges/departments use for • outcomes assessment and strategic planning • determine key program features and their importance to stakeholders • curricular change and innovation • improve performance measurement capabilities • benchmark performance with others • improving customer service

46. Publics CIRP ENT – 6 In House ENT – 20 NSSE ENR – 10 In House ENR – 36 CRS ALU – 1 In House ALU – 30 HERI FAC – 6 In House FAC – 37 Privates CIRP ENT – 17 In House ENT – 12 NSSE ENR – 9 In House ENR – 13 CAAS ALU – 1 In House ALU – 19 HERI FAC – 8 In House FAC - 6 Summary – NC Publics & Privates

47. Conclusions • There are a number of excellent surveys available commercially to meet unique needs at different prices while most allow “additional questions” tailored to the institution • In House surveys certainly have obvious cost and other benefits but are also labor intensive, time consuming and provide no external comparative data