2008 FAEIS Triennial Evaluation

1. 2008 FAEIS Triennial Evaluation The purpose of the FAEIS Triennial Evaluation is to conduct a comprehensive assessment of FAEIS once every three years in order to improve the usability of the information provided by FAEIS, monitor progress during the three-year grant cycle and plan for the future.

2. Objectives • Describe selected demographic and other variables that have potential association with FAEIS use. • Determine the level of usefulness of FAEIS data resources and applications. • Determine the level of satisfaction with selected components of FAEIS. • Determine levels of adoption of FAEIS and factors that explain the levels of adoption. • Determine the relationship of selected demographic and usage variables (involvement, use, experience, primary job title) with perceived usefulness and satisfaction with FAEIS.

3. Methodology • FAEIS Team developed first draft of questions • Peer Panel and USDA partners reviewed and made suggestions to draft questions • Peer Panel members were used to pilot test the on-line survey instrument. Changes made accordingly. • Web survey administered by the VT Center for Survey Research • 1,145 email addresses from FAEIS users used as survey sample group. Reduced to 1,120 by request of sample members. • Follow-up reminders sent to sample members • 396 respondents completed surveys for a response rate of 35 %

4. Question 1: My professional affiliation is best described as ? Analysis – fewer staff respondents in ‘09; greater number of faculty and administrator respondents in 2009 compared to 2007 & 2008.

5. Question 2 2009 Respondents by Institution Type # of Respondents by Institution Type 1862 – 183 1890 – 65 1994 – 6 PNLG – 109 Private – 21 Did not report -12 n=396

6. Question 3 Indicate which of the following organizations, if any, is your primary affiliation. n=384

7. Question 45Years in your current position? N=383

8. Questions 43 & 44 Demographics

9. Three-Year Trends: Question 4 : My total experience using FAEIS is ? Analysis – Users with > 3 yrs experience is trending upward while the other two groups are trending down.

10. Question 5My Involvement with FAEIS Includes: (select as many as apply; all possible combinations displayed) 2009 Results N=396

11. Three-Year Trends:Question 5: Involvement with FAEIS of Survey Respondents( “ none” and “ no answer” responses eliminated in ’07 & ‘08 ) Analysis – Data Entry and Product Developer survey respondents trending downward; Users of FAEIS products is trending upward.

12. Three-Year Trends: Question 6: How often have you used FAEIS in the past 12 months ? Analysis –The more frequent use categories (3-4 times & >4 times) are trending downward. The less frequent use categories (not at all & 1-2 times) are trending upward.

13. Three-Year Trends:Question 7: “My level of adoption (use) of FAEIS is best described as:”(no answer – eliminated from analysis in ‘07 & ‘08) Analysis – Only “unaware” and “use it but not sure of long-term use” are trending upwards. “Do not use” trending downward.

14. “ My level of adoption (use) of FAEIS is best described as:”(no answer – eliminated from analysis in ‘07 & ‘08) Analysis – Average level of adoption of FAEIS decreased in ’09 for non-data entry respondents

15. Question 21Who are the users of FAEIS data at your institution/organization/agency ? n=391

16. Question 9Is FAEIS collecting data from the institutions which you consider your peers? n=395 n=271

17. Question 10If you answered no to: “Is FAEIS collecting data from the institutions which you consider your peers?,” Which Institutions are missing ? • Private schools with agriculture programs • Family and Consumer Sciences programs at NLG

18. Question 11Do a sufficient number of institutions report data to allow for you to have a meaningful and helpful peer analysis? n=389 n=210

19. Three-Year Trends: Question 13: Describe your level of satisfaction with FAEIS components. (i.e., newsletter, help desk, data entry, report builder, instructions) (1=not satisfied; 5 = very satisfied) Analysis – Higher and lower satisfaction ratings are trending downward. The neutral rating (3) is trending upward.

20. Question 14How useful to you are the following FAEIS products and resources? • Highest Ratings: • CIP lists • Newsletter • Web site • Lowest Ratings: • Teaching Awards / Workshops website • FAEIS Data CD n=396

21. Question 14 hAre there any other FAEIS products or resources you would like to add to the list ? • Some Responses: • Salary data for placement data • Trend data such as degree programs that are changing over a 10 year cycle • Summary of IPEDS major trends that influence food and agricultural sciences • Similar data from 1994 land grant universities obtained from sister institutions.

22. Question 8 Does the CIP system adequately allow you to classify your instructional programs and/or address your reporting needs? n=392

23. Question 12What aspect of FAEIS is most important to you ? • Most Common Responses: • Student Enrollment data • Comparisons between institutions • Trends in enrollment, degrees awarded • Faculty salaries

24. Question 15For which of the following purposes have you used FAEIS data and/or reports? ( Please check all that apply) 2009 Results N=393 Analysis – Benchmarking and trend analysis are the two most common uses of FAEIS data.

25. Question 16 Please rate the usefulness of the data resources in FAEIS (for student enrollment, degrees awarded and placement, disciplines, gender, ethnicity, faculty salaries, etc.)(1 = not useful, 5 = very useful) 2009 Results N=397

26. Three-Year Trends: Question 16 : Please rate the usefulness of the data resources in FAEIS (for student enrollment, degrees awarded and placement, disciplines, gender, ethnicity, faculty salaries, etc.). Analysis – The higher usefulness scores (4, 5) are trending upward. The lower usefulness scores (1, 2) are stable & trending downward.

27. Question 17Please indicate how much confidence you have in the following FAEIS data resources. 1st 3rd 2nd n=386 = Least confident = Most confident

28. Question 18Is FAEIS the primary place you go to seek higher education data and other information relating to the food and agricultural sciences, natural resources and conservation, and family and consumer sciences/human sciences ? n=391

29. Question 19Do you use any of the following databases ? • Other: • CRIS, • NAUFRP, • USDA databases n=280

30. Question 20To what extent does FAEIS add value to other information sources that are available ? n=384

31. Question 24 Have you contacted the FAEIS Help Desk for assistance ? n=388

32. Question 25How would you rate the support you received from the Help Desk ? n=153

33. Question 26If you have contacted the FAEIS Help desk , how could its operation be improved ? (n=146) • 22 % - No improvement needed. • 1 % - Increase hours of operation • 0.3 % - Improve email response time • 12.4 % - Do not know • 1.8 % - Other suggestions: • Help Desk staff is limited by the lack of thorough, well-edited, accurate and consistent documentation, web screen content and communications. • Information provided by the Help Desk is not consistent from time-to-time. • Develop databases more thoroughly • Help Desk employees need better training • Clear understanding of the parameters of the data being entered • More understanding of the difficulties institutions are having entering the data

34. Question 27How would you rate the ease of navigation of the FAEIS web site? n=390

35. Question 28Have you used customized FAEIS data reports produced by the FAEIS Report Builder application ? n=393

36. Question 29 Have you personally used the FAEIS Report Builder application ? n=390

37. Question 30 How well does the FAEIS Report Builder meet your needs ? n=71

38. Question 31How easy is it to use the FAEIS Report Builder ? n=70

39. Question 32Are the instructions for the FAEIS Report Builder clear ? n=71

40. Question 33Would training on how to use the FAEIS Report Builder be useful to you? n=71

41. Question 34Do you enter FAEIS data, or are you responsible for data entered into FAEIS ? n=389

42. Question 35How would you rate your ability to obtain the following data in the format required by FAEIS ? Enrollments by degree level, discipline, gender, and ethnicity Degrees awarded by degree level, discipline, gender, and ethnicity N=262 N=260

43. Question 35How would you rate your ability to obtain the following data in the format required by FAEIS? Faculty by discipline, rank, gender, and ethnicity Placement by degree level into various CIP disciplines and work categories N=257 N=260

44. Question 36 Are data on placement of students collected at your institution ? N=134

45. Question 37At your institution, from where do you acquire the data for the various surveys ? N=257-261

46. Question 38What time of year would be best to enter the following types of FAEIS data ? N=253-260

47. Question 9What data entry method do you use for the following surveys ? N=244-256

48. Question 41Are the instructions clear for FAEIS data entry methods that you use ? • If answered no, which instructions need to be improved? • Would prefer questionnaire by mail • Navigation on screen a pain • Difficult to discern how to move from in-progress to completed • Finishing an entry is difficult • Difficult to navigate and decide when to save • Not sure what majors you want to be collected • Submission instructions are vague • Time frames for info requested needs to be stated clearly N=240

49. Question 22How can FAEIS be more useful to your institution or be improved ? • Most Common Responses: • Data input is cumbersome, needs to be easier, bothersome. • Need more information about what FAEIS is and what it can do for us. • I need to use it more. • Concerned about the accuracy and credibility of the data. • Complaints about CIP codes

50. Question 23 Please share any comments about when FAEIS was helpful to you or about when you had a problem with FAEIS. • Most common “Helpful” Responses: • Staff was helpful. (21) • Useful for faculty salary comparisons.(4) • Helpful in grant boilerplates (1) • Student diversity by CIP info (1) • Dropping / adding programs (1) • Growth of programs, trends in majors (2) • Long-term planning (1) • Most Common “Problem” Responses: • Data not dependable or accurate.(6) • Data entry not user friendly (5) • Report builder not user friendly. (2) • Poor job of marketing / facilitating use of FAEIS (1)