Transiting to an Online Course Evaluation Model: The Online Student Survey of Instruction

1. Transiting to an Online Course Evaluation Model: The Online Student Survey of Instruction Florida Association of Institutional Research 2010 Annual Conference February 2010

2. Student Survey of Instruction Presenters • Dr. James Coraggio, Director, Academic Effectiveness and Assessment • Magaly Tymms,Assessment Coordinator, Academic Effectiveness and Assessment With contributions from… Debbie Montalvo, SSI Coordinator Richard Jakubowski, SSI DB Administrator Florida Association of Institutional Research

3. SPC Background • SPC, established in 1927, is the oldest 2-year college in Florida • First Community College in Florida to offer 4 year degrees (2002) • Nine Campuses throughout Pinellas County • 2008-09 FTE: 16,967 (LD), 1,677 (UD) • Opening Fall 2009 credit enrollment: 29,560 • Annual 2008-09 headcount (credit and non-credit): 61,414 Florida Association of Institutional Research

4. Purpose • This presentation describes the development and implementation of a college-wide online process to administer course evaluations, the Student Survey of Instruction (SSI), which takes advantage of existing technology to efficiently collect student responses, disseminate the results in a more timely manner, and save the institution time and money. Florida Association of Institutional Research

5. Agenda • Issues with the prior process • Intended outcomes • New online process • Dissemination of results and reports • Response rates • Next steps Florida Association of Institutional Research

6. Issues with Prior Process Expenses related to Scantron forms, and numerous hours of sorting, organizing, and transporting forms Time consuming process of transcribing open-ended items Not all Scantrons were scanned and processed (scanning issues, data integrity issues, etc.) Scantron results and online results captured separately and reported separately Process conducted by multiple departments with no central ownership Florida Association of Institutional Research

7. Original SSI Items • Original SSI forms were developed by a college committee a number of years ago • Items were reviewed for content and a lecture form was aggregated into four categories (e.g., faculty student interaction, organization, presentation, and evaluation) • There were four SSI forms (Scantron): Lecture, Non-lecture, Clinical A*, and Clinical B* • Results were scanned by a third party (PCSB) and entered into PeopleSoft Florida Association of Institutional Research

8. eCampus SSI Items • Online SSI form was developed later by eCampus to assess online courses • This form was similar to the Lecture SSI form with some slightly modified items and the same four categories • Original online survey tool created to capture summative information in database Florida Association of Institutional Research

9. Performance Improvement • From Compliance to Performance Improvement Florida Association of Institutional Research

10. Intended Outcomes • New SSI online process was designed to: • Provide stakeholders (Faculty, Program Administrators, etc.) information to assist in making decisions that will result in performance improvement • Provide more reliable and valid course evaluation information • Create centralized coordination of the process • Save the college time and money (est. $45,000 per year) • Eliminate the need for in-class administration (more course time for instruction) Florida Association of Institutional Research

11. SSI Online Process Timeline • Fall 2007 • Established an SSI Team • Factor analysis (exploratory and confirmatory cross validation) determined that lecture and online SSI forms had three factors (e.g., faculty engagement, preparation and organization, and course instruction) • Spring 2008 • Pilot of 100 FTF course sections conducted • Fall 2008 • First collegewide administration (All course ANGEL shells enabled) • Created an online reporting tool (program administrators only) • Revised older survey administration tool Florida Association of Institutional Research

12. SSI Online Process Timeline • Spring 2009 • Created customized usage reporting tools • Created new survey interface with an opt-out capability • Added Faculty Representative to SSI Team • Created online reporting tool access for Faculty (No more paper) Florida Association of Institutional Research

13. SSI Online Process Timeline • Summer 2009 • First collegewide summer administration • Modified faculty email notices to include link to real-time response data • Began working with HEC on Clinical Solutions • Evaluated current SSI items and forms (SSI Revision Committee) • Fall 2009 • Implement self-service module so faculty can revise their own administration dates Florida Association of Institutional Research

14. SSI Online Process Timeline Spring 2010 College-wide Pilot of new SSI items set Pilot Clinical SSI in the areas of MLT, EMS and COE Add standard deviation information to reports Identify best practices from faculty members and programs with the highest response rates Florida Association of Institutional Research

15. SSI Link in ANGEL • Course instructors are required at the start of each term to: • enable their ANGEL course shells, • upload their course syllabus, and • attached their SSI link into their course. Florida Association of Institutional Research

16. Faculty Notification Emails • Faculty are provided four informational emails during their SSI administration. • Five days before the beginning • At the 50% mark • At the 75% mark • Two days after the end • Emails contain course, form, date, and response information. Florida Association of Institutional Research

17. Faculty Notification Emails Florida Association of Institutional Research

18. SSI Link in ANGEL Florida Association of Institutional Research

19. SSI Link in ANGEL Florida Association of Institutional Research

20. SSI Link in ANGEL Florida Association of Institutional Research

21. Accessing Information • A site was developed to provide faculty and program administrators information about the SSI online process: http://www.spcollege.edu/central/ssi/ Florida Association of Institutional Research

22. Information about the Process Florida Association of Institutional Research

23. Information about the Process Florida Association of Institutional Research

24. Accessing Results • All SSI survey reports are available on the online SSI Administration site: http://it.spcollege.edu/ssi/ • Faculty and Program Administrators can login by entering their SPC network username and password • Access is limited to only those courses under their area of responsibility. Florida Association of Institutional Research

25. Accessing Results Florida Association of Institutional Research

26. Accessing Results • Instructors and Program Administrators can select ‘Term’ from the drop-down menu (i.e. 0405) and select how they would like to view the results • Each individual section with its own results OR Combined sections by instructor Florida Association of Institutional Research

27. Accessing Results • After selecting a term and indicating ‘Each Individual Section with its own results’, users can view each individual section with its own results, as selected on the previous screen for Term: 0405. • By selecting any of the courses, users can view the SSI report for that course. Florida Association of Institutional Research

28. Accessing Results Florida Association of Institutional Research

29. Accessing Results • For the ‘Combined sections by Instructor’ options, users can view combined sections with results, as selected on the term selection screen (Term: 0405 in this example). • By selecting any of the instructor names, users can view a combined SSI report for that instructor. • Note: multiple reports for a single instructor may be listed if the instructor taught courses with either different SSI survey forms, different academic orgs, or on different campuses. Florida Association of Institutional Research

30. Accessing Results Florida Association of Institutional Research

31. Accessing Results • For an electronic version of the report (pdf), users can select ‘View in Adobe Reader’ • ‘Each Individual Section with its own results’ report produces individual SSI results, three comparative measures, graph, and student comments. • ‘Combined sections by Instructor’ report produces an instructor’s combined sections SSI results with three comparative measures, graph, and student comments by course. This report is somewhat similar to the SSI reports generated by AIS in the past. Florida Association of Institutional Research

32. Administration Dates • SSI is intended to be an evaluation of the instruction provided during the course. • It is administered after the students have participated in the majority of the course (no sooner than the 60% mark of the course) in order to provide the student an opportunity to experience the course instruction. • Length of the administration period is approximately the last 20%-25% of the course. • Faculty are provided default SSI administration start and end dates, and are permitted to make minor adjustments to these dates within a predetermined timeframe. This ensures a more standardized and consistent data collection process. Florida Association of Institutional Research

33. Changing Administration Dates Users can locate the course that they would like to change and select the Calendar icon Florida Association of Institutional Research

34. Changing Administration Dates • They then click on “Next Month” on the upper right hand side of the calendar to select the next month. The sample calendar above shows the course ends on 12/18 and the SSI closes on 12/11. Florida Association of Institutional Research

35. Changing Administration Dates • To extend the SSI to remain open until 12/18… • Select 12/18. It will turn blue through 12/18. This changes in real-time. • Should you make a mistake, simply click back on the original survey end date of 12/11 or select another. • The latest you can extend the survey to is 12/24. Florida Association of Institutional Research

36. Response Rates • Student response rates are calculated and compared each term by program (academic org), campus, form type, and overall college. Florida Association of Institutional Research

37. Response Rates Florida Association of Institutional Research

38. Response Rate Comparison • Comparison with another Florida public community college • Another community college debuted its online course evaluation system in Fall 2007. Until the Spring 2007 term, this community college relied on a paper-based course evaluation system for its traditional classroom courses. • Response rates:Pilot testSpring 2007: 30% classroom, 26% distance learningCollegewideFall 2007: 20% classroom, 26% distance learningSpring 2008: 24% classroom, 29% distance learning Florida Association of Institutional Research

39. SSI Revision Committee • An SSI Revision Committee was established in the Summer of 2009 to review the current SSI forms and make suggestions for revisions to the current items sets. • The new modality-neutral form was designed to address faculty and student engagement in the learning process and is being administered this term as a college-wide pilot. Florida Association of Institutional Research

40. Increasing Response Rates • While not the primary focus of the SSI Revision Committee, there was considerable interest in identifying and implementing ideas and processes to increase the response rate of the SSI. • Through creative brainstorming at the meetings and online, a list of “best practices” was developed. Florida Association of Institutional Research

41. Increasing Response Rates SSI Revision Committee Recommendations • Send Constant Reminders (‘inescapable messaging’) • Stress Confidentiality • Imbed link in announcement page -- “inescapable messaging” • Move every week to top of unit • Link to “content” – graded or non-graded • Bring students to lab /use of lab • Give extra credit • Genuinely convey the message that the information will be used to make this class better • Hold activities on campus, e.g., a table highlighting importance for accreditation  Florida Association of Institutional Research

42. Increasing Response Rates • There was some concern that item #5 (linking the SSI to accessing exams or other course materials) was not acceptable from a legal standpoint. • The question was posed to the Office of General Counsel, and, in brief, the answer was ‘yes’, the faculty could block access to course materials assuming that the content ‘blocked’ is just the next module or assignment the student would see, not specifically an exam. • The Office of General Counsel also advised against giving extra credit for taking the SSI. Florida Association of Institutional Research

43. Next Steps • Work with programs to improve response rates • Finish creating forms for programs using clinical items • Develop reports which provide trend information across terms and years • Finalize new items Florida Association of Institutional Research

44. Questions Florida Association of Institutional Research

45. Transiting to an Online Course Evaluation Model: The Online Student Survey of Instruction Florida Association of Institutional Research 2010 Annual Conference February 2010