New Web-Based Course Evaluation Services Available to Schools and Departments

1. New Web-Based Course Evaluation Services Available to Schools and Departments Presentation to Faculty Council November 6, 2009

2. Course Evaluation at Carolina • UNC Board of Governors policy: • Requires only that student evaluations of instruction must be conducted “…at regular intervals” (at least one semester each year) and “on an ongoing basis” for each instructional faculty member. • Carolina does not have a policy mandating the use of a common method or instrument for student evaluation of instruction. • Deans are authorized to determine how their courses will be evaluated. • The Provost is committed to providing centralized services to support the basic course evaluation needs of schools and departments. However, use of these services – including the new product to be implemented this year -- is optional.

3. Methods Currently Used By Professional Schools • Some have developed their own course evaluation processes to meet curriculum, accreditation, or other local needs. Examples: • Medicine: Comprehensive system of web-based student evaluations tailored for each component of the MD curriculum. • Business: Web-based system developed in-house to administer common evaluation form. • Law: Paper survey with open-ended questions distributed in class. • Others: Customized solution developed by outside vendor.

4. Current Services Offered Centrally:Carolina Course Evaluations • Standard instrument designed by 1999 Provost’s Task Force on Student Evaluation of Teaching in response to concerns about the validity of the old Carolina Course Review for use in tenure and promotion decisions. • Instrument contained sets of standard items to provide: • Instructors with feedback to help improve teaching skills, • Administrators with data use in personnel decisions, • Students with information to aid course selection. • Also intended to include: • Flexibility to allow departments, schools, and instructors to choose items that would be diagnostic for improving teaching and/or applicable for certain course settings. • Separate distribution of results to intended audiences: administrators, instructors, and students. • Ongoing evaluation of psychometric properties.

5. Development of Current Services • Task Force report endorsed by Faculty Council and Student Government in 1999 • Additional Faculty Council resolution in 2001 to provide funding for complete implementation • Center for Teaching and Learning charged with implementation • Two Formats: • Traditional paper (bubble sheet) version administered in class and taken to Information Technology Services (ITS) for scanning. • Web-based version developed by ITS in 2006 and piloted with a few schools. • Feedback collected from instructors, administrators, and students by Office of Institutional Research & Assessment in 2007-2008

6. Limitations to Current Paper and Web-Based Processes • Many of the flexible features originally intended to allow instructors and departments to add items for instructional improvement purposes were not fully implemented in either the paper or web-based system. • Example: Item bank was available in web-version but it was not possible for instructors to create their own items. • One-size-fits-all design – Could not adjust format for multiple instructors or schedule different evaluation periods to meet needs of courses with non-traditional session lengths • Required core questions did not appear to be relevant to distance education courses • Specialized reports for instructors, administrators, and students were difficult to deliver to intended audiences • Very resource-intensive • Would require reprogramming to work with PeopleSoft

7. Search for a New System or Services • Course Evaluation Advisory Committee appointed and charged by Provost to work with Institutional Research & Assessment (OIRA) to identify a new web-based evaluation system or services from an outside vendor. • Reviewed feedback from pilot studies, surveys, and interviews with campus users to establish requirements. • Researched commercial products and services, surveyed peer institutions, investigated open-source solutions (e.g., Sakai) and future capacity of Blackboard and PeopleSoft to evaluate courses. • Course Response™by Digital Measures offered the best combination of functionality, ease of administration, and cost. • Very positive feedback from advisory committee and academic unit representatives attending demonstrations this summer.

8. Features of Course Response™ • Flexible functions that can be set centrally or controlled at school, department, or course section level to: • Customize contents of evaluation instruments; add items supplied by Provost, dean, chair, or instructor. • Schedule evaluations when needed • Evaluate multiple instructors, TA’s, guest lecturers, etc. • Define who will receive what results (within limits set by University Counsel) • Create custom reports and do ad hoc analysis of results • A hosted, turn-key solution: Vendor provides all services including software, helpdesk, and security in IBM-owned and managed data center. • Students, faculty, and administrators enter through existing campus portals and authenticate using Onyen • Anonymity of student responses; FERPA-compliant • Quick implementation; can be ready to use with fall 2009 courses

9. Costs • Total annual cost to Carolina is based on number of schools participating and faculty FTE • Cost per school ranges from $2,500 for < 25 FTE to $4,500 for 100+ FTE • Discounts for additional schools and three-year agreement • Estimated total annual cost for campus: $35,000 - $40,000, to be funded by Provost’s Office

10. Implementation • Participants for Fall 2009 include: • School of Nursing • School of Education (selected courses) • School of Journalism & Mass Communication • College of Arts & Sciences – Departments of Romance Languages, Music, Sociology, and others • Training and support provided for faculty and staff with eventual goal of self-management by academic units • January 2010 – OIRA will collect user feedback and work with schools to plan for spring 2010 term. For more information, contact Larry Mayes (ldmayes@email.unc.edu) or Lynn Williford (lynn_williford@unc.edu) in the Office of Institutional Research & Assessment or call 962-1500.