Please silence cell phones.

1. Please silence cell phones. THANK YOU!

2. Challenges and Advantages of Online Class Evaluations Shohreh Bozorgmehri Manager, Instructional Web Technologies & Briandy Walden Web Developer, Instructional Web Technologies

3. TOPICS • BACKGROUND • CHALLENGES • TRANSITION • ADVANTAGES & RESULTS • FACTS & FIGURES • FUTURE • CONCLUSION • QUESTIONS

4. Inspiration BACKGROUND Inspiration • Paper • Online • Version 1: Instructor-initiated • Version 2: Department Administrative Tool for Evaluations (D.A.T.E.)

5. BACKGROUND • Paper Process • Purchase Scantron forms • Organize & distribute packets • Fill in forms • Collect & return forms • Scan forms • Analyze results • Distribute results to instructors

6. BACKGROUND Academic Senate Role • EEE final evaluations are based on the Campus-wide Teaching Evaluation Form (CTEF) approved by the academic senate committee, the Council on Student Experience (CSE). • Senate proposal: http://www.senate.uci.edu/9_IrvineManual/3ASMAppendices/Appendix11.html http://www.senate.uci.edu/9_IrvineManual/3ASMAppendices/Appendix11App2.html

7. BACKGROUND First Online Evaluations • Instructor initiated • Campus-wide Teaching Evaluation Form (CTEF) • Secure and anonymous • Fixed one time email announcement • Automated analysis of results

8. BACKGROUND Online evaluations had initially been driven by faculty requests. With broader interest, the needs of entire Schools or Departments became the new driving force behind the direction of the online evaluation tool’s evolution.

9. School of Engineering BACKGROUND • Engineering was the first school to use EEE’s tool school-wide • Entire school switched to online Final Evaluations • Simple Administrative Web page was developed • School’s evaluations fully online by Fall 2003 • Learned from experience and incorporated feedback into version 2

10. CHALLENGES • Significant Culture change • Concern of response rate decrease • Students would become responsible for taking and submitting evaluations on own time and outside of class • Duration changed from 15 minutes inside the classroom to what individual students chose on their own • Instructors would need to diligently notify and remind students in order to receive desirable submissions • Departments would need to communicate to faculty and/or directly enrolled students ENCOURAGEMENT & COMMUNICATION BECAME VITAL

11. CHALLENGES • Evaluation requirements: • CTEF format • Anonymity • Department Customizability • Cooperation from academic units • Willingness to take part in a potentially difficult transition • Strong history of good support and cooperation helped convince Departments that their needs would be met, and any unexpected issues would be addressed • Instructor & Student participation • Notification of students encouragement to participate required for best results

12. TRANSITION PROCESS OVERVIEW • TOOL FEATURES • ACTIVATION • ANNOUNCEMENT • MONITORING • RESULTS • WEB INTERFACE • STUDENT EXPERIENCE

13. TRANSITION Tool Features Departments steered development with requests: • Optional addition of four custom questions • Scantron-like results file • Printable report with page breaks • Response rate monitoring tools • More department control over instructor options • Email reminders for multiple evaluations combined into one • Only non-respondents receive reminders

14. Process TRANSITION Activation Administrators • Create and upload CSV files containing instructor IDs and course codes • Specify the bounds of instructor control over evaluation windows

15. Process TRANSITION Announcement • Administrators compose, schedule, and send email announcements and/or reminders using a simple online form through the Web interface • No limit to the number of announcements sent • Only non-respondents do receive reminders • Reminder emails can have different subject-line or body each time

16. Process TRANSITION Monitoring • Administrators monitor rate of evaluation submissions • Constant monitoring helps department administrators decide when to send announcements or reminder emails

17. Process TRANSITION Results Speed: • Administrators, evaluated instructors, and teaching assistants view and download results files in desired format(s) after the “Registrar’s Final Grade Submission Deadline”. Flexible Formats: • Comma Separated Value (CSV) • Print-friendly report • Scantron compatible

18. Web Interface TRANSITION • Error check • Prevents user errors from adversely affecting the process • Instructor abilities • Preserves option to allow or disallow instructors from changing evaluation dates/times, or canceling evaluations • Retains midterm evaluation options for all instructors, regardless of Department-activated final evaluations The D.A.T.E. Web Interface makes Department activation of final evaluations for multiple instructors easy and efficient.

19. Student Experience TRANSITION • Online form • Requires secure authentication • Available anywhere, at any time during the evaluation window • Quick and easy multiple choice questions • Provides space for typed open-ended responses • Incentives • “Respondent List” makes incentives possible • Anonymity preserved • Names cannot be associated with responses • List only available if 5 or more responses received (minimum increasable at department discretion)

20. ADVANTAGES & RESULTS • Save Time: 90% savings common • Save Money: $2000/yr for Scantron (PhySci) • Add Customizability: Add 4 custom questions • Unexpected side-effect: • With online evaluations, students have begun to submit more thorough and detailed comments, presumably because they are under no time pressure, and are able to type, rather than handwrite, their comments. • Because comments are already typed, they can be distributed and read more easily and quickly. • Results are compiled and ready for distribution and review in matter of hours rather than months. • Faculty are enabled to powerfully and immediately reflect the evaluation results to the upcoming class. • Some departments more rapidly are able to renew or take away TAs or lecturers appointments. DEFINITE IMPACT ON QUALITY OF TEACHING

21. ADVANTAGES & RESULTS EEE’s Evaluations tool is the fastest growing tool in the EEE course management system toolbox

22. Comparison FACTS & FIGURES 1 2 3 4 5 SurveyMonkey QuestBack Zoomerang Paper Evaluations D.A.T.E. Supported Unsupported

23. FACTS & FIGURES

24. FACTS & FIGURES Submissions Engineering Physical Sciences Biological Sciences Classes

25. FACTS & FIGURES * Projected Department participation for Fall, 2005

26. FACTS & FIGURES * Numbers in parentheses represent percentage of time saved per department

27. FUTURE • More customizability • New survey engine will power tools: • Evaluation tool • Survey tool • Quiz tool • More mobility • Ability to share Evaluation forms with others • Re-use frequently used questions & answers

28. CONCLUSION D.A.T.E.’s strength and key success come from the solid partnership our team has created and maintained with a wide range of senior academic faculty, committees, and campus units.

29. QUESTIONS? http://eee.uci.edu/iwt/uccsc2005/ For questions, please contact: or eee@uci.edu