Reviewing Course Evaluation Data: Perception vs reality Peter D Munn and Sheila D Scutter

1. Reviewing Course Evaluation Data: Perception vs reality Peter D Munn and Sheila D Scutter

2. Introduction • Graduate Course Evaluation Questionnaire (GCEQ) provided to all graduates Australia wide • Comprises 25 Likert scale statements and two free response items • Six key areas of graduates’ experience • Clear goals and standards • Appropriate workload • Appropriate assessment • Good teaching • Generic skills • Overall satisfaction

3. Introduction • University of South Australia uses good teaching, generic skills and overall satisfaction as key performance indicators of perceived teaching quality • Areas of concern with Nursing program GCEQ scores. • Assumed to be due to course content • A comprehensive review process to understand specific areas of concern in nursing.

4. Nursing program • Three year undergraduate program • Offered in 2 city and 1 regional campus • Offered in internal and external mode • Approximately 1000 students • Many part time students • Access via Year 12, STAT, bridging programs • Special entry test for ATSI students • Very high participation of equity groups.

5. Method • Graduate Course Evaluation Questionnaire(GCEQ) • GCEQ data for the nursing programs across the city and regional campuses were analysed by external and internal mode of study • Thematic analysis on comments on the ‘best aspects’ and ‘areas most in need of improvement’ responses analysed by mode of study • Individual item scores reviewed

6. Student experience questionnaire (SEQ) • Distributed online to all currently enrolled students • Feedback on aspects of academic life, resources and services • Sixteen Likert scale items and two open response items concerning course and program quality • Responses compared by mode of study • Thematic analysis of comments by mode of study

7. Course Evaluation Instrument • Each course in the University is evaluated every time it is offered • Course Evaluation Questionaire (CEI), online instrument developed by the University • CEI contains 10 core Likert-scale questions concerning course quality • Additional items may be added, this analysis concentrated on the 10 core items

8. Focus Groups with Students Three focus groups were conducted: • Two with interstate, rural and metropolitan South Australian students • One with indigenous students and their learning support coordinator • Focus Groups with Staff • Eight staff representing both campuses

9. Results GCEQ Scores, Good Teaching

10. Results GCEQ data • Concerns over good teaching, overall satisfaction and generic skills; • Low response rates may be a source of bias

11. Results Student Evaluation Questionnaire • Responses to most statements were positive with a mean score over 4 (maximum 5); • Lower scores received for items relating to: • Support provided by teaching staff; • Timeframe for return of assignments; • Knowing what is expected in assignments; • Consistency in marking.

12. Results CEI data • Limited response by students made interpretation of data difficult; • Feedback on courses ranged from very good to very poor; • Main areas of concern: • I felt there was a genuine interest in my learning needs and progress; • The workload for this course was reasonable given my other study commitments; • I have received feedback that is constructive and helpful.

13. Focus Groups: GCEQ/SEQ analysis • Course Materials and Content • Little comment • Valued flexibility • Often arrived late • External students felt “second best • Revised courses in 2004 to update content

14. Communication between students and academic staff • A major concern of external students • Interpreted as lack of interest by academics • External students felt disadvantaged compared to internal students. • Staff concerns about students accessing email • Geographical support groups disestablished

15. Learning Support Service Availability • Requirements for assignment writing • Particular concern for indigenous students • Available online but prefer alternative delivery • Need for support/study groups • Difficulty accessing learning advisers. • Teaching and Learning Issues • Time not spent productively in workshops • Clinical experiences very positive • Staff/student ratio low • Lack of training for staff in external delivery • Student preparation for the online environment • High percentage for exams in science based courses

16. Assignments and Assessment • Consistency of requirements within courses • Turnaround times major concern • Clarity of expectations. • Large number of students affecting turnaround

17. Discussion • GCEQ scores indicated concerns about overall satisfaction, good teaching and generic skills • The instrument • Response rates • Timing • Interpreting the data • Students are not given the opportunity to provide many areas of this feedback in GCEQ. • Where is was possible to provide comment in the GCEQ, it was not reflected in item scores.

18. Discussion Key concerns of students identified from thematic analysis and focus groups: • Timely, consistent and useful feedback; • Communication with academic staff; • Learning support; • Practical workshops/professional placements • Lack of preparation for study

19. Recommendations Agreed practice model to be implemented • Provide up to date study guide • Early arrival of course materials • Fortnightly email communication • Respond to emails and calls in 24 hours • Contribute to discussion board weekly • Harvard reference material • Geographical location maps • Information packages for Maths and English support

20. Agreed Practice model… Key points sheet for assignments and marking • Moderation of assignment marking • Staff development for external mode • Two week turnaournd for first assignment • Feedback before next assignment due • Promote course evaluation • Respond to student feedback

21. Initiatives around GCEQ response rates and distribution. • Extension of preparatory courses (Sciences) • Review nursing workshops • Review assessment

22. Conclusion • GCEQ scores alone do not provide direction for program improvement. • Thorough analysis of course and program feedback is necessary to understand problems and to develop appropriate changes to programs.