1 / 28

Using MEQs to inform teaching excellence

Advance HE Surveys Conference 8 th May 2019. Using MEQs to inform teaching excellence. Dr Tim Linsey Head of Academic Systems & Evaluation Academic Systems & Evaluation Directorate for Student Achievement Kingston University t.linsey@Kingston.ac.uk.

wetherby
Download Presentation

Using MEQs to inform teaching excellence

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Advance HE Surveys Conference 8th May 2019 • Using MEQs to inform teaching excellence Dr Tim Linsey Head of Academic Systems & Evaluation Academic Systems & Evaluation Directorate for Student Achievement Kingston University t.linsey@Kingston.ac.uk

  2. Background – Reintroduction of MEQs • Decision taken in January 2017 to reintroduce MEQs • MEQ Working Group • 10 Quantitative + 2 Qualitative questions • March 2017 – University using Blue and Paper surveys • November 2017 to July 2018 – Primarily online surveys • September 2018 – All online surveys (with option for paper) • MEQ Environment: Blue from Explorance

  3. Orchestrated approach • Briefing guide and PowerPoint for all module leaders • Set of agreed statements to conveyed to students • Student created video introducing MEQs • Staff asked to find a slot in class • Staff requested to leave class for 15 mins. • Use of course representatives

  4. VLE Integration My Module Evaluations

  5. Processes & Timing • MEQs run all year but two main survey windows (16 days) • Automatic publishing of MEQs in to each module in the VLE • Reports automatically Published into the VLE within a few hours of an MEQ completing • Systems – mostly automated • Integration of Blue with the SIS and VLE • Tableau Dashboards • Aiming for full automation for 2019/20

  6. 2018/19 (to March) • 832 MEQ reports generated (exceeding minimum threshold of 4) • 76% of student responses contained qualitative feedback • 38% students completed one or more MEQs • 47% completed via mobile devices • Communications Plasma screens University Buses Emails VLE Intranet

  7. Module Reports • Staff and student reports similar except the student version excluded comments and comparisons (Department and Faculty averages)

  8. Best things Improve

  9. Further Reports • Department, Faculty and University aggregate reports • Summary reports for each Faculty • Modules with zero responses or not met threshold • Custom Reports

  10. Summary Report for all Modules 2016/17 Summary table ranking all modules by their mean overall score. Colour coded =< 3.5 => 4.5

  11. Summary Report for all Modules 2017/18 • Colour coding was problematic • Staff suggestion to rank by standard deviation from the overall university mean.

  12. Additionally • Comparison of 2016/17 vs 2017/18

  13. Statistical Analysis • Wilcoxon test used to compare aggregate data between 2017 & 2018 (mixed and Faculty aggregated level) • Weak but significant –ve correlation between module size and mean MEQ score (Spearmans Rank) • Weak but Significant +ve correlation between mean score and completion %. (Spearmans Rank)

  14. We noted • Care needed to be taken with aggregated data and inferences drawn from it • An individual MEQ report is informative for the module team knowing the local context but care needs to be taken without looking at trends and other metrics. • Significant churn in MEQ Module rankings 2017 vs 2018

  15. Summary Report for all Modules 2018/19 Reviewed our approach to consider issues raised in the literature: • Comparisons between modules of different types, levels, sizes, functions, or disciplines • Averaging Ordinal scale data • Bias • Internal consistency (e.g. Boring, 2017; Clayson, 2018; Hornstein, 2017; Wagner et. al. 2016)

  16. November 2018 Summary Report Sorted by Faculty, Level, Response rate

  17. Statistical confidence Methodology: Dillman, D. Smyth, J, Christian, L. 2014 Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method. John Wiley & Sons.

  18. Ranking by % Agree

  19. Frequency Distributions • Request that staff also review the frequency distribution of their responses • Is the distribution bimodal, and if so why? Mean = 2.9

  20. Aggregating Questions to Themes • Teaching • Assessment • Academic Support • Organisation

  21. Data Warehouse • Raw data passed to the KU Data Warehouse • Tableau Dashboards (Strategic Planning and Data Insight Department). • Dashboards accessible by all staff including showing top 5 and bottom 5 modules for each level. • Data aggregated with ability to drill down to module level

  22. Annual Monitoring and Enhance Process • MEQ results are pre-populated into Module Enhancement Plans • Course Metrics dashboard

  23. Issues & Developments • When should the MEQ be distributed? – Focus Group feedback • Staff being named in qualitative feedback & issues of etiquette • Students concerned about anonymity • GDPR • 47% students completing MEQs via Mobile Devices • Automation – Administration & Analysis • Response rates – followed up with modules with high response rates. • Feedback to Students • Demographic analysis

  24. Collaborative • Led by Academic Systems & Evaluation Team • Information & Technology Services • Strategic Planning and Data Insight • Academic Registry • Faculties via the MEQ Working Group • Student Course Representatives • Explorance

  25. Any Questions?

More Related