1 / 24

On-line formative assessment: Does it work? Paul Catley

On-line formative assessment: Does it work? Paul Catley. Expectations. “C & IT will have a central role in maintaining the quality of higher education in an era when there are likely to be continuing pressures on costs and a need to respond to an increasing demand for places in institutions.”

cameron-richard
Download Presentation

On-line formative assessment: Does it work? Paul Catley

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. On-line formative assessment: Does it work?Paul Catley

  2. Expectations “C & IT will have a central role in maintaining the quality of higher education in an era when there are likely to be continuing pressures on costs and a need to respond to an increasing demand for places in institutions.” – Dearing Report, 1997: para 13.2

  3. Growth of VLEs • No mention of computer based learning in Gibbs & Jenkins’ Teaching Large Classes in Higher Education: How to Maintain Quality with Limited Resources (1992) • Mention of computer based learning in Gibbs’ Improving Student Learning through Assessment and Evaluation (1995) • 80% of HE institutions ran a VLE for their students - Jenkins et.al. (2001) • Now probably 100%

  4. What works? • The need to “carry forward strategies based on evidence of what works” – HEFCE strategy for e-learning, 2005: 1

  5. What works? • “Little is known about the effectiveness of these web-based supplements to face-to-face instruction. How intensively will students utilize on-line course materials? Does access to on-line course materials increase comprehension and retention? Despite the paucity of answers to these and similar questions the rush to make on-line technologies an important component of higher education continues.” (Coates & Humphreys, 2001: 133)

  6. What works? • On-line discussion groups • Students who post more messages on average do statistically better • But students who read but did not post (“lurkers”) did worse than those who neither read nor posted, despite the fact that those who did not engage with the discussion groups had prior to the module done less well than those who only read • Hoskins & Van Hooff (2005)

  7. Background • Running a first year, first term, compulsory, traditionally taught, undergraduate law module (Legal Method) • Results before the intervention were mixed • Decided to introduce on-line support in the form of optional formative multiple choice questions to accompany each of the eight lectures

  8. Why on-line quizzes might work • value of timely and immediate feedback (Brass & Pilven, 1999; Dalziel, 2001; Hammer & Henderson, 1972; Mazzolini, 1999) • easy access + ability to monitor own progress (MacDougall et. al., 1998) • perceived by students to be valuable (Hester, 1999) • valuable information for tutors (Mazzolini, 1999) • improved student performance by those who score well in quizzes (Bailey et. al., 2001)

  9. Results over two yearsprior to on-line support

  10. Results over two yearsprior to on-line support

  11. Comparing quiz takers and non-quiz takers

  12. Comparing the whole pre on-line support cohort with non-quiz takers

  13. Who took the quizzes?Analysis of the 2003-04 cohort • Total cohort (n=201) • Quiz takers - 54.2% • 157 accepted on the basis of their A-level grades • Average grade of quiz takers (n=81) - 22.12 • Average grade of non-quiz takers (n=76) - 22.63 • Average UCAS points of quiz takers - 335.55 • Average UCAS points of non-quiz takers - 323.68

  14. Who took the quizzes?Analysis of the 2003-04 cohort • All students (n=201) - 54.2% • Female (n=118) - 56.8% quiz takers • Male (n=83) - 50.6% quiz takers • Students aged 18-20 (n=164) - 50.0% qt • Students aged 21-29 (n=24) - 70.8% qt • Students aged 30+ (n=13) - 76.9% qt • Prior study of law (A/AS) (n=44) - 59.1% qt

  15. Who took the quizzes?Analysis of the 2003-04 cohort • All students (n=201) - 54.2% • A-level entrants (n=157) - 51.6% qt • Non A-level entrants (n=44) - 63.6% qt • British nationals (n=156) - 50.0% qt • Other EU (n=12) - 58.3% qt • International non-EU (n=33) - 72.7% • All non-British (n=45) - 68.9%

  16. Who took the quizzes?Analysis of the 2003-04 cohort Summary • Generally non-traditional entrants (non A-level, non-British, mature) students were more likely to take the quizzes. • Amongst A-level entrants prior performance of quiz takers and non-quiz takers virtually identical

  17. Performance and quiz attempts

  18. Comparing performance by number of quizzes attempted

  19. Impact of various factors on Legal Method performance

  20. Performance in other first year modules (2003-04 cohort)

  21. Performance over three years (2003-04 cohort)

  22. Performance over three years(2003-04 cohort)

  23. The Impact of MCQs Conclusions • On-line formative MCQs appeared to have a very significant impact on performance in the module and in other first year modules • Impact persisted throughout the degree programme

  24. The Impact of MCQs Questions • Could the impact be replicated on other courses and in other disciplines? • Is there something special about the module? • Is there something special about law? • Was there something special about the particular cohorts? • Was there something special about the on-line quizzes?

More Related