1 / 20

Centre for Open Learning of Mathematics, Science Computing and Technology (COLMSCT)

Online interactive assessment: short free-text questions with tailored feedback Sally Jordan, Barbara Brockbank and Philip Butcher GIREP-EPEC-PHEC 2009. Centre for Open Learning of Mathematics, Science Computing and Technology (COLMSCT). Pushing the boundaries….

maina
Download Presentation

Centre for Open Learning of Mathematics, Science Computing and Technology (COLMSCT)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Online interactive assessment: short free-text questions with tailored feedbackSally Jordan, Barbara Brockbank and Philip ButcherGIREP-EPEC-PHEC 2009 Centre for Open Learning of Mathematics, Science Computing and Technology (COLMSCT)

  2. Pushing the boundaries… • We wanted to be able to ask questions requiring free text answers of a phrase or sentence in length; • This required us to mark many different answers as correct.. • and many different answers as incorrect… • We have used a commercially provided linguistically-based authoring tool (from Intelligent Assessment Technologies Ltd.); • The system copes well with poor spelling and, usually, with poor grammar; • It can handle answers in which word order is significant and it accurately marks negated forms of a correct answer.

  3. Novel features • The IAT questions sit within OpenMark and students are offered three attempts with increasing feedback; • We provide tailored feedback on both incorrect and incomplete responses; • We have used student responses to developmental versions of the questions, themselves delivered online, to improve the answer matching; • We wrote 82 questions, 26 of which are now in use in regular formative and summative iCMAs on three interdisciplinary science courses. Link to demonstration site

  4. Evaluation 1:Usability lab observations • Six students were observed in June 2007; • They reacted to the questions in interesting ways; most gave their answers as phrases or in note form, even when it had been suggested that answers should be given as a complete sentence. • One student said ‘I’m going to give my answers in the same way as I would for a tutor marked assignment’ – and he did exactly that, composing his answers carefully as grammatically correct sentences. • Students told us that they found the feedback useful but they were observed to engage with the feedback provided in very different ways.

  5. Evaluation 1:Further investigation into student responses Responses to questions in summative use are, in general, more likely to be: • correct • expressed in a sentence • longer • altered for subsequent attempts at a question than responses to formative-only questions.

  6. sometimes longer to quite a ridiculous extent… 1) a train slow down on a straight track: there is no change in direction but there is a change in speed, so there is a change in velocity. 2) a car goes around a bend at constant speed: there is no change in speed but there is a change in direction, so there is a change in velocity. 3) a shark turns to chase it unfortunate prey and increases its speed as it does so: there is a change in speed and a change in direction, so there is change in velocity on both counts.

  7. Length of responses to a question in formative-only use

  8. …and to the same question in summative use

  9. …and with a filter applied to limit length to 20 words

  10. Evaluation 2:Human-computer marking comparison • The computer marking was compared with that of 6 human markers; • For most questions the computer’s marking was indistinguishable from that of the human markers; • For all questions, the computer’s marking was closer to that of the question author than that of some of the human markers; • The computer was not always ‘right’, but neither were the human markers.

  11. Evaluation 3Computer-computer marking comparison • An undergraduate student (not of computer science) developed answer matching using two algorithmically based systems, Java regular expressions and OpenMark PMatch; • These are not simple ‘bag of words’ systems; • Student responses were used in the development of the answer matching, as had been the case for the linguistically based IAT system; • The results were compared.

  12. Ongoing work • OpenMark’s own PMatch answer matching is now providing results that are at least as accurate as IAT’s for a range of short-answer free text questions; • The project currently hinges on accurate human marking of thousands of responses, and the development of the answer matching in response to these is time-consuming and tedious; • Alistair Willis is working to use machine-learning to remove some of the drudgery; early results are encouraging.

  13. For further information: • Jordan, Sally (2009) Assessment for learning: pushing the boundaries of computer based assessment. Practitioner Research in Higher Education, 3(1), 11-19.Available online at http://194.81.189.19/ojs/index.php/prhe • Jordan, Sally and Mitchell, Tom (2009) E-assessment for learning? The potential of short free-text questions with tailored feedback. British Journal of Educational Technology, 40, 2, 371-385 • Butcher, P.G. and Jordan, S.E. A comparison of human and computer marking of short free-text student responses (recently submitted – please contact the authors for a copy)

  14. Acknowledgments • Funding from COLMSCT and piCETL; • The assistance of many people associated with COLMSCT and piCETL, especially Alistair Willis and Richard Jordan; • Tom Mitchell of Intelligent Assessment Technologies Ltd.

  15. Sally JordanOpenCETL The Open UniversityWalton HallMilton KeynesMK7 6AAs.e.jordan@open.ac.uk http://www.open.ac.uk/colmsct/projects/sallyjordan/

More Related