Drawing Diagrams in an Online Examination

Slide 1:Drawing Diagrams in an Online Examination Pete Thomas with help from Neil Smith & Kevin Waugh

Slide 2:Online Examinations

Slide 3:Issues What features should a drawing tool provide? How familiar should students be with the tool prior to the examination? How should the tool be provided to students in order to be used under examination conditions? How should a diagram be represented for transmission to the server? How should a diagram be represented for grading purposes? How to grade a diagram? How to provide feedback on a student’s diagram?

Slide 4:A typical exam question

Slide 5:A simple drawing tool

Slide 6:A typical student drawing

Slide 7:Opinions of the Drawing Tool

Slide 8:Student comments “The scroll bars in the boxes I found offputting, I started off drawing boxes that were too small - I nearly didn’t do this question because of that.” “Text boxes needed to be bigger so could see whole answer at once without scrolling” “The text boxes could be a bit bigger – at least big enough to contain the expected answers. It was a bit disconcerting trying to scroll up and down the boxes.” “Very easy once I got the hang of it”

Slide 9:Opinions of the Online Examination

Slide 10:The second trial 15 students 12 attempted the drawing question 9 answered the question successfully (but not necessarily correctly) 3 tried to draw a diagram but gave up 3 did not attempt the question

Slide 11:Discussion There is a definite reluctance on the part of some students to attempt to use a software tool under examination conditions, primarily due to the anticipated length of time it would take to complete a drawing. The exam question, whilst in the compulsory part of the paper, was of a ‘low stakes’ variety. Only 3 out of 15 students did not attempt the question - but this could be because they did not know the answer! If the students had had experience of the tool prior to the examination, this hesitancy would have been reduced. One would normally expect students to be exposed to the tool during their normal studies.

Slide 12:Imprecise diagrams Incomplete Contains extraneous material Malformed Typical student diagrams!

Slide 13:A model answer

Slide 14:A typical student drawing

Slide 15:Understanding diagrams Segmentation and Assimilation Translate a raster-based image into a set of diagrammatic primitives such as boxes, lines and text Identification Use domain knowledge to identify minimal meaningful units (MMUs) Aggregation Combines MMUs into higher level abstract features Interpretation Look for meaning in the diagram (e.g. comparing a student diagram with a specimen solution)

Slide 16:Marking diagrams

Slide 17:12 student answers to a drawing question Tutor mark: x = 2.78 (out of 5) s = 1.05 Auto mark: x = 2.73, s = 1.09 Spearman’s rho = 0.529, significant at the 0.05 level (2-tailed), N = 12 Pearson’s r = 0.763, significant at the 0.01 level (2-tailed), N = 12 But, N = 12 Automatic marking (pipelines)

Slide 18:Current work Automatic marking of tutor marked assignments Entity-Relationship (ER) diagrams Feedback Building a tool for revision purposes

Slide 19:The revised tool for E-R diagrams

Slide 20:Automatic marking (ERDs) 26 student answers to a drawing question Tutor mark: x = 21.35, s = 3.43 (range 13-25) Auto mark: x = 22.42, s = 2.12 (range 18-25) Spearman’s rho = 0.957, significant at the 0.01 level (2-tailed), N = 26 Pearson’s r = 0.939, significant at the 0.01 level (2-tailed), N = 26

Slide 21:Marking ERDs

Slide 22:Feedback – student answer

Slide 23:Feedback – compare with specimen solution

Slide 24:Conclusions Students seem to like online exams. Using a drawing tool under online exam conditions does not seem to pose major problems, particularly if the tool has been used prior to the exam. A simple drawing tool can be an effective addition to an online exam. Having some success in automatically marking diagrams. Making progress towards meaningful feedback.