Teaching and Learning Applications related to the automated interpretation of ERDs

1. Teaching and Learning Applications related to the automated interpretation of ERDs Pete Thomas, Kevin Waugh, Neil Smith Centre for Research in Computing Department of Computing The Open University, UK TLAD 2007, 2nd July, Glasgow

2. outline for this presentation • background to our work on diagram understanding • experiments marking ERDs • applications using the ERD drawing tool and marker • the marking tool (mark scheme setter) • the student exerciser (revision tool) • the marker’s assistant (tutor tool v0.1) • where next….

3. diagram understanding – our approach } A five stage process 1. segmentation 2. assimilation 3. identification 4. aggregation 5. interpretation general diagram knowledge convert raster-based images to general diagram ‘features’: lines, arrows, boxes, circles, arcs, etc. } domain specific knowledge identify minimal meaningful combinations of features, combine into larger units and interpret

4. diagram understanding - marking 3 Feature-based diagram Identified MMUs 4 ********** ********** ********** ********** 5 Interpretation (grades) (Aggregated MMUs) Meaningful Units

5. diagram understanding – our approach minimal meaningful units (MMUs) a minimal collection of diagram features that, in the given domain, have a recognisable interpretation. Relationship Entity Type 1 Entity Type 2 Super Type SubType 1 Entity Type Sub Type 2

6. diagram understanding – our approach aggregations of MMUs combinations of MMUs that together form meaningful units (MUs) with domain specific interpretations • fixed aggregations Entity Type 1 Entity Type 3 Entity Type 2 • rule-based aggregations …….. Entity Type 1 Entity Type 3 Entity Type 2

7. the diagram notation is straightforward • a range of styles of question • closed with a single solution • open with multiple acceptable alternative solutions • a potentially wide range of student variations and errors (imprecision) • large number of student answers available to us research contextwhy use ERDs? however • we’re confident that the approach will work with (most) graph-based diagrams.

8. the marking problem Specimen Solution Student Answer How similar are they? Can we compute a similarity measure comparable to that a human marker would award? (for a given marking scheme)

9. what does the marking algorithm do? this is probably best described using a demonstration to show the first stages in the marking process… (the marker support tool)

10. establishing confidence in the marking algorithm • before we can start using the algorithm in teaching and learning support tools we need to be confident that it will work. • we have a standard approach of using examination questions and student answers to evaluate the quality of the grading algorithm. • the next few slides give the results of two major experiments we have undertaken so far.

11. experiment 1 Question: (3rd level database course, exam 2004) Consider the following scenario: The following data requirements relate to the operation of a company, TrainingU, that provides training courses for the staff of client organizations. Each training course is assigned an identifying code and has a descriptive title. Each course is made up of one or more units. Each unit has a title and a unit code. The unit code is unique within the course containing that unit, but not unique across all units. Each unit is used in exactly one course. ……… (a) Give an E-R model to represent these requirements. Your model should include an E–R diagram, showing the degree and participation conditions for all relationships, entity types, any additional constraints not expressed ……

12. Sample solution: Marks available: 7 experiment 1 Sample size 591 scripts: development set 197 diagrams test set 394 diagrams

13. experiment 2 Sample solution: Marks available: 7 Question: (3rd level database course, exam 2006)

14. experiment results Experiment 1: test set size = 394 Human – Tool marking differences Experiment 2: test set size = 72

15. experiment results Experiment 1 frequency distribution of marks awarded Experiment 2

16. the marking (grading) tool • grades student answers • against one or more sample solutions • using one or more marking schemes other things we can do… • report on common student errors • monitor human graders • cross compare “common unusual answers”, this might have an application in plagiarism detection • identify alternative plausible solutions and remark the scripts (dynamic adjustment of solution sets)

17. success… • the marking tool works well enough … at this stage in its development • our experience with the marking algorithm and drawing tool has lead us to develop several T+L support tools • work being funded by a HEFCE teaching fellowship held by Pete Thomas.

18. demonstrations: • students revision tool (mark student attempts at exercises, with question specific advice) • markers support tool (redraws student answers, to support tutor-based marking) • tutor tool (create exercises and marking schemes, for the above tools) • marking tool (marks against marking schemes)

19. student revision tool

20. markers support tool

21. mark scheme encoding and revision tool question setting tutor tool

22. marking tool • a very boring demonstration (sorry) the marking tool engine is at the heart of the student revision tool, tutor tool, etc. so we can see how “improvements” to the marking engine behave with large population marking tasks before implementing the changes in any additional tools.

23. where next…. the majority of what has been done so far (experiments, pilot studies with student volunteers, etc.) has been about proving the technology; the marking engine accuracy and tool usability. need to consider a formal evaluation of the revision tool, does the tool benefit the students, how do the students use the tool, etc. we will be including the revision tool as a standard part of the 3rd level course in relational databases during 2008 with a formal framework for student feedback and an analysis of student performance against tool use.

24. and thank you Questions?