Embedding Computer-Aided Assessment in Curriculum: A Dilemma Resolved

1. Embedding computer-aided assessment and other electronic resources in the curriculum – did I really offer this title?!! Does it count? Martin Greenhow Mathematical Sciences Brunel University

2. A dilemma • Students do things only for marks (from school?) • They work to the next test only (even missing lectures!) • So individual lecturers need to set plenty of assessments in order to get their share of students’ attention/engagement (else they fail the module) • So students have no time for reflection and understanding • Go to 1) above! Result: the course is no fun!

3. CAA – a way out? • Low-stakes summative assessment - invigilated or not. • Preferred scheme: best-ever mark from their first 5 attempts counts towards their module mark; not invigilated and group work is allowed/encouraged. Exam pass required! • Blended assessment (CAA, traditional exams, labs, reports & essays …)

4. Economies? • From Teaching Mathematics and Its Applications doi:10.1093/teamat/hrv012 Simplifying the logistics of setting tests, particularly in-course or continuous assessment tests of large cohorts (up to several hundred in an HE context). Depending on the embedding scenario chosen (see below), CAA can remove many of the administration tasks such as timetabling, booking rooms, informing students, making arrangements for students with special needs, repeat tests for those unable to do the first test instance . . . and finally the whole process again for resittests.

5. CAA at Brunel University • Javascript, MathML, SVG, web delivery • Questions database spans GCSE, A-level, undergraduate topics, ADULT LEARNERS/employability aptitude • >3000 questions in the database span MC, NI, RNI, PNI, TFU, MR, NI+confidence, Revealed MC, drag&drop etc. (no free-form maths input … yet): PROS/CONS? • Hundreds of users take circa 30,000 tests p.a. Students from Maths, Economics, Electrical and Electronic Engineering, Computing, Foundations of Engineering, Foundations of IT, PGCE and Sports Science, even MSc student diagnostics. Much commonality

6. Adult numeracy? • Pictogram changes • Scenario changes • Notice the key -> mal-rule

7. Reading a cumulative table Randoms in numbers in table and in statement wordings here

8. Accessibility font sizes & colours under user’s control Surface effects - towns, - given names (ethnic balance) - gender (balanced M/F) - game/sets/match randomised Games/Sets/Matches are partially randomised consistent with the rules of tennis – quite complicated to code in fact! From an actual aptitude test but original was wrong – that author never played tennis!

9. Feedback has dire warnings about borrowing on credit cards! Reading a table and synthesising information

10. Lots of randoms here including stated grade and dynamic diagrams via SVG

11. Nurses - contextualised • The only question with Imperial units! • Realistic weights • Realistic weight loss!

12. Two ‘realisations’ Note accessibility features: fonts, colours and input box size

13. Randomised names (surface effect) Randomised volume (1,2 or 4) "cefodizime","amoxicillin","sodium valproate","streptomycin","temocillin","voriconozole" Coding for those interested in it:QuestionText%QUESTION.NUMBER% = "Patient "+person_name+ " needs a "+injection_volume+" mL injection containing "+drug+" "+drug_concentration+" mg/mL. The displacement volume for "+drug+" is "+drug_disp_vol+". What volume of water should be added to the correct weight of "+drug+" to produce "+injection_volume+" mL of the injection?<p>"; Randomised concentration (125,250 or 500) Matched drug displacement volume • Where I need pharmacists: • Randomisation must be limited to reflect clinical practice • Incorrect answers can also be programmed based on common mistakes • Full feedback is given, but will students understand it? • Other information can be put in the feedback if needed (e.g. side effects of drug) • Links to web resources

14. Biosciences - decontextualised • What feedback should I be writing here? • Rich area to get involved with (I hope rich!) • Other life sciences? E.g. pharmacy?

15. Typical algebra MC question Not that easy even though MC (distracters formed by mal-rules). Data from rearranging equations (see later) can show that the TOPIC is hard regardless of how presented Random numbers (with control a != 1) Fonts & colours

16. Typical algebra MC question FEEDBACK –> LEARNING RESOURCE Related material links to stuff on web (Mathcentre, Exam Solutions, PurpleMaths, etc.) but this needs to be SECTIONED UP and be there (not U-tube) What do we need to teach in class? Probably this, probably not gen. quad. formula.

17. MCQs a bad image? • MCQs merely ask students to spot the correct answer when they see it. • They are very heavily ‘scaffolded’ with the key and distracters suggesting the form or size of the answer, e.g., not taking the square root of the variance would give a distracter that is unrealistically large for the standard deviation. • Partial knowledge may eliminate some distracters and encourage guessing, for example by using symmetry (a good skill, but probably not the one intended to be tested by the question). • Trial values such as x=0, 1, 2 . . . might be used until all but the correct answer is eliminated. • The question might be done backwards, e.g., differentiating all the options rather than integrating the integrand in the question. • Pattern spotting might be employed where the student looks for commonality in parts of the options. For example, what are the coordinates of a calculated point in space, with options (1,1,1), (1,1,0), (1,0,1) and (0,1,1) suggest that (1,1,1) is the correct answer.

18. Differentiate cos(2x) • Mal-rules? Pick up usual errors easily … Differentiate cos(x2) … needs a human to explain?

19. Reverse engineering Scenarios MathJax

20. TF(U) question • decontextualised • numbers and variable names (here d and S) randomised • Fonts & Colours • Low facility – high discrimination • Success largely independent of question type

21. Logic question • Pre-processing of input • Table generated on the fly • Required element positions randomised • TESTING PROGRAMMING very possible (we have VB and Java – others similar

22. Computer science question

23. Randomly-chosen algorithm Options to select from Detailed explanation More feedback adds counts and operations count for given n or as ORDER asked for here

24. Communication with marking scheme is difficult for free-form Input, so here students are asked to order some pseudocode (not all of which is needed).

25. Randomly chosen common algorithm with words that could apply to all algorithms. Tests students ability to distinguish between various algorithms.

26. Feedback – can we have too much? • Engineering/built env/maths question with schematic numbers change • ALL of the formative feedback

27. Feedback - contextualised • economics question • part of the formative feedback • SVG diagram realised according to the random parameters in the question • related material button links to any web resource (via centrally-held lookup array)

28. Mathematics for Economics(year-by-year changes in assessment) 2006-07, 2007-08: no CAA; A and non-A cohorts – results indistinguishable; two class tests, one 3-hour exam set by economics staff (10 days - times 2?) 2008-09: CAA introduced; 3-hour exam only (2 days - times 2?) 2009-10, 2010-15: one cohort, admission AS level mathematics; exam pass mandatory; 2-hour exam only (set by MG) (3 days!!!!!!)

29. Maths for Economics year-on-year results(% in grade/% mark) 272,000 questions done New topic Mystery!

30. DeSTRESS JISC Project • Builds on the Metal question design methodology and experiences to test basic stats for social sciences • Released new material end August 2011 • Hand calculation with realistically-sized data sets – solution to link with external software such as Excel, SPSS • Real data? Data cannot be ‘designed’ to have certain features or not. Problems with keeping the data and answers current without accessing live web sites (problems of access and communication with the marking scheme). • Another challenging area is the interpretation of charts and graphs

31. A bad question in my view

33. A statistics question - tables or formula?

34. Adult literacy? • In the following sentence, Violet has made at least one error. Fully correct each word (possibly including an apostrophe or a hyphen) in the sentence and press SUBMIT. • Either the enviroment will sopport this exploitation, or it will be disasterous for this under-privilaged socciety. • Scenario changes to give skeleton sentence with random number of errors dropped in in ‘break points’ • Students cannot do these even with Word available • Word should be turned off for marks.

35. What about students? • Some things can be done • Must disable Word • Should enable internet • VERY hard!!!

36. Programming syntax

38. Teacher interface – shop on Amazon?

39. Javascript, MathML and SVG provide a rich environment for setting objective questions • Positive effects on students’ perceptions and on exam performances • Widely applicable database of questions • Good source of reverse-engineered questions for all, especially teachers • MathsE.G. Try it at: http://www.mathcentre.ac.uk:8081/mathseg/ http://www.mathcentre.ac.uk:8081/mathsegteacher/ works on all Java-enabled browsers, mobile devices(this summer), no link to VLEs yet!