Aplication of CAS in automatic assessment of math skills

1. Aplication of CAS in automatic assessment of math skills Przemysław Kajetanowicz (przemyslaw.kajetanowicz@pwr.wroc.pl) Jędrzej Wierzejewski(jedrzej.wierzejewski@pwr.wroc.pl) Wrocław University of Technology

2. Agenda • Knowledge assesment in math instruction • Automatic assessment in algebra E-course: history, functionality, implementation, results and students’ feedback • How automatic tests work • Recent developments: Java+Mathematica-based tests • Future

3. Assessment of progress in math • Measuring progress in math = measuring mastery in problem solving. • Grading of math problems • Grading procedure usually reflects typical solution structure • Common though not always applicable: partial credit for partial solution • Assessment tools in computer-aided learning systems • LMS-dependent • Stand-alone

4. E-course in algebra - overview • Content structure • Lecture notes • Interactive exercises (Java-driven) – over 120 problem types supported • Practice tests (Java-driven) • Graded exams (Java-driven) • Functionality of assessment tools • Random generation of data • Controlling the difficulty level at design time • Flexible grading (partial credit) • Step-by-step solution presentation • Completeness check • Initial correctness check

5. Lecture notes (1) Studying… … and exploring math

6. Lecture notes (2) Studying… …and self-testing

7. Exercises Exercise… ...and solution on demand

8. Functionality of tests (1) Graphing problems and related tools

9. Functionality of tests (2) Dedicated tools for Gauss elimination

10. Functionality of tests (3) Correct solution immediately available

11. Completeness check A student can go back to test as many times as he/she wishes.

12. Initial correctness check Chances to make corrections (enthusiastically greeted by students) – teachers decides on # of chances

13. E-course - implementation • Spring 2005 - 55 students • Fall 2005 – 400 students • Fall 2006 – almost 1000 students • Hybrid instruction • Classroom meetings (3 or 4 hours weekly) • (Fall 2006) grading based on • 5 online e-exams (play the role of homework) • Final in-lab e-exam (proctored) • In-class activity (not much credit though)

14. Administrative grading system • Available credit • 5 online e-exams: 5 problems, 1 hour, 25 points available; best four exams count • In-class activity – 4 points available • In-lab final exam: 8 problems, 90 minutes, 40 points available • Grading procedure based on student’s systematic work • A minimum of 20 on the final necessary (but not sufficient) to pass • 20% of online e-exams total score added to final (available maximum = 20) • In-class credit added (maximum = 4) • Necessary total to pass = 31

15. Grading system in practice Required total to pass Required minimum on final

16. Fall 2005 – Final grades

17. E-course results vs. traditional course (data after make-up exams) Traditional course (Fall 2005 results) E-course

18. Question 1 To what extent did you find the new form of the course (in particular, automatic tests and exams) helpful in your mastering the course material (as confronted with traditional way of learning)?

19. Question 2 How willing would you be to sign up for other math courses, were they offered in a similar form?

20. Question 6 How fairly, in your opinion, was your knowledge assessed by the system of automatic exams? Note. The exam questions were of similar type to those given on traditional exams. Presently, no statistical comparison is possible between the new and the traditional form in terms of how students judge the fairness (no survey was given to „traditionally assessed students”. The majority of students (74%) judged the system as fair (grades 4 to 6).

21. Students’ comments The course is a new appealing form of study to me. It is stress-free. (…) Giving tests and exams that way implies that the grading system is fair. The course is a great thing, and studying algebra that way is truly easy. Thank you !

22. Student’s comments (2) At the beginning I was terrified at the perspective of taking a course that was delivered in that way, but now I would like to wholeheartedly thank the instructors for the opportunity of being a part of this “experiment”. I can definitely say that thanks to this course I understood things that had been all Greek to me before (...) I truly believe that more courses should be taught that way. I wish to thank again for the opportunity to have been in this course.

23. Inside a test (Java-driven only) • Hard-coded • Problem data generation • Solution presentation • Initial grading • Completeness check, initial correctness check • Database-related operations • Parameter-driven • Difficulty level, problem sub-type • Scores for individual problems • Number of problems • Time assigned for solution • Specific test behaviour (completeness warning, # of tries to make corrections etc.) • Training vs. administrative purpose of test

24. Java+webMathematica (1) • Java: • Formulates problem (generates problem data) • Accepts student’s entry(ies) • Sends data and solution to webMathematica • webMathematica: • Compares solution with correct result • Sends outcome back to Java • Generates and sends back other elements (e.g. graphs or expressions needed for solution presentation)

25. Java+webMathematica (solution presentation)

26. Java+webMathematica (2)

27. Java+webMathematica (2 - solution)

28. Java+webMathematica (3)

29. Java+webMathematica (3 - solution)

30. Java+xml+Mathematica – version 1 • Xml: • Holds problem formulation (parametrized, so many problems of a common type are supported) • Holds solution (parametrized) • Holds problem data sets • Java: • Reads xml and behaves correspondingly • Mathematica: • Encodes formulation and solution in xml, provides necessary graphics etc. • Generates and encodes data sets in xml Space-consuming (especially if graphics involved)

31. Java+xml+Mathematica – version 2 • Xml: • Holds problem formulation (parametrized, so many problems of a common type are supported) • Holds solution (parametrized) • Java: • Reads xml • Generates data sets • Mathematica: • Encodes formulation and solution in xml, provides necessary graphics etc. • NO data sets generated by Mathematica Quick and space-saving

32. Java+xml+Mathematica (formulation) Formulation - xml <zestaw podtyp = "Y" > <dK>8</dK> <dX>2</dX> <dO>OY</dO> <dP>8</dP> <dA>-1/8</dA> <dB>2</dB> <o><![CDATA[%M y %N = &nbsp;]]></o> <o><![CDATA[ %M x + %N ]]></o> <p>-1/8</p> <p>2</p> </zestaw> Data sets - xml (including correct result)

33. Java+xml+Mathematica (solution) Solution (one of steps)

34. Java+xml+Mathematica (v. 2) Formulation & solution - xml Data sets generation - Java

35. Java+xml+Mathematica (v. 2) - solution Solution (parametrized) - xml Specific values - Java

36. Pros and cons of automatic assessment • Automatic assessment – main highlights • Proves far more effective (see the following slides). • Gives a student more opportunity to demonstrate skills. • E-course: 25 problems on online e-exams + 8 on the final • Traditional: 2 x 3 problems on mid-term „paper” exams + 5 on the final • Motivates a student to systematic work. • Every student gets different problems. • Saves time. • Automatic assessment – downsides • Certain problem types (e.g. „prove that…”) cannot be supported („as yet” - as dr. Wierzejewski says…). • Some students focus on solution algorithms rather than on math concepts and methods.

37. Forthcoming future • E-course in Calculus • Remedial course in secondary-school math • Continuation of teaching the e-course in algebra online • Completion of e-course in linear algebra

38. Thank you for your attention