Improving performance on a Marketing module through the use of Formative Computer-Assisted Assessment

1. Improving performance on a Marketing module through the use of Formative Computer-Assisted Assessment

2. This paper was first presented at the 2nd European Conference on The First Year Experience Gothenburg University, 9th - 11th May 2007

3. Work in progress

4. Introduction The delivery mode at MK1004 – Marketing Principles follows very much the traditional lecture model. Students have two in-class examinations. The first one is made of two parts: a MCQ (Multiple Choice Questions) test, consisting of 30 questions, and five open questions. The MCQ test accounts for 25% of the final student grade for this module.

5. Introduction There are two main reasons that pressed the author to conduct this research. First - the University of Wolverhampton is moving to a more virtual and self-learning environment through the use of WOLF. Second - the week before the first examination students are presented with an in-class formative MCQ test.

6. Literature review There are a number of ways in which a VLE can be used to assess students. The most widespread one is computer-assisted-assessment (CAA). One of the tools supplied by a VLE is self-assessment, through the use of a databank of questions, linked to automated marking and instant feedback; other frequent tool is the tracking of the students’ use of materials within the VLE. Both facilities are present in WOLF.

7. Literature review The correlation between the use of formative CAA and exam results have been researched by several authors and a positive link has been found. However, according to Clarke et al. (2004) research into the use of online formative MCQ still remains scarce and more studies are needed.

10. Evaluation Methodology Reeves (1993) suggest that using a combination of quantitative and qualitative approaches is more suitable to assess learning supported by a VLE. In order to improve the validity of our study a number of sources – access logs, survey (with close and open end questions), automated tracking and analysis of in-class test results was used.

11. Evaluation Methodology Access logs give us the opportunity to find out about accessing patterns of each student, in relation to date, hour and number of times a MCQ test was accessed and number of times it was completed. Automated tracking of results allowed the researcher to find out about the marks gained for each test.

12. Table 4 – Tracking for first and last day of the online MCQ tests Total of 3512 page views and 1014 interactions

13. Evaluation Methodology Analysis of in-class test results allow for historical comparisons with the three last cohorts of students (2005-06 Semester 1 & 2 and 2006-07 Semester 1) as well as within this specific cohort - by comparing students who did take the online MCQ tests with those that did not take them and within the group that did take the tests, by comparing formative grades with summative ones.

14. Evaluation Methodology • Finally, surveys were chosen as a way to get students’ opinions regarding the usefulness of the online MCQ tasks towards their learning and final results. • Questionnaire: • 38 closed questions and 5 open questions • paper-based and completed in-class after the test • applied to 50% of the students enrolled in the module • a total of 114 questionnaires were received.

15. Qualitative analysis The overall conclusion is that the online MCQ tests were a valuable learning resource for many students, helping them prepare for the summative in-class test. “the online MCQ tests motivated me to learn and to keep trying” (86%) “the online MCQ tests forced me to study more” (81%) “the marks helped me to access how I was doing on my learning process” (80%) “as a method of learning I enjoyed doing the online MCQ tests” (76%) “the online MCQ tests were useful in revising the content of the module” (76%)

17. Qualitative analysis Question 3 asked the students if they have enjoyed the online MCQ tests as a method of learning. Two students who did not access the programme agreed with the questions and one student who accessed it just once and spent only 4:04 minutes strongly agreed. One may ask why he didn’t spend more time in such an enjoyable activity!

18. Qualitative analysis Question 4 asked the students if the online MCQ tests were well organised and structured. One student who accessed the task only once and for just 15 seconds did agree. We are delight to see that in just 15 seconds the student was still able to appreciate the organisation and structure of the task. Relying on a questionnaire alone seems, from these findings, to be highly problematic

19. Quantitative analysis The last three cohorts had average grades of 16.58 (sd 3.75), 18.49 (sd 4.1), and 18.04 (sd 4.53). The current cohort had an average of 16.94 (sd 3.83)*. But *sample of 64 students

20. Quantitative analysis If we take into consideration only the students that fully used the online tool (12 tests ) then the average goes up to 19.67 (sd 4.5).

21. 236 observations

22. Model Nb of page views R² = .002 R² = .002 β = .041* β = .046* R² = .066 Nb of tests taken R² = .103 β = .257 β = .321 R² = .304 Summative result CAA results β = .551 Average time spent per test R² = .109 R² = .021 β = .330 β = .144* Total time spent on tests R² = .097 R² = .291 β = .312 β = .539

23. Conclusion and further research

24. Work in progress If you want to receive a copy of the final report please e-mail me at: mmartins@wlv.ac.uk Thank you