How good is a robot tutor? The effectiveness of excel as a teaching resource multiplier in teaching statistics

1. How good is a robot tutor? The effectiveness of excel as a teaching resource multiplier in teaching statistics Dave Nunez Colin Tredoux Susan Malcolm-Smith ACSENT Lab University of Cape Town Jacob Jaftha Dept. of Mathematics and Applied Mathematics University of Cape Town

2. Context • UCT Psychology has an extensive statistics teaching programme (1st year to honours) • Research focus makes this an imperative • By honours, are expected to apply stats to a significant individual research project • A mixed group of students entering • All have high-school maths, or have completed/concurrently completing a year-long numeracy course • Stats is largely disliked, and provokes significant anxiety

3. Context • Large classes, few tutors • Typically 40:1 student:tutor ratio • Excel based tutorials developed to counter this (lab facilities can cope with numbers) – “tutor in a can”; “tutorbot”; “tutortron-2000” • Positive student feedback from excel tuts • Liked that they could take them home • Seemed to compensate for poor lecture attendance • BUT – very little interaction between teachers & students (how were explanations/queries handled? Was it necessary?)

4. The excel based tutorials used • In development since 2003 • Almost all technical glitches resolved • Contain text, exercises and evaluation • Text supplements textbook (text and images); also includes animations & simulations • Teaches concepts and tools • Provides exercises which are immediately scored (feedback given for each question) • Each tut ends with a mini-test which must be submitted [electronically] • Each tut takes 120-150 minutes to complete

5. The excel based tutorials used • The tutorials aim to be more than simple exercises • Embed some teaching by interaction & feedback • Raises the issue: Can interactive, discovery based learning surpass student-tutor interaction for learning statistics • Some topics are well suited for discovery (sampling distribution of the mean) • Some topics are poorly suited for it (probability) • Do the excel tutorials lead to skill transfer?

6. Methods used in the past • Pre-test/post-test • Without a control, cannot show the tutorial is the cause (even a bad tut teaches something) • Voluntary assignment • No control for motivation variables • No control for repetition • Performance often measured by means of psychological variables • Confidence, mastery, conceptual learning • No absolute task-based criteria

7. Deficits in past methods • Poor controls • No proper control within subjects (natural learning) • No proper control across conditions (subjects self-assign to conditions) • These are often related to ethical concerns • Measures are generally poor • Single measure of complex, time-dependent phenomenon • No criterion based assessment (i.e. low ecological validity of findings)

8. Research questions • Do Excel based tutorials (EBTs) compare in performance (marks scored) to pen-and-paper tutorials (PnPs)? • Is there a difference in terms of psychological variables (mastery, confidence) between EBTs and PnPs?

9. Strengths of the current study • Two-group quasi-experiment • Pseudo random assignment of students to excel/pen-and-paper tutorials • Strong control/similarity of tutorials (we think) • Semester long, continuous assessment • Standard test after each tutorial (criterion and psychological measures) • Final exam at the end of the semester

10. Sample • The 2007 PSY2006F class • Statistics lecture each Friday; One stats tut a week • 172 students (only Humanities students) • Almost all have been through 3 tutorials in PSY1001W on using excel for stats • 2007 cohort not significantly different from other years • Not told about the study; simply told strange tutorial structure was due to logistical reasons

11. Materials • PnP tuts are ‘traditional’ as done in the dept. before advent of excel tuts • Published in a textbook (we partly wrote) – in 2001 • Choose tutors who excel (!) at statistics • They lead students (groups of 30-40) through worksheets and explain problems and theory as they go along • Students are given 2 hour classroom sessions to complete tuts (mostly don’t finish) • Students are required to submit the completed worksheet a week after the classroom session

12. Materials • Excel tuts (latest versions) • Developed by us (2003-2007) • 1 senior tutor in the lab for stats queries, junior tutors for technical problems • Students are given 2 hour lab sessions (groups of 30-40) to complete tuts (mostly don’t finish) • Students are required to submit the completed excel worksheet a week after the lab session

13. Design • Control for individual variation and cross-group effects • Each student does 4 EBTs, 4 PnPs (8 topics in the course) • Two ‘streams’ – EPEPEPEP, PEPEPEPE • Within subjects design, and cross-group comparison • The non-statistics marks in the course (research methods, psychometrics & qualitative methods) can be used to validate (traditionally high R2 between them)

14. Measures • Exam at the end • 2 hour practical exam (given data, problem solving – no concepts) • Do each exam section in the same technology form as the tuts were done in

15. Measures • Monday assessments • Each tut has a set of MCQ items • 6 MCQ items, 3 concepts, 3 calculations; one each easy, moderate, hard • 5 Likert items about confidence with the material, usefulness of tut, degree of understanding, how much extra help is needed

16. Measures (3) Two students, Able and Baker, want to get into the honours class, but they have taken different third year subjects. Able did the PSY300X course (which had a mean mark of 53% and a standard deviation of 11%) and he got a mark of 80%. Baker on the other hand did the PSY300Y course (mean mark of 57% and a standard deviation of 7.5%), and got a mark of 77%. If honours places are awarded to students who stand out the most in their courses, which one of the students should get into honours and why? • Able should get in, because he scored 27% above the course average • Baker should get in, because he scored 20% above the course average • Able should get in, because he scored proportionately higher above the course average • Baker should get in, because he scored proportionately higher above the course average Distribution X is normally distributed; distribution Y has a standard normal distribution. Which of the following statements MUST BE FALSE? • The mean of distribution X is 2 • The standard deviation of distribution Y is 1 • Distribution Y must always give the same proportion of high scores as low scores when sampled randomly • Distribution X never gives scores lower than distribution Y when sampled randomly.

17. Validation

18. Validation

19. C C C C C C Attituderesults

20. Preferenceeffects

21. Preferenceeffects

22. C C C C C C Mondayassessments

23. C C C C C C C C Examresults

24. C C C C C C Testingeffects

25. What the data shows • The EBTs can function as a robot tutor • With small tutor team, marks at least as good as traditional tutorials, better in a few topics for some students • Student preference/attitude is not associated with performance • Lack of significant findings • No patterned differences

26. What the data shows • EBTs can show an advantage • At exam time rather than test time • May indicate poor test or that EBTs need repetition to take effect • It is a weak effect - does not generalize to the entire class easily (group B only)

27. What the data DOES NOT show • Excel based statistics teaching is better • Content is confounded with form • Tutor ability is confounded with form • Students enjoy/get confidence from the EBTs • Only differences show the opposite • Students can leverage existing computer skills for learning statistics • Skills were pre-existing and not manipulated