1 / 23

On-line auto-graded assignments in economics: A test of their effectiveness

On-line auto-graded assignments in economics: A test of their effectiveness. Brendan Kennelly 1 , John Considine 2 and Darragh Flannery 1 1 Department of Economics, NUI Galway 2 Department of Economics, UCC. Structure of Presentation. Background Experiment Descriptive statistics

tao
Download Presentation

On-line auto-graded assignments in economics: A test of their effectiveness

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. On-line auto-graded assignments in economics: A test of their effectiveness Brendan Kennelly1, John Considine2 and Darragh Flannery1 1Department of Economics, NUI Galway 2Department of Economics, UCC

  2. Structure of Presentation • Background • Experiment • Descriptive statistics • Econometric analysis • Discussion

  3. Background: What is Aplia? • Designed by Paul Romer in 2000 • Course management tool • Facilitates learning and assessment • Using diagrams • Practice Assignment • Instant Feedback Qy Qx Income doubles for this consumer. Place new budget constraint on diagram.

  4. Background: Previous Work • Technology • Teaching/Learning • Cost to Instructor • Aplia • Collins et al (2008) • O’Dea and Ring (2008) • Aplia in Nat. Univ. of Ireland, Galway • Kennelly and Duffy (DEE 2007) • All Assignments on Aplia • Evaluated By Student Survey

  5. A New ‘Experiment’ Brendan Kennelly and David Duffy. 2007. Poster Presentation to Developments in Economic Education Conference. Evaluating the Use of Aplia as a Substitute to Alternative First Year Tutorials Using Student Survey Darragh Flannery (econometrics) and John Considine (student opinion) replace David Duffy Aplia being used as a complement to tutorials for Second Year students Evaluate by Student Survey and Focus Groups Evaluate by examining grade outcomes

  6. A New ‘Experiment’ • 390 students took managerial economics • Analysis mainly about 254 Commerce Students • 10 assignments; all did 1 and 2 on Aplia • Thereafter ¼ did a paper assignment instead of Aplia • Exam in 8 sections corresponding to the last 8 assignments • Students had to do 6 sections in the exam

  7. 25% Assignment grade based on best 8 out of 10 Group 1 Ai Aii A1 A2 A3 A4 A5 A6 A7 A8 Group 2 Ai Aii A1 A2 A3 A4 A5 A6 A7 A8 Group 3 Ai Aii A1 A2 A3 A4 A5 A6 A7 A8 A8 Ai Aii A1 A2 A3 A4 A5 A6 A7 Group 4 Exam Question Q1 Q2 Q3 Q8 75% Examination grade based on 6 question out of 8

  8. Summary Statistics

  9. Descriptive Statistics • Table 2: completion rate was lower for the written assignments than for the Aplia ones. • Table 3: no consistent pattern in the marks across assignment format (Aplia or written) • Table 4: no consistent pattern in the average mark per examination section by each group • Table 5: average mark varied considerably across exam section

  10. Table 2: Percentage of Group Completing Assignment

  11. Econometric analysis: overall exam performance

  12. Overall exam performance • The results are in line with expectations. We replaced overall assignment mark with a variable that measured the number of assignments attempted. This variable is also significant and has a very large coefficient. However there is relatively little variation in this variable.

  13. Econometric analysis: section specific regressions

  14. Econometric analysis: section specific regressions

  15. Section specific regressions • Aplia/paper dummy is statistically significant only in Section 2. In that case students who did the assignment by Aplia performed somewhat better in that section of the exam. • The number of marks that a student received in an assignment is statistically significant with a positive sign in the regressions on Section 3 and Section 8 and is weakly significant in the regression on Section 1.

  16. Panel analysis • The variation across students and across sections in our dependent variable meant that our data could be analysed using panel data techniques • Pooled data • Fixed effects • Random effects

  17. Econometric analysis: Pooled Data

  18. Econometric analysis: fixed effects

  19. Econometric analysis: Random Effects

  20. Discussion: Results • Very little evidence that Aplia v Written format of the assignment matters • Little evidence of a direct link between performance in an assignment and mark on the corresponding section in the exam. • If there is no difference in learning outcome then • Cost (student & instructor) becomes important • Students’ preferences become important

  21. Discussion: Issues • Where might learning from Aplia take place? • Between practice assignment and actual assignments? • Incrementally so that ALL students do a better examination – all have used Aplia 6 times? • Does the examination test learning from Aplia? • Is the experiment testing the benefits of technology and/or assignments rather than Aplia? • Is there cheating and/or cooperative learning?

  22. Discussion: Future Work • More analysis of existing data • Including two stage estimation • Change Experiment • Do not allow practice assignments for some of 2009-10 assignments • Compare with other 2009-10 assignments • Compare with exact same 2008-9 assignment • Is Aplia fit-for-purpose? • What purpose?

More Related