1 / 26

Automated marking of individualised spreadsheet assignments: the impact of different formative self-assessment options

Automated marking of individualised spreadsheet assignments: the impact of different formative self-assessment options. Paul Blayney University of Sydney Australia p.blayney@econ.usyd.edu.au Mark Freeman University of Technology Sydney Australia mark.freeman@uts.edu.au. Outline.

chalice
Download Presentation

Automated marking of individualised spreadsheet assignments: the impact of different formative self-assessment options

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Automated marking of individualised spreadsheet assignments: the impact of different formative self-assessment options Paul BlayneyUniversity of Sydney Australia p.blayney@econ.usyd.edu.au Mark Freeman University of Technology Sydney Australia mark.freeman@uts.edu.au

  2. Outline • Objectives & motivation • Relevant research • Context & CAA tool • Method • Results • Conclusion

  3. Research objectives & motivation for the paper • Objective • To demonstrate how spreadsheets can be used to provide formative feedback to students • To investigate the effects on students of different types of formative feedback (usage restrictions) • Effects on academics and the university • Motive • Higgins 2002 – lack of feedback research • Improve academic productivity • Improve student motivation, knowledge and skills

  4. Relevant Literature • IT and learning - no significance difference • Russell (99) • Limited impact when IT introduced into learning if no change in assessment • Biggs (99), Alexander & Mckenzie (98) • New opportunity set for CAA • Scanned paper MCQ • Specialist software (e.g. WebMCQ, QuestionMark) • Standardised course mgmt. system (WebCT)

  5. CAA Opportunities • Assign assessment • Clarifying assessment • Complete assessment • Deliver formative feedback • Submit summative assessment • Mark assessment • Generate feedback • Deliver feedback • Evaluate assessment effectiveness

  6. The CAA tool • Overview • Worldware …Spreadsheet software (e.g. Excel) • Excel VBA (Visual Basic) incorporated to provide feedback and mark numerical problems • Problem solving application • Financial accounting • Management accounting • Tax • Finance • Etc (maths, stats, physics)

  7. CAA Opportunities • Assign assessment • Clarifying assessment • Complete assessment *** • Deliver formative feedback *** • Submit summative assessment • Mark assessment *** • Generate feedback *** • Deliver feedback • Evaluate assessment effectiveness

  8. Research Method • Students • Pre and post course surveys • Review of unsolicited comments from online discussion forum • Analysis of learning behaviour from files • Staff • Academics • VBA specialist reflections • Users interviews & surveys • Department/Faculty • Teaching & learning specialist interview • Head of department interview • Uni • IT interview

  9. What a student does • Receives weekly assignment file • Completes it • Self-assesses draft answers (different options) • Submits final weekly assignment • Gets mark

  10. Demographics of students surveyed • Gender: Male 53% Female 47% • Age: < 21 = 92% 21-25 = 7 >25 = 1% • 1st language: English 49% Other 51% • Working for $’s – hours/week: none 37% 1-5 15% 6-10 23% > 10 25%

  11. Student perceptions overall PrePost General • Feedback crucial 94% n/a • Pseudo self-assessment gauge ability 94% n/a • Pseudo self-assessment taken serious 82% n/a VBA • Self-assessment motivated learn n/a 60% • Self-assessment motivated try n/a 82% • Productive way to learn 76% 76% • Non-threatening 76% 79% • Confidence in CAA 64% 55% • Workload for summative value OK n/a 64% • CAA developed accounting problem solving n/a 61% • CAA developed spreadsheet skills n/a 73% • CAA more enjoyable n/a 75% .

  12. Demographic analysis of student pre-course perceptions that “feedback is crucial”

  13. Demographic analysis of student post-course perceptions • Gender table 3 • No significant differences • Native language – table 4 • No significant differences • Age – table 5 • No significant differences except less confidence in accuracy with older students (p=0.04) • Working hours – not reported in paper • No significant differences except less motivation to learn with working students (p=0.01) and lower confidence in program’s marking accuracy (p=0.08)

  14. Analysis of free-response comments solicited from survey

  15. Analysis of free-response unsolicited comments • Online discussion forum (Blackboard) • Bulk • Administrative • File location on network? • Digital drop box workings? • Ad hoc • Content • Assignment marking system

  16. Student learning behaviour with different self-assessment options Option 1 students performed best: table 7 (p=.0002)

  17. Usage of Self Assessment Details of Self Assessment Use (1 student for 1 assignment) 97 times !!

  18. What the academic does • Steps to produce self-assessable files • Design question • Create Excel spreadsheet file • Add VBA forms and code modules • Create unique files for students • Update VBA code in Master Marker file • Provide unique files to students • Receive files submitted by students • Run Master Marker file. Post results.

  19. Academic staff (developer) reflections and perceptions • The Developer • Accountant by training (academic since 1985) • Lotus 123 user from 1983 to 2000 • Close working relationship with two regimes of IT staff • The CAA Tool • Initial system developed over 10 years ago • Lotus 123 worksheet files and macros • Converted to Excel in 2001 • Used in business education • UG & PG and Professional Education • E.g. Financial Accounting, Management Accounting, Tax, Information Systems • Assignment submission • Email • CMS Digital drop box

  20. Academic staff (developer) reflections and perceptions • Benefits • Improve learning • Productivity • Opportunity for reduced teaching load • General support – colleagues and HOD • Opportunity for creativity and challenge • Research opportunity • Contributing to department productivity and reputation • Costs • Exposed to vagaries of IT • Time consuming trouble shooting student errors • When outsourced… • Exposed to poor academic practice • Reliance on others’ expertise

  21. Other academic users’ perceptions Free response comments from survey (n=8) support interview feedback

  22. Other academic users’ perceptions: potential diffusion of this particular CAA innovation Rogers (95) 5 factors effecting the mainstreaming of innovation (n=8)

  23. Effects on Department / Faculty • Benefits • Learning reputation for innovation • Improved productivity • Intra-departmental cooperation • Enhanced morale by supporting innovation • Plagiarism disincentive • Via file individualism • Individual data • Linked cells • Costs • Need to protect against intellectual property exit • “Black box” • Back up re reliance on developer • Exposure to system reliability (CMS, Internet, viruses) • System capacity requirements

  24. Effects on University • Benefits • Boundary testing • IT reputation • Learning reputation • Productivity • Breakdown of academic boundaries • Costs • Assessment innovation is high risk • IT assessment innovation is even riskier

  25. Conclusions • Confirmation that feedback is crucial • Best learning where students had a limited number of self-assessments • Limitations • Specialist VBA required • Sydney University context • Student propensity to circumvent • Controlling assignment difficulty

  26. Conclusions • Future research • Effect of different self-assessment options on summative assessment • Closer analysis of self-assessment and different learner attributes • Relationship between self-assessment assignments and assessment value and assignment frequency • Self-assessment options and learner confidence

More Related