1 / 24

 Does the Medium have an Effect on Assessment? 

 Does the Medium have an Effect on Assessment? . George McGuire Martin Youngson Douglas McMillan with Dave Fiddes and Athol Korabinski. Modes of exam delivery. Paper - current exam delivery Computer - growing trend For Mathematics, attractions include

adora
Download Presentation

 Does the Medium have an Effect on Assessment? 

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1.  Does the Medium have an Effect on Assessment?  George McGuire Martin Youngson Douglas McMillan with Dave Fiddes and Athol Korabinski

  2. Modes of exam delivery • Paper- current exam delivery • Computer - growing trend ForMathematics,attractions include • automatic (instantaneous) marking • logging of results Are exam results affected by mode of delivery?

  3. Changing the mode of delivery • Rewording effect - to fit the assessment package • Medium effect - questions read from the screen input of answers by typing ‘on’ the screen What is the significance of each of these effects for exam marks? Pilot study to investigate this question.

  4. The questions in the pilot study • Typical of the short questions in paper 1 of the Scottish Higher exam • Require few steps to reach the answer but still involve a reasonable amount of mathematics.

  5. Paper version of a test (P Format) Example of a paper question 1. If f (x) = 2x - 3 and g (x) = 2x2 - 3 find an expression for g (f (x)). Write your answer in the form ax2 + bx + c.

  6. Computer version (ICT format) Example of a computer question 1. If f (x) = 2x - 3 and g (x) = 2x2 - 3 then g (f (x)) = ax2 + bx + c. What are a, b and c? 1. What is a? 2. What is b? 3. What is c? CUE

  7. Determination of the medium effect • Comparison between paper and computer versions would have to take account of both rewording and medium effects • Exact paper copy of the computer version (RT format) eliminates any rewording effect ICT versus RT for medium effect only

  8. Reverse translation version (RT format)Example of an RT question 1. If f (x) = 2x - 3 and g (x) = 2x2 - 3 then g (f (x)) = ax2 + bx + c. What are a, b and c? 1. What is a? 2. What is b? 3. What is c?

  9. Determination of the rewording effect • No medium effect between P format and RT format (both paper) • Working influences marking in a paper test • Modification of the RT test, RTW format, allows space for working

  10. Rough Working 1. If f (x) = 2x - 3 and g (x) = 2x2 - 3 then g (f (x)) = ax2 + bx + c. What are a, b and c? Question in (RTW format) 1. What is a? 2. What is b? 3. What is c?

  11. Pilot study • 90 minutes in April 2001 • Two schools with Higher Mathematics pupils • The Tests: • P format - Three 30 minute tests with SQA questions • ICT format - Three computer tests using CUE • RT format - Three reverse translations of the computer tests • Each candidate sat the 3 tests, one in each format

  12. Preparation for the pilot study • Pupils familiar with paper exams but generally unfamiliar with computer exams • A visit to each school before the day of the pilot study • Explain what would take place on the day of the pilot • Practice test of some simple questions to give pupils some familiarity with CUE CUE

  13. Setting up the groups for the test • SQA split the pupils into 3 groups based on • gender • mathematical ability • Mathematical ability measured by: • standard grade results • higher prelim results

  14. Running the tests • Each candidate took each test - one in each format • Group sat tests in different orders • At any time no group was sitting • the same test as another group • a test in the same format as another group • Time was not an issue

  15. Setting up the pairings for the analysis • 62 matched pairs in total • 59 matched for gender and ability • 3 matched for ability not gender From different groups, matched pairs were set up:

  16. Marking of the tests • Paper tests marked by SQA • ICT tests marked automatically by computer • RT tests marked by me, looking only at final answer • RTW tests marked by me, taking account of rough working • Average mark over all tests: 39% • Individual marks range from 0% to 100%

  17. Analysis of marks • 2 statistical analyses: P - RTW and ITC - RTC • Both done on 62 differences in marks from matched pairs • Null hypothesis is true underlying mean is zero • One sample t-test performed • 62 is large enough to validate analysis

  18. The results for the rewording effect • Well outside the 95% confidence interval • Strong evidence of a difference • Was this a marker effect? • Marker effect eliminated by P papers remarked by me • Still strong evidence of a difference

  19. The results for the medium effect • Well inside the 95% confidence interval • No evidence of a difference

  20. The gender effects • Analysis repeated for male and female pairs • Same results as before

  21. Conclusions on the rewording effect Differences may have been due to • Rewording • Reformatting • The way the answers were set out • Pupils less inhibited doing rough working More detailed study needed to clarify

  22. Conclusions from the medium effect • Candidates had only limited prior experience of computer tests • There is no evidence of a difference due to the medium effect

  23. Does the Medium have an Effect on Assessment?  V  Thank you

  24. Directions for further study • Partial credit using steps • Steps may give strategy • May be able to test areas not currently tested CUE

More Related