1 / 31

Assessing assessment: the role of student effort in comparative studies

Assessing assessment: the role of student effort in comparative studies. Ray Adams Jayne Butler. Participation. 2000 OECD 28, non-OECD 4 PISA+, 11 non-OECD 2003 OECD 30, non-OECD 12 2006 OECD 30, non-OECD 28. Content. Coverage of domains Reading literacy

mauve
Download Presentation

Assessing assessment: the role of student effort in comparative studies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Assessing assessment: the role of student effort in comparative studies Ray Adams Jayne Butler

  2. Participation • 2000 • OECD 28, non-OECD 4 • PISA+, 11 non-OECD • 2003 • OECD 30, non-OECD 12 • 2006 • OECD 30, non-OECD 28

  3. Content Coverage of domains • Reading literacy • Mathematical literacy • Scientific literacy

  4. Methods Pencil and paper test • Multiple choice questions • Constructed responses Student background questionnaire School Principal survey

  5. Reading literacy Mathematical literacy 2000 2003 2006 Scientific literacy CYCLE OF MAJOR DOMAINS

  6. PISA outcomes • Profiles of knowledge and skills • Contextual indicators • Trends and league tables

  7. Instructions to students before the test You have been chosen to take part in an important international education study. This study is called the Programme for International Student Assessment, ‘PISA’ for short. Its goal is to find out what students your age all around the world know about reading, mathematics and science. There are about <number of> students representing <country>. Around the world there are about 200,000 students involved, from more than 7000 schools in 40 countries. The results of the study will help countries determine what students are learning. Because the study may affect students all over the world in the future, we ask that you do the very best that you can.

  8. Instructions to students at the end of the cognitive session Please stop. Now turn to the last page or so in your booklet, where there is a question about calculator use and a question about effort. Please answer these now, and then close your booklet.

  9. The Effort Thermometer How much effort did you invest?

  10. Please try to imagine an actual situation (at school or in some other context) that is highly important to you personally, so that you would try your very best and put in as much effort as you could to do well.

  11. Combined sample sizes for Country X and Y

  12. Percentage of invalid cases

  13. New variable Relative Effort = School mark effort – PISA effort Idealists: negative relative effort Supporters: zero relative effort Realists: positive relative effort Cynics: extreme relative effort

  14. Relative Effort Distribution

  15. Relative Effort Distribution

  16. Relative Effort Distribution

  17. Relative Effort Distribution

  18. Significant decline in effort Significant increase in effort Means for Relative Effort Note: higher scores mean lower effort

  19. Means for Relative Effort by gender 40 16 23 23 Note: higher scores mean lower effort

  20. realists realists cynics cynics idealists idealists supporters supporters diligent realists diligent realists Country X 2003 2000 Plausible value in Reading Relative effort

  21. Country Y 2000 2003 Plausible value in Reading cynics supporters cynics supporters realists idealists realists idealists diligent realists diligent realists Relative effort

  22. Relative effort by gender for reading Female Male Plausible value in reading idealists realists cynics supporters diligent realists Relative effort

  23. Transformed mean scores for reading

  24. Does effort count?

  25. Males Consistent reduction in performance after adjustment for effort Decrease rather than increase in performance after adjustment for effort

  26. Females Relatively consistent performances, both with and without adjustment Improvement (some, but not all) accounted for by effort

  27. Country X Decline in boys over time leading to increased gender differences. Effort not affecting the pattern

  28. Country Y Effort explains some of an increase in female performance Adjusted results show a widening of gender differences that is more consistent with Country X Effort disguises a decline in male performance

  29. Summary • Effort is related to achievement but the relationship is non-linear • Effort is less in Country Y than Country X but Country Y improved its effort investment from 2000 to 2003. • Effort is less for boys than girls • Effort explains some but not all of the differences between the achievement of boys and girls • Effort explains some but not all of the improved performance of Country Y

  30. Future Direction • Compare students matched by effort • Compare effort distributions across countries • Examine variance explained by effort • Explore group differences after controlling for effort

More Related