1 / 39

Perspectives on design and instruments

Explore perspectives on designing assessment instruments to improve opportunity to learn (OTL) and ensure curriculum sensitivity in international assessments like PISA and TIMSS. Understand the importance of a structured assessment framework for defining what is measured, ensuring fair opportunities for all students to learn, and verifying curriculum alignment.

jganz
Download Presentation

Perspectives on design and instruments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Improving opportunity to learn (OTL) and curriculum-sensitive measurement in international assessments Perspectives on design and instruments Stephan Daus

  2. Outline

  3. Outline

  4. Multiple countries • Eg. PISA, TIMSS, PIRLS, SAQMEC, ICILS Eg. mathematics, science, reading • International large-scale student assessments • All countries use same test • Contextual information • Mostly cross-sectional

  5. Some examples

  6. A structured and specific assessment framework • Organizes development of items and assembly of test • Clarifies what we measure • Creates stability of measurement • Allows investigation of to what degree assessment is linked to the curriculum

  7. Outline

  8. Opportunity to learn Curriculum-sensitive measurement Purposes and practical issues

  9. Example 1 Grade Grade Grade 10 9 8 7 6 5 4 3 2 1 10 9 8 7 6 5 4 3 2 1 10 9 8 7 6 5 4 3 2 1 Norway’s grade shift: A reminder of difficulty in assuming similarity across countries TIMSS 2011 TIMSS 2015 TIMSS 2019

  10. Example 2 Eighth graders performance on the TIMSS 2011 topic «electricity» High International average Norway Other topics: Daus, Nilsen & Braeken, 2018 Low

  11. Opportunity to learn Curriculum-sensitive measurement

  12. Ensure students have fair opportunities to learn across groups we compare Ensure curriculum-oriented assessment is curriculum-sensitive Verify assumptions

  13. How does assessment overlap with curriculum? • Specificity is needed 1 • Curriculum in country • International assessment Covered Not covered (not tested) 1 Schmidt, Jakwerth & McKnight, 1998; Schmidt & McKnight, 1995

  14. Brief historical review of opportunity to learn variables at the national level

  15. National level • International • assessment • National curriculum • Survey of Mathematics and Science Opportunities project TIMSS 1995

  16. National level • International • assessment • National curriculum TIMSS 2015

  17. Brief historical review of opportunity to learn variables at the classroom level

  18. % of students having had the ‘opportunity to learn’ the item 2 FIMS / FISS 1965 / 1970 First International Mathematics/Science Study 2 Critique of item-focus: McDonnell, 1995; Porter, 1991; Muthén et al., 1991

  19. Number of lessons per topic % of students will get the item correct?2 Coverage of the contents needed to answer the item (whether/when) Emphasis of various types of problems SIMS 1982 Second International Mathematics Study 2 Critique of item-focus: McDonnell, 1995; Porter, 1991; Muthén et al., 1991

  20. % of students will get the item correct?2 Coverage of the contents needed to answer the item (whether/when) SISS 1984 Second International Science Study 1 Critique of item-focus: McDonnell, 1995; Porter, 1991; Muthén et al., 1991

  21. What was taught last lesson? Number of lessons per topic Is example item appropriate for class? Coverage of the contents needed to answer the item (whether/when) Relative emphasis of topic Out-of-school learning opportunities? Which textbook is used? TIMSS 1995 Third International Mathematics and Science Study

  22. Coverage of the contents needed to answer the item (whether/when) TIMSS 2015 TIMSS 2003 TIMSS 2007 TIMSS 2011 TIMSS 2019 TIMSS 1999 Trends in International Mathematics and Science Study

  23. Students’ familiarity with math topics PISA 2012

  24. Outline

  25. Evidence of relationship Stronger relationship between OTL in the classroom and achievement when3 • using FIMS, TIMSS 1995 or PISA 2012 • in mathematics (than science) • with high-specificity analyses (item/topic and person level) • using (quasi-) longitudinal designs • for lower levels of student abilities 3 Suter, 2017; Schmidt et al., 2001; Luyten, 2016; Schmidt et al., 2015; Yang-Hansen & Strietholt, 2018; Daus & Braeken, 2018; Li et al, 2014; Carnoy et al, 2016

  26. Outline

  27. Balancing international and national needs Relevance of assessment framework Classroom OTL instruments Longitudinal features Appropriate secondary analysis methods

  28. Balancing international and national needs • TIMSS assessment • Curriculum of country A • Curriculum of country C • Continue negotiations • National options for items 4 • Curriculum of country B 4 Similar to: Rutkowski, Rutkowski & Liaw, 2018

  29. 2. Relevance of assessment frameworks Updating the Survey of Mathematics and Science Opportunities from 1995? NoS: Nature of Science SSI: Socio-scientific issues

  30. 3. More precise classroom OTL variables TIMSS 2015 Forces and motion……………..……… TIMSS 1995

  31. 3. More precise classroom OTL variables How about instructional quality of the specific content being taught? 5 Have successfully taught this year strongly disagree agree strongly disagree agree Forces and motion…………………… 5 Wang & Goldschmidt, 1999; McDonnell, 1995; Levin, 2009; Wang, 1998

  32. 3. More precise classroom OTL variables PISA 2012 Mathematics

  33. 4. Longitudinal design Cross-sectional design (repeated every 3-4 years) Longitudinal design following each student Adjacent grade design within each school Achievement Achievement Achievement Student grade 8 9 Student grade 8 Student born 1967 2006 Student grade 7 8 1995 2015 2011 1980 1981 2019 2020

  34. 5. Appropriate secondary analysis methods • Insufficient to have highlydetailed variables if not put to use Acceptable way to summarize OTL variables 6 Questionable way to summarize OTL variables ‘Opportunity to learn electricity’ ‘Opportunity to learn science’ Textbook space on electricity Electricity taught last lesson Coverage of electricity Coverage of forces and motion Coverage of light and sound Time spent on electricity 6 See e.g. Schmidt et al., 2001

  35. Collection and analysis of OTL variables needed when using international assessments to compare groups or say something about the curriculum.

  36. Current design and instrument features need improvements • Balance needs of international comparisons with national curriculum-specificity • Consider updating assessment framework and its organizing principles to tomorrow’s curriculum reforms • Revisit OTL instruments used in TIMSS 1995 and reconsider OTL measures • Reintroduce longitudinal designs (more often) • Ensure proper analytical approaches when checking for curriculum-sensitivity

  37. References I • Carnoy, M., Khavenson, T., Loyalka, P., Schmidt, W. H., & Zakharov, A. (2016). Revisiting the relationship between international assessment outcomes and educational production: Evidence from a longitudinal PISA-TIMSS sample. American Educational Research Journal, 53(4), 1054–1085. • Daus, S., & Braeken, J. (2018). The sensitivity of TIMSS country rankings in science achievement to differences in opportunity to learn at classroom level. Large-scale Assessments in Education, 6(1), 1–31. • Daus, S., Nilsen, T., & Braeken, J. (2018). Exploring Content Knowledge: Country Profile of Science Strengths and Weaknesses in TIMSS. Possible Implications for Educational Professionals and Science Research. Scandinavian Journal of Educational Research. • Klieme, E., Pauli, C., & Reusser, K. (2009). The Pythagoras study: Investigating effects of teaching and learning in Swiss and German mathematics classrooms. In T. Janík & T. Seidel (Eds.), The power of video studies in investigating teaching and learning in the classroom (pp. 137–160). Münster, Germany: Waxmann. • Levin, H. (2007). On the relationship between poverty and curriculum. North Carolina Law Review, 85, 1381–1418. • Li, H., Qin, Q., & Lei, P.-W. (2014). An Examination of the Instructional Sensitivity of the TIMSS Math Items: A Hierarchical Differential Item Functioning Approach. Educational Assessment, 22(1), 1–17. • Luyten, H. (2016). Chapter 5: Predictive Power of OTL Measures in TIMSS and PISA. In J. Scheerens (Ed.), Opportunity to Learn, Curriculum Alignment and Test Preparation: A Research Review (pp. 103–119). Dordrecht, the Netherlands: Springer. • McDonnell, L. M. (1995). Opportunity to Learn as a Research Concept and a Policy Instrument. Educational Evaluation and Policy Analysis, 17(3), 305–322. • Muthén, B. O., Kao, C.-F., & Burstein, L. (1991). Instructionally Sensitive Psychometrics: Application of a New IRT-Based Detection Technique to Mathematics Achievement Test Items. Journal of Educational Measurement, 28(1), 1–22. • Porter, A. C. (1991). Creating a System of School Process Indicators. Educational Evaluation and Policy Analysis, 13(1), 13–29.

  38. References II • Rutkowski, D.; Rutkowski, L. & Liaw, Y. (2018). Measuring Widening Proficiency Differences in International Assessments: Are Current Approaches Enough? Educational Measurement: Issues and Practice, 37(4), 40-48. • Schmidt, W. H., & McKnight, C. C. (1995). Surveying Educational Opportunity in Mathematics and Science: An International Perspective.Educational Evaluation and Policy Analysis, 17(3), 337–353. • Schmidt, W. H., Burroughs, N. A., Zoido, P., & Houang, R. T. (2015). The Role of Schooling in Perpetuating Educational Inequality: An International Perspective.Educational Researcher, 44(7), 371–386. • Schmidt, W. H., Houang, R., Cogan, L. S., & Solorio, M. L. (2018). Schooling Across the Globe: What We Have Learned from 60 Years of Mathematics and Science International Assessments. Cambridge: Cambridge University Press. • Schmidt, W. H., Jakwerth, P. M., & McKnight, C. C. (1998). Curriculum sensitive assessment: Content does make a difference. International Journal of Educational Research, 29(6), 503–527. • Schmidt, W. H., McKnight, C. C., Houang, R. T., Wang, H., Wiley, D. E., Cogan, L. S., & Wolfe, R. G. (2001). Why schools matter: a cross-national comparison of curriculum and learning. San Francisco, CA: Jossey-Bass. • Suter, L. E. (2017). How international studies contributed to educational theory and methods through measurement of opportunity to learn mathematics. Research in Comparative and International Education, 12(2), 174–197. • Wang, J. (1998). Opportunity to Learn: The Impacts and Policy Implications. Educational Evaluation and Policy Analysis, 20(3), 137–156. • Wang, J., & Goldschmidt, P. (1999). Opportunity to Learn, Language Proficiency, and Immigrant Status Effects on Mathematics Achievement.The Journal of Educational Research, 93(2), 101–111. • Yang Hansen, K., & Strietholt, R. (2018). Does schooling actually perpetuate educational inequality in mathematics performance? A validity question on the measures of opportunity to learn in PISA. Zdm, 1–16.

More Related