Benchmarking to International Standards TIMSS & PIRLS: A Bridge to School Improvement

1. Benchmarking to International StandardsTIMSS & PIRLS: A Bridge to School Improvement Ina V.S. Mullis and Michael O. Martin, DirectorsTIMSS & PIRLS International Study Center Boston CollegeInstitute of Education SciencesMay 30, 2008 This paper is intended to promote the exchange of ideas among researchers and policy makers.  The views expressed in it are part of ongoing research and analysis and do not necessarily reflect the position of the National Center for Education Statistics, the Institute of Education Sciences, or the U.S. Department of Education.

2. What Are TIMSS and PIRLS? International Assessments of Educational Achievement TIMSS • Mathematics and science • Fourth and eighth grades PIRLS • Reading • Fourth grade

3. Why TIMSS and PIRLS? Provide comparative information about educational achievement across countries in relation to the approaches used to improve teaching and learning in mathematics, science, and reading • Monitor curricular implementation and effectiveness • Identify most promising instructional practices • Measure progress in achievement over time and across grades

4. Trends Over Time Grade 4 • TIMSS in 1995, 2003, 2007, 2011… • PIRLS in 2001, 2006, 2011… Grade 8 • TIMSS in 1995, 1999, 2003, 2007, 2011… Grade 12 • TIMSS Advanced in 1995, 2008…

5. TIMSS Benchmarking U.S. States and Districts 1995 First in the World Consortium, Colorado, Illinois, Minnesota 1997 Missouri and Oregon 1999 13 states and 14 districts and consortia 2003 Indiana 2007 Massachusetts and Minnesota

6. IEA’s Curriculum Model: An Educational Perspective on Educational Issues

7. What Data Are Collected? Intended Curriculum • Participating countries, curriculum experts • Routinely published in Encyclopedia Implemented Curriculum • Teachers and principals of participating students • The students themselves Attained Curriculum • The mathematics, science, and reading tests • Routinely published in International Reports

8. 13 above average, U.S. • 3 about average • 9 below average • Wide range of achievement – across and within countries

9. 13 above average7 about average, U.S.25 below average • Wide range of achievement – across and within countries • Range between 5th and 95th percentiles approximately 270-300 • Average in Singapore same as 95th percentile in lowest countries

10. Widespread Participation TIMSS 2007 Grade 8 • 50 Countries • 242,000 Students TIMSS 2007 Grade 4 • 37 Countries • 183,000 Students PIRLS 2006 Grade 4 • 40 Countries • 215,000 Students

11. Comprehensive Content Coverage Mathematics Framework Grade 4 – 179 items (192 score points) • Number (50%) • Geometric shapes and measures (35%) • Data display (15%) Grade 8 – 215 items (238 score points) • Number (30%) • Algebra (30%) • Geometry (20%) • Data and chance (20%)

12. Comprehensive Content Coverage Science Framework Grade 4 – 174 items (194 score points) • Life science (45%) • Physical science (35%) • Earth science (20%) Grade 8 – 212 items (240 score points) • Biology (35%) • Chemistry (20%) • Physics (25%) • Earth science (20%)

13. Mathematics and Science Cognitive Domains – Grades 4 & 8 Reasoning (20-30%) • Analyze, integrate/synthesize, generalize, hypothesize/predict, draw conclusions, justify, solve non-routine problems and conduct investigations Applying (35-40%) • Compare/contrast, model, represent, use relationships and concepts to solve problems Knowing (30-40%) • Recall, recognize, classify, define, measure

14. Comprehensive Coverage PIRLS Framework – Grade 4 Purposes for Reading • Reading for Literary Experience (50%) • Reading to Acquire and Use Information (50%) Processes of Comprehension • Retrieval and Straightforward Inferencing (50%) • Interpreting, Integrating & Evaluating (50%) Assessment: 10 passages – 126 items (167 score points)

15. Comprehensive Background Information About Contexts for Teaching and Learning Questionnaires: • Country – intended curriculum • Student – home and classroom experience • Teacher – implemented curriculum, education and preparation, instructional practices • School – climate, resources, composition • Parent (PIRLS) – early literacy activities, home resources, parental attitudes toward reading

16. Process for Updating Frameworks Goal: Include topics regarded as important in a significant number of countries • Recommendations by subject matter experts • e.g., committees, NSF initiative for TIMSS 2003 • Questionnaires completed by participating countries • Topics included in their curricula • Desirability of assessing particular topics • Iterative process of reviewing and revising successive drafts

17. Aspects of Comparative Validity • Are target populations comparable? • Was sampling conducted properly? • Are translations comparable? • Were the tests administered appropriately? • Was scoring done correctly? • Are the data comparable?

18. In Conclusion… • Much has been learned about effective education since the pioneering days • Different countries use different approaches but an effective educational system always requires enormous effort • High percentages of students completing high school, and taking advanced courses • A rigorous and progressive curriculum • Well-prepared teachers • Resources for facilities and materials • Students ready to learn and encouraged by society

20. Benchmarking to International StandardsTIMSS & PIRLS: A Bridge to School Improvement Ina V.S. Mullis and Michael O. Martin, DirectorsTIMSS & PIRLS International Study Center Boston CollegeInstitute of Education SciencesMay 30, 2008