Cautions About Inferences From International Assessments

1. Cautions About Inferences From International Assessments • Kadriye Ercikan • University of British Columbia • This discussion is based on a paper co-authored with Wolff-Michael Roth and Mustafa Asil, • in press in the Teachers College Record

2. Key Inferences From International Assessments • Compare country performances for identifying the countries with the best education systems • Generate insights about effective policy and practice strategies that are associated with higher learning outcomes.

3. Some Examples • TIMMS video study data (e.g., Shimizu, 1999) • Germany’s efforts to make many system level changes as a reaction to the “PISA Shock” (Ertl, 2006) • England’s importing of mathematics teachers from China as a reaction to the country’s low rank in the most recent PISA mathematics assessment (FlorCruz, 2014)

4. Validity of Inferences • Do country rankings really reflect the quality of education in different countries? • What are the fallacies of simply looking to higher performing countries to identify strategies for improving learning in our own countries?

5. Focus • Describe limitations of using country rankings as indicators of quality of education • Discuss limitations of using correlates of higher performance in high ranking countries as a way of identifying strategies for improving education in our home countries

6. Sequence of of Inferences

7. Inference 1: Identifying countries with the most effective education • Implied Propositions

8. Inference 2: Identifying effective policies or practices responsible for high ranks • Implied propositions

9. Inference 3: Emulating policies and practices will lead to improvement in education Implied Propositions

10. Limitations in Identifying Countries with Most Effective Education Systems • There are several sources of incomparability in international assessments that create inaccuracies in comparing and ranking countries • Overall country performance is one of many indicators of education systems • Multiple indicators such as school drop out rates, school climate, student and teacher behavior, and students’ perceptions of benefits of schooling additional criteria to consider • Focusing on performance at the level of entire countries may obscure important within-country differences in rankings for subgroups, such as gender or ethnic groups

11. Graduation Rate Rank versus Reading Score Rank

12. Comparison of gender groups within the five countries on PISA 2009

13. Conclusions • Do higher ranks indicate better education systems? • Comparability • Multi-dimensionality of quality of education • Should we look toward other jurisdictions to copy strategies and practices for the purpose of changing education in another jurisdiction? • Variations in achievement are likely due to much more complex interrelations of cultural, societal, and educational factors. • Taking this or that student variable and education practice in one context and trying to replicate it somewhere else may not result in the aspired to learning outcomes from the original jurisdiction.