Erin Godfrey 1 David Osher 2 Leslie Williams 1 Sharon Wolf 1 Juliette Berg 1 Catalina Torrente 1

1. Cross-national measurement of school learning environments Creating indicators for evaluating UNICEF’s Child Friendly Schools Erin Godfrey1 David Osher2 Leslie Williams1 Sharon Wolf1 Juliette Berg1 Catalina Torrente1 Elizabeth Spier2 Larry Aber1 International Society for Child Indicators 3rd International Conference 28th July, 2011 The University of York 1New York University 2American Institutes for Research

2. Overview of presentation • Introduce importance of schooling and UNICEF’s “Child Friendly Schools” Initiative • Describe methods for creating multi-item indicators to evaluate Child Friendly Schools • Highlight implications for the development of cross-national indicators for use in comparative policy analysis and advocacy efforts

3. Schooling important aspect of child well-being • Schooling is one experience that most children worldwide have in common and the most common means by which societies prepare their young for the future.” (UNICEF, 2009) • More than a billion children worldwide are in school • Primary mechanism for economic and social development of nations

4. Schooling important aspect of child well-being • In the past decade, significant progress has been made towards increasing school enrollment • However, 101 million primary school age children do not attend school regularly (UNICEF, 2009) • Poor educational quality, combined with other family and community factors • constrain enrollment and attendance • thwart learning and development “Never let schooling interfere with your education.” - Mark Twain? Grant Allen?

5. UNICEF’s Child Friendly Schools Initiative • Rights-based perspective rooted in CRC • Practical guide in pursuit of MDG 2 • Embraces a multidimensional concept of school quality addressing the total needs of “the child as learner” • Based in WHO’s mental health initiatives on connectedness, caring and support • UNICEF’s interest in child-, family- and community-centered approaches to school improvement • School effectiveness research (e.g. Battistich & Horn, 1997; Christenson & Thurlow, 2004; Osher et al, 2007; Osher & Kendziora, 2010) “Quality education is education that works for every child and enables all children to achieve their full potential.” (UNICEF, 2009)

6. UNICEF’s Child Friendly Schools Initiative • Based on three key interrelated principles: • child-centeredness • democratic participation • inclusiveness • Seeks to promote change and set quality standards across entire sector • Recognizes local realities and promotes pragmatic “pathways towards quality” rather than a rigid blueprint

7. Global evaluation of CFS • UNICEF supports CFS initiative in 95 countries worldwide • First global evaluation of CFS conducted in 2008 by American Institutes for Research (AIR, 2009) • 6 countries in 4 regions • thePhilippines and Thailand (East Asia and the Pacific Region) • Guyana and Nicaragua (Latin America and the Caribbean Region) • South Africa (East and Southern Africa) • Nigeria (West and Central Africa Region)

8. Global evaluation of CFS • Mixed-methods evaluation of 20-27 child friendly schools in each country • Surveys from students, teachers and school heads • School and classroom observations • Interviews/focus groups with school heads, parents, teachers

9. Global evaluation of CFS • Results provided important information about how key principles of CFS are understood and implemented in multiple contexts • Lack of resources limited the ability to reach other goals • Determine the extent to which CFS objectives are being realized within and across countries • Create tools to monitor future progress

10. Global evaluation of CFS • Relied heavily on multi-item scales • Initial analyses established internal reliability of scales (Cronbach’s) • Advantages to factor analytic techniques in structural equation modeling framework (Carle, Blumberg, Moore & Mbwana, 2011)

11. NYU Collaboration • Deeper psychometric exploration of the validity and reliability of multi-item scales used as indicators of school quality • Can a common core structure of perceptions of school quality be identified across diverse countries, • or are country-specific measures needed to characterize these perceptions? • First step to create cross-national tools to monitor CFS efforts • Provide roadmap to others creating cross-national indicators

12. Initial focus • Student perceptions of quality of learning environment • Students’ experiences of connectedness, safety, respect and engagement in school related to attendance, learning and drop out • Consistent with child-centered approach to indicator development (Ben-Arieh, 2008) • Not as tied to implementation differences • Three countries: South Africa, Nicaragua, the Philippines • Diverse in local context and CFS implementation and evaluation

13. CFS Conditions for Learning Survey: three dimensions • Emotionally supportive climate (15 items,  = .74) • How much students feel listened to, cared about, and helped by teachers and other adults in the school • Challenging, student-centered learning environment(14 items,  = .79) • How strongly students perceive that teachers and other adults in the school encourage the active engagement of students in the learning process and the academic success of all students, and feel that what they are learning is interesting • Safe, inclusive and respectful climate (27 items,  = .83) • Two sub-dimensions: (1) the extent to which students perceive the school to be inclusive and respectful of all types of students and (2) how emotionally and physically safe students feel

14. Sample • 3,072 children • 807 in South Africa • 1431 in the Philippines • 835 in Nicaragua • Grades 5 and 6 (roughly 9-12 years old) • 53% female

15. Analytic approach • Exploratory and confirmatory factor analyses, two steps • Determine best set of items for measuring each domain within countries • More comprehensive information about psychometric properties of items • Relative importance of each item for measuring theoretical construct and substantive interpretation

16. Analytic approach • Exploratory and confirmatory factor analyses, two steps • Determine best set of items for measuring each domain within countries • Formally test measurement invariance to determine whether items and scales operate the same way across countries • Particularly important in creating comparative indicators out of scales • Otherwise, differences in mean levels of scales or relationships between scales could be methodological artifacts, not true differences

17. Analytic approach • Mplus used to handle complexities in data (Muthen & Muthen, 1998-2007) • Can treat indicators as categorical • Can account for nesting of students within schools • Items are reverse-coded to reflect positive association with the construct

18. Step 1 results: Measurement models within countries • Ran exploratory and confirmatory factor analyses separately for each dimension • Emotionally-supportive climate • Challenging, student-centered learning environment • Safe, inclusive and respectful climate • Each dimension had sound psychometric properties with at least a subset of items • Items loaded significantly onto factor • Fit statistics indicated adequate model fit (CFI and TLI ≥ .90; RMSEA ≤ .06; WRMR ≤ 1.0 (Hu & Bentler, 1999; Kline, 2005)) • Each dimension characterized by a core set of items, as well as some differences

19. Step 1 example:

20. Step 2 results: Measurement invariance across countries • Perform formal tests of measurement invariance across countries (Gergorich, 2006; Temme, 2006) • Estimate a model in which item intercepts and loadings are allowed to vary across countries • Estimate a model in which item intercepts and loadings are constrained to be equal across countries • Evaluate change in model fit using the 2 difference statistic • Tests whether each item measures the underlying dimension in the same way across countries

21. Step 2 results: Measurement invariance across countries • For each dimension, we were able to identify a subset of items that demonstrated measurement invariance across countries • These items measure each dimension in the same way across countries • free from differential response bias due to cultural, linguistic or other factors

22. Step 2 example:

23. Summary of results • A common core set of items does characterize student perceptions of school quality across three diverse countries/regions • A subset of these items demonstrated measurement invariance • These can be used to create indicators of school quality that can be compared across countries

24. Insight into substantive interpretation of each dimension • Challenging, student-centered learning environment • How much students perceive that teachers and other adults in the school encourage the active engagement of students in the learning process and the academic success of all students, and feel that what they are learning is interesting • Core set of items does capture all of these features

25. Insight into substantive interpretation of each dimension • Challenging, student-centered learning environment • Emotionally supportive climate • how much students feel listened to, cared about, and helped by teachers and other adults in the school • Core set of items assess how much students feel cared about and helped, but not how much they feel listened to

26. Insight into substantive interpretation of each dimension • Challenging, student-centered learning environment • Emotionally supportive climate • Safe, inclusive and respectful climate • Two sub-dimensions: (1) the extent to which students perceive the school to be inclusive and respectful of all types of students and (2) how emotionally and physically safe students feel • Analyses revealed two different dimensions (1) general feelings of safety, inclusion, fairness and respect and (2) harsh treatment in school

27. Insight into substantive interpretation of each dimension • Challenging, student-centered learning environment • Emotionally supportive climate • Safe, inclusive and respectful climate • Limit interpretation of scales to these aspects, consider adding additional items

28. Important cross-country differences • Able to identify core similarities in the measurement of each dimension • Also found notable differences in the structure and substantive interpretation of each dimension within countries • Reflected each country’s specific cultural, linguistic, and sociopolitical context, as well as differences in CFS implementation and evaluation

29. Implications for CFS • Create two sets of monitoring tools • The core set of invariant items can be used to create tools for cross-national comparisons • This set can be augmented with additional country-specific items that measure unique aspects of school quality • Analyses based on only three countries • Further work will extend analyses to all six countries in global evaluation, others where these items are used • Focus on reliability of scales • Further work to explore concurrent and predictive validity and sensitivity to change

30. Implications for indicator development • Process holds promise for creating reliable and meaningful monitoring tools for international policies and programs with varying conditions for implementation and evaluation • Provide a roadmap for future evaluation efforts • Foster a new more stringent concern with psychometric properties of multi-item indicators