PISA for Development Technical Strand 1: Contextual questionnaires Overview of PISA instruments

1. PISA for Development Technical Strand 1: Contextual questionnaires Overview of PISA instruments (constructs, indices and variables) EDU/DCD

2. PISA for DevelopmentContextual questionnaires • Purpose of PISA contextual questionnaires • How are they used? • How they are developed • What will PISA for Development seek to do? • Questions for discussion

3. PISA for DevelopmentPurpose of PISA contextual questionnaires • Student and school questionnaires are part of the core assessment design of PISA • Countries may also opt for additional questionnaires: • Parents, Teachers, Health, Time Use, ICT familiarity, Career… • Studentquestionnaires: ~ 30 minute* stand-alone document (students respond after test questions) • School questionnaires: ~ 30 minute stand-alone document (school authority responds before or on day of testing) • * Rotated questionnaires in 2012 to cover additional material

4. PISA for DevelopmentPurpose of PISA contextual questionnaires • Contextualise the performance results in reading, mathematics and science* • Allow analyses of equity issues across sub-groups and populations • Provide indications of possible policy interventions • Some information collected is used to scale cognitive assessment data • Country-specific questions can be included for additional analysis • * Additional domains such as financial literacy and problem-solving

5. PISA for DevelopmentPurpose of PISA contextual questionnaires • Have been used in 74 countries/economies • In more than 40 languages • Publicly available • PISA 2009 Student Questionnaire • PISA 2012 Student Questionnaire • PISA 2012 School Questionnaire • PISA 2009 School Questionnaire

6. PISA Contextual QuestionnairesHow they are used This is the PISA Analytic Framework that guides data collection and analysis Outputs and Outcomesimpact of learning Policy Leversshape educational outcomes Antecedentscontextualise or constrained policy Individual learner Quality and distribution of knowledge & skills Individual attitudes, engagement and behaviour Socio-economic background of learners Quality of instructional delivery Student learning, teacher working conditions Instructional settings Teaching, learning practices and classroom climate Levels of an education system The learning environment at school Community and school characteristics Output and performance of institutions Schools (other institutions) Social & economic outcomes of education National educ., social and economic context Country or system Structures, resource allocationand policies

7. PISA Contextual QuestionnairesHow they are used This is how the PISA Analytic Framework is operationalised for data collection and analysis • Main data collection • Student Questionnaire, School Questionnaires and System Level* Variables included in instruments: *Options include questionnaires geared for Teachers, Parents, Health, Time Use, ICT familiarity, Career

8. PISA Contextual QuestionnairesSystem-level data collection • Standards and procedures for data collection in PISA • Questionnaires for non-OECD economies • Panel of experts to complete some data collection These are examples of the types of information collected at the system level: • Ratio: students / teaching staff • Spending /student • Instruction time (by age) • Teachers’ salaries (by education, experience) • Evaluation & accountability • Educational stratification • Decision-making at different levels • School choice • Changes in education policy • Others (…) Sample of relevant areas to address in developing contexts • Within-country disparities • Quality, supply and distribution of educational materials and resources • (Extra-)tuition • Parental support • Contractual arrangements of school teaching staff

9. PISA Contextual QuestionnairesSchool questionnaires • School resources, policies and practices This is the type of information collected through the school questionnaires, that can then be cross-referenced with other sources These are the constructs that will be reviewed as part of the technical work of PISA for Development • Grades served • Public-Private • Funding sources • Community type • Enrollment (boys, girls) • Average class size • Non-native speakers • Grouping/tracking • Teachers, by qualifications • Teacher shortages • Physical building limitations • Instructional materials • Teacher attitudes, expectations, relations • Computer/network/web access • Absenteeism, disruptions • Teacher morale • Student morale • Teaching environment • Time for special programmes • Non-native speaker offerings • Admissions requirements • Assessment types • Uses of assessment • Assessment of teachers • Decision making authority • External decision makers

10. PISA Contextual QuestionnairesStudent questionnaires Student attitudes, behaviours and approaches to learning These are examples of the types of non-cognitive outcomes on which PISA collects information These are the constructs that will be reviewed as part of the technical work of PISA for Development

11. PISA Contextual QuestionnairesExamples of analysis … How are questionnaire data used for relevant analysis and comparative policy insights? (a few examples from PISA results)

12. Examples of analysis • Range of performance among non-OECD countries PISA Results for different years

13. Mexico Student performance PISA Index of socio-economic background Disadvantage Advantage • Examples of analysisSchool performance and socio-economic background • PISA 2009 Results School performance and students’ socio-economic background within schools Private school Public school in rural area Public school in urban area Student performance and schools’ socio-economic background Schoolswith a similar socio-economicprofile

14. Kyrgyzstan Student performance PISA Index of socio-economic background Disadvantage Advantage • Examples of analysisSchool performance and socio-economic background • PISA 2009 Results School performance and students’ socio-economic background within schools Private school Public school in rural area Public school in urban area Student performance and schools’ socio-economic background OECD average

15. Examples of analysisPolicy findings from contextual information Policy briefs on these issues can be found in the PISA in Focus series

16. Based on the analysis of cognitive outcomes and contextual information • Examples of analysisPolicy insights using contextual information Lessons from PISA from successful education systems (equitable and improvers)

17. PISA Contextual QuestionnairesHow they are developed • ‘Crowd sourcing’ and collaboration • PISA draws together leading expertise and institutions from participating countries to develop instruments and methodologies… … guided by governments on the basis of shared policy interests • Cross-national relevance and transferability of policy experiences • Emphasis on validity across cultures, languages and systems • Frameworks built on well-structured conceptual understandingof assessment areas and contextual factors • Continuous review and updating of assessment frameworks (cycles) • Guided by participating countries and various expert groups formed by leading international experts in different areas related PISA • Technical Advisory Group, Reading Expert Group, Mathematics EG, Science EG, Questionnaire EG, and international contractors

18. PISA Contextual QuestionnairesHow they are developed • PISA 2012 Questionnaires • Linkages between cognitive and non-cognitive outcomes Student questionnaire: Rotated design (total of 65 questions – 25 core and 3 blocks of 27 questions) School questionnaire: 33 questions (5 additional questions on financial education at school) Parental questionnaire: 25 questions ICT questionnaire: 12 questions Education careers: 14 questions

19. PISA Contextual QuestionnairesHow they are developed • Purpose of the PISA Field Trial (for Main Study) Selection of cognitive test items to be included in the PISA Main Study instruments (e.g. countryXcountryDIF, gender DIF) Validation of psychometric equivalence of translated instruments (and adaptations from source versions)  issues addressed Identify constructs from contextual questionnaires (indices and variables) that are associated with performance for inclusion in the Main Study Determine cross-national validity of the questions in the questionnaire Technical standards ensure that data from participating countries are internationally comparable

20. PISA Contextual QuestionnairesWhat will PISA for Development seek to do? • Adapt and enhance contextual instruments to “better fit” diverse contexts found in developing countries while maintaining the comparability with the main PISA scales and international results • Adapt existing constructs (and indices and variables) • Identify and introduce new constructs (indices and variables) • Confirm and validate enhancements through field trials and main data collection in participating countries

21. PISA Contextual QuestionnairesWhat will PISA for Development seek to do? • Examples of variables and indices Student Questionnaire Section 1: ABOUT YOU • Section 2: YOUR FAMILY AND YOUR HOME • Section 3: YOUR READING ACTIVITIES • Section 4: LEARNING TIME • Section 5: YOUR SCHOOL • Section 6: YOUR SCHOOL LESSONS AND SUBJECTS • Section 7: YOUR STRATEGIES IN READING AND UNDERSTANDING TEXTS • Section 8: YOUR VIEWS ON <BROAD SCIENCE> • Section 9: CAREERS AND <BROAD SCIENCE> • Section 10: YOUR MATHEMATICS EXPERIENCES

22. PISA Contextual QuestionnairesStudent questionnaires • Examples of variables and indices • Student Questionnaire SECTION 2: YOUR FAMILY AND YOUR HOME 2009-ST08: Family structure 2009-ST09: Mother’s main job (ESCS, HISEI, BMMJ) 2009-ST10: Mother’s education (ESCS, HISCED, PARED, MISCED) 2009-ST11: Mother’s qualifications (ESCS, HISCED, PARED, MISCED) 2009-ST12: Mother’s employment status (ESCS, HISEI, BMMJ) 2009-ST13: Father’s main job (ESCS, HISEI, BMFJ) 2009-ST14: Father’s education (ESCS, HISCED, PARED, FISCED) 2009-ST15: Father’s qualifications (ESCS, HISCED, PARED, FISCED) 2009-ST16: Father’s employment status (ESCS, HISEI, BFMJ) 2009-ST17: Country of birth for student and parents (IMMIG) 2009-ST18: Age at arrival in country of test 2009-ST19: Language spoken at home 2009-ST20: Home resources (ESCS, HOMEPOS, WEALTH, HEDRES, CULTPOS) 2009-ST21: Family wealth (ESCS, HOMEPOS, WEALTH) 2009-ST22: Books in home (ESCS, HOMEPOS)

23. PISA Contextual QuestionnairesStudent questionnaires – examples of indices PISA index of Economic, Social and Cultural Status (ESCS) • Comparability across countries (not a country-specific income indicator) WEALTH CULTPOSS Highest educational level of parents PARED HEDRES # of books in home Home possessions HOMEPOS Highest occupational status of parents HISEI

24. PISA Contextual QuestionnairesStudent questionnaires – examples of indices PISA index of Economic, Social and Cultural Status (ESCS) Index of family wealth a room of their own, a link to the Internet, a dishwasher (treated as a country-specific item), a DVD player, and three other country-specific items (some items in ST20), and their responses on the number of cellular phones, televisions, computers, cars and the rooms with a bath or shower (ST21). Index of cultural possessions classic literature, books of poetry and works of art (ST20) WEALTH Index of home educational resources For example, availability of a desk, a quiet place to study, a computer that students can use for schoolwork, educational software, books to help students, technical reference books and dictionary (some items in ST20) CULTPOSS HEDRES # of books in home

25. PISA Contextual QuestionnairesStudent questionnaires – examples of indices Self-efficacy in science and mathematics • 2006-ST17: Self-efficacy in science (SCIEEFF) • 2003-ST31: Self-efficacy in mathematics (MATHEFF)  Stays the same in PISA 2012 Relationship between constructs, indices and variables (questions)

26. PISA Contextual QuestionnairesStudent questionnaires – examples of indices Self-efficacy in science • 2006-ST17: Self-efficacy in science (SCIEEFF)

27. PISA Contextual QuestionnairesStudent questionnaires – examples of indices Instrumental motivation to learn science • 2006-ST35: Instrumental motivation to learn science (INSTSCIE)

28. PISA for DevelopmentHow will we do this? • Identify constructs in need of adaptation and important constructs that are missing • Obtain input and advice from experts in the field (grounded in developing-country contexts) • Obtain input and guidance from participating countries • Draw on existing work and empirical evidence • Test the constructs (indices and variables) to ensure comparability with PISA scales

29. PISA for Development Technical Strand 1: Contextual questionnaires

30. PISA for DevelopmentQuestions for discussion • Are the intentions and rationale of this strand of work clear and understood? • Which components of the PISA questionnaires appear to demonstrate the greatest need of adaptation? • Where do you see the largest hurdle to overcome? • What existing studies and evidence should contribute to this strand of work?