PISA for Development 1 st International Advisory Group Meeting

1. PISA for Development 1st International Advisory Group Meeting Component Terms of Reference for the IC 27 – 28 May 2014 Paris, France EDU/DCD

2. PISA for DevelopmentInternational Advisory Group Meeting Expected Results from Meeting • Agreement on the main components of the Terms of reference for the International Contractor(s) • General approach • Cognitive instruments • Contextual questionnaires • Next steps for managing the tendering process

3. PISA for Development General Approach

4. General Approach - 1 • To focus on the tasks as they relate to the development of the cognitive instruments and of the context questionnaires. • The elements relating to other tasks (e.g. sample design, translation and verification, survey operations, analysis and reporting) are not covered in this discussion and will be added in the final ToR. • The elements relating to approach to OOS are also not covered in this discussion: these will be focus of a separate tendering exercise

5. General Approach - 2 Focus on: the constraints that should be imposed on the bidders for the International Contract(s); the requirements expected of bidders; and questions that the bidders should respond to as part of the tendering process

6. General Approach - 3 principles and components of the ToRwould, once agreed by the International Advisory Group (IAG) of the PISA for Development project, provide the framework for the development by OECD of a complete ToR and associated tendering documents that would serve as the basis for a call to tender by the OECD

7. General Approach - 4 ToR and tendering process will: reflect the general PISA principles; reflect and support the rationale of the countries for participating in PISA for Development; emphasise capacity building; emphasise peer-to-peer learning emphasise consultation and communication with and support for participating countries emphasise track record of bidders in building capacity for student assessment in developing countries

8. PISA for Development Cognitive Instruments

9. ToR – Cognitive Instruments Countries require an assessment that: • Reportsresults on the PISA scale and evidence supports comparability to international PISA results; • Allows students to demonstrate the full range of proficiency levels; and • Adheres to all PISA standards • unless certain modifications are agreed for the implementation of PISA for Development.

10. ToR – Cognitive Instruments Fundamental constraints for bidders: No new cognitive items will be developed Review the secure pool to inform the selection of items based upon their cultural and contextual suitability The targeted test should give a robust and targeted measure of the country's strengths and weaknesses

11. ToR – Cognitive Instruments Fundamental requirements of bidders: Assess students' abilities as they are now, note where it is thought the students’ performance should be. Better targeting but also assess across the full range of the PISA scale. Must ensure sufficient PISA framework coverage and coverage of a full range of proficiency levels.

12. 4 Key elements of the ToR and bidding documents • Review of assessment framework and items • including cross-cultural validity and test targeting; • Test design; • Review of the proficiency levels; • Review of scaling models.

13. Review of assessment framework and items - 1 Constraints for bidders: The selection of all assessment items is based on the PISA assessment frameworks for reading, mathematics and science Any extension of the framework will need to continue to incorporate the original

14. Review of assessment framework and items - 2 Requirements of bidders: To carry out a complete review of the assessment frameworks Consider the implications of PISA 2015 move to a computer-based assessment

15. Review of assessment framework and items - 3 Requirements of bidders: • Need to review PISA’s technical standard on language of instruction? • Processes to review item suitability, translation and verification of test • Variations needed?

16. Review of assessment framework and items – 4. Cross-cultural validity Item-by-country interactions (country DIF). Need to examine the potential impact of these interactions on the validity of reporting. Bidders’ approach to this and proposed solutions.

17. Review of assessment framework and items – 5. Test targeting • Cognitive instruments to be developed from the pool of 337 secure PISA items • Bidders’ proposals for test design should: • deal with the challenge of designing a test that draws upon easier items • But still matches the framework specifications, including with regard to item-types

18. Test design - 1 Requirements of bidders: • To reliably forecast from a targeted test the proportion of participating students who would successfully complete the most difficult items • Achieves the desired test targeting but • provides good coverage of the frameworks • provides results comparable with the main PISA assessment • Propose an equating study

19. Target population • International target population: • 15-year-old students attending educational institutions in grades 7 and higher • Bidders to make provisions to allow a participating country to supplement the sample

20. Review of the proficiency levels Propose how to ensure these are meaningful How to better describe the proficiency of students who perform below the lowest proficiency levels

21. Review of scaling models - 1 Should PISA for Development pursue alternative scaling models that allow features such as: • varying discrimination across items; • dependencies between items clustered in units • guessing; and, • parameter variation (including difficulty) across countries?

22. Review of scaling models - 2 To what extent do the observed item-by-country interactions undermine developing a PISA-like learning metric for developing countries?. How can their proposals for scaling overcome this?

23. Draft Components of the ToR for the IC(s): Contextual Questionnaires Tramonte & Willms Paris, May 28, 2014

24. The 7 themes • Add items on students’ early learning experiences, • Add items on students’ familiarity with the language of the test • Measure parental involvement, social capital and cultural capital • Measure the role of other community members and of types of community • Enhance measure of school attendance • Add items on participation in formal and informal labor market • early learning opportunities • language at home and school • family and community support • quality of instruction • (effective) learning time • student socioeconomic status (SES) • school resources

25. The last two themes – student SES and school resources – should be emphasised in the ToRfor the International Contractor(s). In presenting their proposals, bidders will be asked to consider the guidance provided by the expert paper, to elaborate extensively on the last two themes, and to justify alternative approaches.

26. Implications for the Components of the ToR for the International Contractor(s) The 7 themes will be addressed with theoretical, methodological, and technical discussions in the ToR for the International Contractor(s).

27. Requirement in the enhancement of the questionnaires according to the 7 themes The revised questionnaires must be: • comparable internationally and • consistent with the current PISA frameworks.

28. Structure of the ToR-contextual questionnaires This component of the ToR is articulated in three parts: • Part One refers to the first 5 themes that require minor intervention; • part Two and Three refer to the two critical themes, SES and school resources, that warrant greater attention; • finally, the contextual component of the ToR contains indications on general expectations, specific questions, and underlying considerations on choice of informants and synergy with national assessments.

29. Key elements of the ToR: Part One Five areas for enhancement and measurement: • early learning opportunities, language at home and school, family and community support, quality of instruction, learning time. Four underlying aspects for bidders to respond to: • the best informants; • the core questionnaires; • PfD and its synergy with national assessments; • the collection of data on quality of instruction that are subject specific;

30. Key elements of the ToR: Part Two One area for enhancement and measurement: • Student SES Four underlying aspects for bidders to elaborate on: • offer options to address the extension of SES: adding to PISA ESCS more items at its lower end and/or develop new “poverty-related” measures; • capture the experiences of different countries regarding their own variables for measuring SES; • create a global measure of poverty that can be applied consistently across countries for comparative purposes; • ensure comparability of results of the project on ESCS scale with international results, even if the scale is extended.

31. Key elements of the ToR: Part Two Requirement for bidders: the measure of socioeconomic status (SES) that is developed must be comparable across participating countries and with the results of PISA international assessment, even if the scales are extended. In addition, a measure of SES for PfDshould be: • a reliable and valid measurement of SES within each country; • a tool for accurate assessment of low levels of SES and poverty; within each country and across countries; and • a comparable measure of SES and its variability across countries.

32. Key elements of the ToR: Part Three Area for enhancement and measurement • Measuring school resources OECD and participating countries agreed on the need to integrate the PISA questions with few new measures on school resources that provided data on basic services, didactic facilities, and didactic materials. The bidders should enhance the PISA tools to capture: • availability, conditions, and use of basic services, didactic facilities, and didactic materials; • conditions and quality of school infrastructure; • school infrastructural features, services, and safety.

33. Key elements of the ToR: Part Three Expectations and challenges: • Bidders should propose viable options for maintaining comparability with the main PISA instruments where essentially the same construct is being measured. They should therefore explain the extent to which their proposal will achieve this.

34. Bidders should explain how to capitalise on potential synergies in data collection with on-going national/international assessments and • they should identify who are the optimal respondents about availability, conditions, and use of basic services, didactic facilities, and didactic materials. • Note: this issue relates to the broader discussion on the extension of PISA core questionnaires to accommodate teacher or parent questionnaires (see later).

35. General Expectations: Bidders should provide a detailed account of: • the methods proposed to measures of SES and school resources, as well as to conduct pilot testing for the revised questionnaires; • the strategies to deal with the challenges associated with pilot testing, obtaining reliable and valid data from school administrators, and integrating the new content into the current PISA framework.

36. General Expectations (continued) Bidders should provide a detailed account of how to: • ensure international comparability once new measures are introduced; • maintain comparability with PISA current measures; • allow nationally relevant analyses; • provide data that inform of equity and equality.

37. Expanding on questionnaires and informants. Three questions for the bidders Currently PISA assesses students and schools with two core questionnaires (student and school); parent and teacher questionnaires are optional. • Given the 7 themes that need to be captured in PfD, are the core questionnaires enough? • If new questionnaires have to be introduced, what should they be?Why? • Should any of the core questionnaires be dropped? Why?

38. More specific questions: • Who are the best informants for the 7 themes? • How can bidders insure that the informants provide effectively their information? • How do bidders ensure that the current PISA questions and the new questions are relevant for the partner countries? • How do bidders plan on collecting data on quality of instruction that are subject-specific? • How do bidders propose to capture students’ learning time, in and out of school?

39. On Informants The bidders should discuss the rationale for choosing their informants and the need for expanding the set of core questionnaire to guarantee: • limited number of non-responses or missing data resulting from: • low level of literacy of the respondents; • impoverished living conditions of the respondents • gap between language of the informant and language of the questionnaire. • representation of the variability in family structures; • comparability between and within participating countries; • triangulation of information from different questionnaires; • contained length of the questionnaires

40. PfD and National/International Assessments PfDdata should be a tool and a resource for each participating country. Bidders should propose how to maximize the synergy between PfD and national assessments. While maintaining the core PISA content, they should explain how to ensure: • stability and consistency of data collection within the country; • comparability of data between PfD countries and PISA countries; • validity and reliability of the indices used to conduct meaningful analyses within each country; and • manageable national datasets that can be analyzed by PfD national analytical teams.

41. On sampling and methods Bidders are expected to design national samples that are structured using the same stratification variables used in PISA but also incorporate country specific variables. In the bid, applicants are expected to propose innovative methods of data collection and data merging that allow for effectively linking PfD data to locally-collected national datasets.

42. Call for tender process Roles, Responsibilities and the Process

43. Call for Tender process Set clear evaluation criteria Identify potential bidders Independent experts; Secretariat; IAG; SDG potential bidders

44. Technical Review Process (detail) • The technical criteria reviewed prior to the panel meeting to ensure common understanding among panel members. • Panel members individually score bids on the technical criteria (without knowledge of the budget proposals) prior to convening for the panel meeting. • The panel convene in person to share scores and to discuss reasons for score deductions. • Cost proposals reviewed after all the bids are discussed. • Follow-up questions sent to the top-rated bidders. • The Secretariat drafts the report, incorporating responses to questions, for the panel’s review. • The panel agrees the report and the recommendations therein.

45. Call for Tender process Contract negotiations

46. Best Value for Money • OECD awards the tender based on the concept of best value for money • Best quality to price ratio • Biddersofferingadded value canimprove the quality of the goods and services purchased by OECD • Adds a level of complexitywhichmakes the Tender EvaluationProcess important • EvaluationCriteria balances • Contractcompliance • Performance • Delivery • Ability to meet minimum specifications • Price

47. Timeline and procedures: 2014

48. PISA for Development Capacity Needs Assessment Framework Fernando Cartwright

49. Objective Develop a framework to identify any factors or conditions that are barriers to the successful implementation of PISA, including the production of meaningful information and consumption of information by stakeholders, in participating countries.

50. Needs Assessment Dimensions • Enabling environment • Legislative, political/bureaucratic relationships, cooperating partners • Organization • the National Centre and other sub-national institutions • Individual • National Centre staff, data collection teams, local service providers