1 / 26

Setting Minimum Standards for Outcomes in Education Programs

This study aims to establish minimum standards for educational outcomes in primary and secondary schools, focusing on basic competencies in French and mathematics. The study includes the development of assessment items, experimentation, and analysis of test results to determine the cut-off points for mastery of basic competencies.

eliaso
Download Presentation

Setting Minimum Standards for Outcomes in Education Programs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Setting minimum standards in the frame of outcomes’ indicators : methods, interest and use.Thierry Rocher Ministry of Education – DEPPFrance

  2. Context MISSIONS OBJECTIVES EDUCATION PROGRAMMES LEAD ALL THE PUPILS TO THE MASTERY OF BASIC COMPETENCIES AT THE END OF PRIMARY SCHOOL LABOR HEALTH PUBLIC PRIMARY SCHOOL … … PUBLIC SECONDARY SCHOOL … • LOLF (fundamental law of finances) : a new public budget framework • Indicators

  3. Indicators • Proportions of pupils who acquired the basic competencies (in reference to standards of ‘socle commun’) : • In french and maths • At the end of primary school (grade 5) and at the end of lower secondary school (grade 9) • Indicators calculated by : • Type of school : ‘priority education 1’, ‘priority education 2’, ‘public except PE’, ‘private’ • 30 educational regions (« académies »)  i.e. 120 « strata » • These indicators must be comparable from one year to another • In order to asses the completion of objectives • Ad hoc programme of students’ assessments

  4. Constraints • Timeline • Beforesummer for the vote of the public budget at the Parliament • Cost • No supplementary funds • Trends • Linking procedures • Implications : • MCQ for fast and cheap marking (i.e. implies no writing) • Samples (but huge if regional indicators) • Pre-experimentation of a large pool of items

  5. Process Definition of basic competencies, framework  Items development  Experimentation 2006  Setting the cut-off points  Test 2007 = year 0  …

  6. Framework • Existing standards (2006) : « socle commun de connaissances et de compétences ». Not yet measured • Basic competencies in reference to the ‘socle’ and defined by responsible for pedagogical policy : • At the end of primary school, at the end of lower secondary school, in french and maths • exemple : maths grade 5

  7. Test development • Working groups (teachers, inspectors, ministry’s representatives) • Mathematics (1 sequence): • Grade 5 : 221 items, BI design, 13 blocks of 17 items, 13 booklets of 68 items • Grade 9 : 169 items, BI design, 13 blocks of 13 items, 13 booklets of 52 items • French (2 sequences) : • Grade 5 : 2 « parallel » versions (resp. 67 and 75 items), 2 booklets to control the order effect • Grade 9 : 2 « corpus » (resp. 77 items and 95 items), 4 booklets with 1 corpus and a half

  8. Experimentation of test and procedures • Sampling in each of the 30 « académies » : • In total : 15000 grade 5 pupils in about 500 primary schools and 15000 grade 9 pupils in about 500 lower secondary schools • Engineering : • Standardized procedures (sampling, administration, controls, etc.) • Important role of local educational and statistical offices • Survey well perceived at a school level • High participation rates (around 90%) • Final samples : about 7000 pupils by grade and subject Experimentation

  9. Preliminary analysis : • Fatigue and training effects • Behaviour of guessing responses • Discrimination • Dimensionality • Adequacy with standards (“socle commun”) • Cleaning • Maths grade 5 : 50 on 221 • Maths grade 9 : 24 on 169 • French grade 5 : 6 on 74 (comprehension) • French grade 9 : 29 on 172 Analyses and cleaning

  10. Setting the cut-off points • How to set the cut-off point ? • Arbitrary from the data (‘relative’) For example, rate of 15% below Problem : relative, no link with the content • Arbitrary from theoretical judgement (‘absolute’) For example, a priori cut-off point Problem : possible gaps with the observed performance • Confrontation between experts’ judgements and students’ performance

  11. Setting the cut-off points • The « miracle method » does not exist • Two methods were considered: • First : from a global point of view (Hofstee) • Second : from a virtual ‘bordeline’ group of students (Angoff) • Application : • One day meeting with each group • Individual work in the morning, confrontation and discussion in the afternoon

  12. minimum cut-off maximum cut-off % pupils who have not acquire the basic competencies data curve Setting the cut-off points – 1st method % maximum - % non mastery judgement zone % minimum - observed score cut-off

  13. Question 1 : « On the basis of your professional experience, according to you, what are the minimum and the maximum percentages of the pupils who have not acquired the basic competencies [at grade 5/9 in french/maths] ?» • Question 2 : « What is the score – number of correct answers – below which one may consider that pupils have not acquired the basic competencies ? More precisely, give a minimum « acceptable » score (lenience) and a maximum « acceptable » score (harshness). » Setting the cut-off points – 1st method

  14. Finally, two groups of pupils : • Pupils who have acquired basic competencies • Pupils who have not acquired basic competencies • Reality is not as simple • ‘Borderline’ group of pupils • Aim : to find the item level of difficulty which is the best adapted to these ‘borderline’ pupils Setting the cut-off points – 2nd method

  15. Ranking of the items by their level of difficulties Setting the cut-off points – 2nd method Easy items 1 success 2 . . . Cursor = item from which the ‘borderline’ pupils have difficulties (i.e. 50% success) failure X Difficult items

  16. 1st method : maths grade 5

  17. 1st method : maths grade 9

  18. 1st method : french grade 5

  19. 1st method : french grade 9 (1)

  20. 1st method : french grade 9 (2)

  21. 2nd method : maths grade 5

  22. 2nd method : maths grade 9

  23. 2nd method : french • Grade 5 • Grade 9

  24. Convergence Item difficulty Students’s performance Difficult items Students who have acquired the basic competencies Cut-off point Students who have not acquired the basic competencies Easy items • After discussion within the working groups, cut-off points were fixed : • Maths grade 5 and 9 : around 10% • French : grade 5 – 14%, grade 9 – 21% • Cut-off items identification • Presentation of the results to the responsible of the pedagogical policy (inspectorates) for validation

  25. Item selection • Optimized at the cut-off points with the content constraint • Test 2007 • National sample (not regional) : about 8000 pupils in grade 5 and 8000 in grade 9 • Anchoring on the 2006 data Test 2007

  26. Setting the cut-points : • Not a pure formal analytical process … • … but necessary to legitimate the numbers • Process also useful for discussion with responsibles of pedagological policies • Compromise between expectations and observations • Methodological issues in the measurement of students’ competencies • Next : • What use of these indicators ? Conclusion

More Related