Improving Education through large scale testing?

1. Improving Education through large scale testing? April 5th, 2012 Society for the Advancement of Education Presenter: Salaeya Butt Based on the study “Improving Education through Large Scale Testing: ?” Authors: Abbas Rashid, Ayesha Awan , IrfanMuzzaffar & Salaeya Butt

2. Overview Overview • Assessments in Pakistan • Why assess students? • What kinds of assessments are conducted? • What does assessment data tell us about student performance? • Key findings of PEC ExamsStudy • Findings • Recommendations • Way Forward • Making decisions on what kinds of assessments are needed • How data can be more effectively used

3. Introduction • If quality education is about imparting a measure of agreed upon knowledge and a set of skills to children, • How are we to determine that such a process is indeed in place? • How much do children in our schools really know? • More than most, it is the teachers who need to know the answers so that they can concentrate their time and attention where it is most needed

4. Why AssessStudents? • To ensure conformity to minimal standards of education • To improve the examination system which also has potential to influence teaching and learning downstream. • To enable critical thinking, analytical and reflective skills in students • To measure student performance on learning outcomes in order to identify needs as well as design policies and interventions

5. Pakistan Overview: Examinations Examinations in Pakistan

6. Pakistan Overview: Assessment Systems • Government Supported Assessments Pakistan

7. Assessment Surveys Pakistan

8. What does assessment tell us about performance? • The average student in public schools in Punjab is performing below acceptable levels of proficiency (PEAS) • In Math 2010 Class-V PEC exam one in every five students scored less than 20. One single digit addition & subtraction Curriculum Standard 2: Children of Grade 1 & 2 should be able to add and subtract up to 3 digit numbers Curriculum Standard 3: In grade 3 to 5, children should be able to multiply and divide upto 6-digit numbers by 2 and 3 digit numbers. (19%)

9. PEC Exams Study (SAHE)Key Findings

10. Exam Design? Good Exam Design Reliable assessment tools require compliance with international standards which are • Adequate coverage of content and the taxonomical objectives implicit in SLOs • Technically equivalent: Comparability of scores across different versions of the exam and from year to year • Reliable: Test items should behave the same way with different populations of test-takers • Valid: Test items actually measure what they purport to measure • Pilot: Items must be piloted to develop empirical data and to conduct subsequent relevant statistical analysis • Technical review & Content review

11. PEC Findings • Item Developers receive training by PEC but still greater in house capacity is needed • Government school teachers have little training in technical aspects of paper setting (content and other types of validity) and marking (reliability). • PEC is developing documented criteria for acceptable coverage of SLOs this year. However, there is no item bank. • Panel consisting of content experts at the specific grade level, as well as psychometric experts are required to review the item writers work • Item writers are required to indicate desired psychometric properties and difficulty levels. But difficulty levels cannot be established by writer’s judgment alone, a pilot is essential.

12. Why Focus on Exam Conduct & Marking? • PEC examinations are conducted on a large scale, with thousands of exam centers and over 2.5 million students across the Punjab • To ensure quality in results it is important that: • Exams are conducted in an efficient and transparent manner • Exam paper open-ended questions are marked accurately to ensure quality in exam results • Marking and scoring scheme does not fluctuate beyond an agreed percentage point

13. PEC Findings • Conduct • The schools designated as exam centres often lack of minimum requisite facilities • Existing students are displaced to make room for exam candidates • Schools experience trouble ensuring adequate number of supervisory staff • Having more than one paper per day • Paper Marking • Lack of adequate subject specialization among examiners • Provincial government adjusts pass percentage to prevent large number of students from failing

14. Data Interpretation & Dissemination • One of the primary purpose of this examination system is to inform the teaching and learning process and improve the delivery of quality education. • This is only possible when the results of this exam are widely disseminated and shared in meaningful manner with a variety of stakeholders.

15. PEC Findings • Information on student scores widely disseminated include: • Gazette (aggregate score) • Student score card (detailed marks) arrive late (2-3 months later) and in some instances not at all • Website is not fully functional and shuts down • Dissemination is more efficient in urban districts and less efficient in smaller rural districts • No widely disseminated information on district based and SLO based analysis for teaching • Only use is of exam results is for admission into next grade.

16. Recommendation - Develop Proficiency Ranges • Detailed feedback should be provided to all stakeholders about the quality of student performance beyond the pass/fail and grade categories. • The scores will be most useful if they provide information about proficiency of students in relation to the SLOs. Consider developing ranges to provide information about student proficiency levels.

17. Recommendation - Delineating Role & Ensuring Ownership • Decision-making atleast with regard to conduct of exam should be devolved to the district level • Developing capacity for item development • An examination as carried out by the PEC should then ideally be a validation exercise. • In principle, assessments are best carried out on a continuous basis and by the teacher who, in effect, ensures that learning takes place for it to be meaningfully assessed. • Given the very large number of exam centers, district and local-level decision-making in this regard should be encouraged • Frequency and regularity of testing from the point of view of the expenditures as well as the burden these tests impose should be kept in mind

18. Communication Strategy Examples • Recommendation - Communication Strategy Examples • Province & District • Develop and disseminate overall report of findings for province and district performance. • Hold dialogue with relevant district officials on uses of data • Schools & Teachers • Report on student performance findings and their relevance for teaching according to key SLOs • Teacher Education Institutions • Feedback relevant to teacher training based on SLOs • Session on current year’s exam data in annual training • Curriculum & Textbook Authorities • Feedback on performance according to relevant SLOs

19. Implications

20. Adequately gauging learning Achievements? • There are technical challenges related to determining whether the assessment actually measures the learning outcome • Technically sound tests are required • Exam boards in Pakistan spend a disproportionate amount of money on invigilation to control malpractice and a fraction of that on improving the quality of examinations or their management. • Even if the grades are the same, it is difficult to establish that they actually reflect similar standards of student achievement • Consequently, there is an underlying decline in standards of student learning and competence.

21. Appropriate apparatus for Policy reform? • A process of deliberation to ensure that the diverse assessment related streams are meaningfully brought together to inform policy is needed to address the following questions: • How can achievement data be used to develop policies and comparison across different groups of students with varied characteristics? • What combination of formative assessment tests and summative exams are actually required at different levels of schooling ? • What kind of assessment is needed at different levels of school education? • At the provincial level a coordinating arrangement or forum is needed to ensure that assessment needs have been met

22. Issues to be addressed? Conclusion • When assessment does not just drive instruction for learning, the important objective of the education system is ceased. • Hence, examinations have multiple audiences and multiple purposes, but they must essentially be seen as serving the curriculum and instruction and not driving it.