National Assessment Findings (Grade 4 & 8)

1. National Assessment Findings (Grade 4 & 8) South Asia Regional Conference on Quality Education for All (October 24, 2007) National Education Assessment System (NEAS) Ministry of Education (Pakistan)

2. Vision & Mission Statement Vision To provide fair and valid Assessments to promote the quality of children's learning in Pakistan Mission To carry out fair and valid National Assessments along with related activities with the overall objectives of enhancing quality, equity & access to education

3. Purpose of National Assessment in Pakistan Monitoring the performance of the education system Providing feed back to improve the quality of services in education system (curriculum, textual materials, teachers’ training and delivery) Setting and disseminating realistic standards of students achievement Identifying correlates of achievement Providing information to policy makers for effective interventions 3

4. National Assessment Schedule Level : Grade 4 and 8 Subjects: Languages, Maths, General Science and Social Studies 4

5. National Sample of NEAS • NEAS uses a two-stage stratified random cluster sample for G-4 and G-8. • Explicit strata are : • Gender: male-female – two explicit strata. • Location: urban-rural – two explicit strata. • Province/Area: four provinces and four areas – eight explicit strata. • Total proposed explicit strata: 2x2x8=32. • The implicit strata are school type and district Contd…

6. National Sample of NEAS • The basic sampling frames are taken from the NEMIS database. • Within a stratum, all schools are included, with selection probability proportional to size (PPS). • Clustering occurs only at the school level. A maximum of 20 and minimum 6 students are selected from each school. • In some rural areas, up to 2% of students in the smallest schools are eliminated from frames, and additional schools are sampled.

7. Geographical Distribution of NEAS Primary Sampling Units

8. Schools Sampled – Planned and Achieved The outcome for achieved school sample at G-8 level(99%) is better than G-4(92%)

9. Students Sampled – Planned and Achieved Students’ participation rate at G-8 level is higher than G-4

10. National Assessment Instruments • Basic competencies in the following core subjects are assessed at both grades levels • Language • Mathematics • Science • Social Studies • The focus of NEAS assessment is to determine the extent which students have acquired reading, writing, problem solving, and critical thinking skills, and not the rote learning from textbooks

11. Development of National Assessment Instruments • The instruments used for national assessment were based on National Curriculum • These instruments were developed through a long process of consultation among national and provincial subject experts therefore, they reflect broad national consensus on both the structure and content • The instruments are based on assessment Frameworks covering three ability strands in each subject. These frameworks are reviewed refined and updated on regular basis to reflect the desired goals of assessment • These instruments undergo rigorous expert review as well as empirical and psychometric evaluation before acquiring the final shape • Several spot/pilot tests are conducted in active collaboration with provincial/area assessment centres to ascertain the reliability of the NEAS assessment instruments

12. Students’ Achievement in National Assessment • Students’ achievement is assessed on the basis of their scores. • These scores provide a measure of how well the students can answer items (questions) based on the National Curriculum. • Scaled scores are used to prevent item selection having an undue influence on student scores. • The scale used in NEAS is similar to TIMSS, SAT, TOEFL etc. The range of scaled scores is between 0-1000. • The achievement scales are constructed so that a student achieving 50% correct marks receives a scaled score of 500and the Standard Deviation of the scale is set at 100. • These scores are used to report trends over time as well as the average and relative performance of different groups of students.

13. National Scores of Grade 4 & Grade 8

14. Achievement of Competencies-Maths (G-4) Measurement, Procedural Knowledge, and Practical Reasoning: Mutual conversion of units of length; Adding/subtracting expressions using problems from every day life involving units of length, mass and capacity. Geometry Procedural Knowledge : Finding perimeter of a square and rectangle. Number sense, Practical Reasoning, and Information Handling : reading and interpreting a line graph Measurement Conceptual Understanding: relationship among units of length; Geometry, Conceptual Understanding: measuring and drawing line segments in MM and CM; Number sense; conceptual understanding, procedural knowledge, : Concept of an improper, mixed and equivalent fractions. The highest common fraction of two or more than natural numbers. • Difficult • Average • Easy

15. Achievement of Competencies-Urdu (G-4) Narrative writing: Use of creativity and power of observation, for example writing a story Persuasive Writing: Using factual information, reasoning and comparison, developing and organizing ideas, using language effectively Example: Letter to Father Reading to perform a task:Reading to accomplish something , using purpose and structure of text; Example: Time Table Knowledge of Language and Vocabulary:Know and distinguish noun, verb, and adjectives; understand the use of idioms Reading for literary experience: Example: Poems and stories Reading for information: Example: essay General understanding :Global knowledge of text, providing title of text, explaining purpose or theme of text • Difficult • Average • Easy

16. Content Domains achieved in Language • Scores are significantly higher on reading as compared to • other domains of Language • Students got the lowest scores on writing domain

17. Location–Wise Achievement Scores

18. Gender-Wise Performance

19. Earthquake effects on Students’ Achievement (Grade 4) • Earthquake significantly affected students’ achievement in all the four subjects. • Students having to migrate after earthquake scored significantly lower on all the four subjects

20. Achievement in Relation to Background Factors

21. Background Data Gathering • During every cycle of assessment, background data are gathered by questionnaire and analysis of fieldworkers’ reports • Questionnaires are given to: • students (including a section for parents), • head teachers and • teachers • Conditions, facilities etc. in schools are observed and recorded by Test Administrators • Background data are correlated with learning achievement to identify needs for improvement and factors associated with achievement

22. Background Data Gathering Framework Background data are classified into five areas following the Craig-Heneveld Framework: • The Students • Supporting Inputs from Outside the School • Teaching-Learning Process • Enabling Conditions • School Climate

23. The Students Data is gathered from students about: • Socio-economic background (e.g. family size, father’s/mother’s education and occupation) • Factors in the home environment related to achievement (e.g. help in homework, space for reading, need to work outside) • Attitudes towards education

24. Example of Student Background Question: Do you work outside the home to earn money? Conclusion: about 23% of Grade 4 students reported having to work outside the home. Their achievement is significantly lower than others’.

25. Supporting Inputs from Outside Background Information is gathered from different sources about: • Parent and community support (e.g. frequent school-parent meetings, financial and other support, parental involvement in homework) • Effective support from the education system (e.g. supervision, availability of teaching materials and supplementary materials) • Adequate material support (e.g. facilities, furniture in school, school budget for supplies)

26. Use of Blackboard and Achievement • Teachers’ use of black board significantly increased students’ achievement in all the four subjects • Ninety-seven percent teachers reported they daily use black boards

27. Multi-grade Teaching and Achievement • Lower frequency of Multi-grade Teaching is linked with increased performance for both students and teachers • Twenty- one percent teachers reported they have to teach more than on classes in one period

28. Rewards and Punishment in School • Student who Never got punished by their teachers performed significantly better Fifty-eight percent students reported they never got punished. • Only .03 % students reported frequent punishment • Rewarding students’ performance increased achievement all the subjects

29. PTA/ SMC in School: Head Teachers’ Report • Around 80% head teachers reported presence of PTA/ SMCs’ in School • Presence, need, funding, or role of PTA/SMC did not increase students performance in any of the NEAS tested school • Only 39% head teachers reported getting funds from Govt. • Ninety-seven percent head teachers reported need for SMCs in Schools

30. Strengthening Links between NEASand the Educational Planning Process • NEAS can Support Educational Planning by: • providing data about the actual conditions in schools; and • indicating the possible impact of policy actions on student achievement. • Educational Planners can make NEAS more Effective by: • proposing topics to be added to background data gathering; and • suggesting relationships to be explored.

31. The Presentation Summary • NEAS has assessed the achievement of Grade 4 & 8 students in four subjects: Urdu, Mathematics, Science and Social studies. • The assessment reveals large differences in average performance among provinces and between students from differing backgrounds. • These differences in average performance can be traced back to differences in student and family characteristics, school processes and inputs.

32. Institutionalization of NEAS and Future Vision of National Assessment in Pakistan • Institutionalization of NEAS: Government of Pakistan declared NEAS as a permanent institution of the Ministry of Education, from July 2007 • Future Vision of NEAS: Recommendations by National NEAS Select Committee: • NEAS future planning till 2020 comprises National Assessment at Grade 4, 5, 8, 10 and 12 • Private schools are included in NEAS sample on Pilot basis in 2008 National Assessment • Listening and speaking skills of student assessment would be part of NEAS future activities • Pakistan can explore the possibility to participate in the next TIMSS in 2012

33. Thank You…