1 / 36

Measuring of student subject competencies by SAM: regional experience

Measuring of student subject competencies by SAM: regional experience. Elena Kardanova National Research University Higher School of Economics. Outline of presentation. SAM based model for assessment of student subject competencies

eshana
Download Presentation

Measuring of student subject competencies by SAM: regional experience

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring of student subject competencies by SAM: regional experience Elena Kardanova National Research University Higher School of Economics

  2. Outline of presentation • SAM based model for assessment of student subject competencies • Regional diagnostic study: sampling and procedures of test administration • SAM regional norms and presentation of results • Interpretation and uses of SAM information: primary analysis of factors that influence educational results in primary school • SAM uses for improving teaching and learning: relation between teacher’s pedagogical approaches and educational results in primary school • SAM experience in other countries

  3. SAM based model for assessment of student subject competencies

  4. Approaches to results interpretation • Norm-Referenced • The result of individual student is interpreted depending on the achievement of the whole population • Each student gets test score • Norms are set

  5. Approaches to results interpretation • Criterion-referenced • The gradual option of achievement scale is developed. It’s based on students integrated test scores and benchmarks that divide all participants of testing into groups that relevant to different proficiency levels • Each participant is assigned a proficiensy level

  6. Students estimation • Rasch model is used as a test model • Test scores are reported on a 1000-point scale with a mean at about 500 and standard deviation of 50 • Test scores of all participants are on the same metric scale regardless of the time of test administration and specific set of test items completed

  7. Estimation of examinees Mathematical competence scale Proficiency level 3 We expect student A to successfully complete at least 50% of level 3 items Student A Items of the 3rd level 570 Proficiencylevel 2 We expect student B to successfully complete at least 50% of level 2 items Student B Items of the 2nd level 500 Proficiency level 1 We expect student C to successfully complete at least 50% of level 1 items Student C Items of the 1st level 430 Below level 1 We expect student D to be unable to successfully complete even 50% of level 1 items Student D

  8. Interpretation of benchmarks Benchmarks: 570 (border btw. 2 and3 proficiency levels) 500 (border btw. 1 and 2proficiency levels) 430 (border btw. 0 and 1proficiency levels)

  9. Regional diagnostic study: sampling and procedures of test administration Velikiy Novgorod and its area

  10. Regional diagnostic study May2012 Sample size: 4406 students of 4-th grade (the region’swhole population of fourth grade students ) No selection at the school or classroom level

  11. Description of research sampling 47% boys, 53% girls 72% urban, 28% rural

  12. Regional diagnostic study: procedure • Paper&pencil form • Administration to the whole class by the teacher • Two 45-minute testing sessions with a 15-minute break • Whole region in 1 week

  13. SAM regional norms and presentation of results

  14. SAM results • Integrated test score( relation of result to the metric scale) • Proficiency level (relation of result to grade scale) • 3D profile (relation btw. results of 3 subtests)

  15. Normative-referenced interpretation: Statistical norms (Mathematics) Average group normsMean Standard deviation 517 34 Socio-cultural norms Mean 561 Average individual norms Mean Standard deviation 52249 Percentile individual norms10th percentile 90th percentile 459 581

  16. Math profile for a sample of students

  17. Distribution of test participants on proficiency levels (Mathematics)

  18. Distribution of students of different schools of the region at proficiency levels (mathematics) • Schools put in order by increasing of the mean test score • For every school the nean test score is indicated in brackets.

  19. Distribution of students of different classes within the same school by achievement levels (mathematics)

  20. Interpretation and uses of SAMinformation: primary analysis of factors that influence educational results in primary school

  21. Key questions • What is the efficiency of different educational programs? • How different factors influence on students learning? • What characteristics of learning environment influence on educational quality in primary school? • How teachers and school work can be improved?

  22. Educational environment and its characteristics that can be examined

  23. Information sources for analysis of factors Contest questionnaries(for teachers, for administrative personnel) • Set of contest characteristics can vary depending on regional research tasks • Focus at characteristics of school environment that can be corrected to improve the quality of education Technical information The detailed information about school learning and teaching features can be collected purposefully (e.g. educational programs).

  24. Relation btw. SAM math results and type of educational institution • There are 15% of student study in gymnasium. • Differences btw. schools are statistically significant: gymnasiums get better tests results. • Number of children at 2nd level is the same. Difference is btw. children at 1st and 3rd levels.

  25. Relation btw SAM math results and type of settlement Percentage of children at 3rd level is higher in the city and decreases in towns and villages. Its vice versa for percentage of children at 1st level – it’s higher in villages.

  26. Relation btw SAM math results and type of settlement Most of students are on the 2nd proficincy level In Velikiy Novgorod the results are slightly better – bigger percentage of children is at 2nd and 3rd levels.

  27. Relation btw SAM math results and size of the class • We can single out 2 types of classes – big and small • Small classes are those that have less than 11 students, big classes have 11 and more students (maximum number of students in one class is 33) • All together we analyzed 76 small and 152 big classes

  28. Relation btw SAM math results and size of the class Results of children in small and big classes are not statisticallydifferent.

  29. SAM uses for improving teaching and learning: relation between teacher’s pedagogical approaches and educational results in primary school

  30. Pedagogical approaches • Currently it is widely assumed that teachers’ beliefs about the nature of teaching and learning include both “direct transmission beliefs about learning and instruction” or, so called, “traditional beliefs” and “constructivist beliefs about learning and instruction” (OECD, 2009). • 2 educational approaches: traditional and constructivist • The traditional approach implies that teacher communicates knowledge in a clear and structured way, explains correct solutions, gives learners clear and resolvable problems and ensures peace and concentration in the classroom • The constructivist approach implies that students are active participants in acquisition of knowledge, students’ own inquiry is stressed developing problem solutions

  31. Teachers survey • Special teachers questionnaire • 228 teachers in total • 56 teachers work in Velikiy Novgorod, and 172 in Novgorod region • 17 teachers work in gymnasium and 211 in comprehensive school • 186 teachers graduated from university and 42 graduated from college • Work experience varies from 2 to 48 years. Average is 25 years

  32. Teachers’ general pedagogical approach Traditional Correlation -,204** is significant at the 0.05 (2-tailed) Constructivist

  33. Clusterization of classes

  34. Pedagogical approaches • Constructive approach positively relates to results of learning in math and Russian language: the higher level of constructivism of a teacher the higher test scores students have. • Constructive approach positively relates to the number of students in class at 3rd level and negatively with number of students at 1st level and below 1st level. • Traditional approach doesn’t have significant relation with learning results – neither with test scores or distribution of children at levels.

  35. SAM experience in other countries • Kazakhstan • Kyrgyzstan • Tajikistan

  36. Thank you for your attention Elena Kardanova ekardanova@hse.ru Center for monitoring and quality of education Institute of education Higher School of Economics http://ioe.hse.ru/monitoring/

More Related