Understanding the Work of the HKEAA: Mark Processing and Grading

1. Understanding the Work of the HKEAA: Mark Processing and Grading

2. Mark processing Some important steps after completion of marking: • Equating of marks of optional parts / papers • Moderation of SBA marks submitted by schools

3. Graded Approach for English Language Papers 1 & 3

4. Why do we need a graded approach? • Adopted in Paper 1 (Reading) and Paper 3 (Listening and Integrated Skills) • Compulsory Part A and either Part B1 (easier) or Part B2 (more difficult) • Cater to candidates with a wide range of ability

5. What are the advantages? • No labeling effect • No need to extend examination time or reduce coverage of the curriculum • Candidates need not indicate choice at registration so possible to catch up before the examination • Statistical equating to ensure that candidates receive comparable scores on the same scale whether they opt for the easier or more difficult part

6. Why is there a capping policy? • Candidates attempting Parts A and B2 will be able to attain the highest level of performance (Levels 5 and 5* / 5**) • Candidates who attempt Parts A and B1 will only be able to attain up to Level 4 in the relevant component • Reason for capping policy is that items in B1 do not assess skills covered by Level 5 descriptors

7. Why is equating necessary? • Easier to get high marks in Part B1 than in Part B2 because the items are easier • 1 mark in B1 is not equal to 1 mark in B2 • Need to convert the marks to the same scale • Use the marks of Part A as the mediator because all candidates do Part A

8. Example • 100 candidates got 36 marks in Part A • 50 of them did B1 and 50 did B2 • The 50 B1 candidates got 40 marks in B1 • The 50 B2 candidates got 32 marks in B2 • 40 marks in B1 = 32 marks in B2

9. What is equipercentile equating? • Large candidature for English Language (e.g. ~ 80,000 in 2013) • All candidates do Part A and can be rank ordered based on their marks in Part A • B1 marks are statistically adjusted for difficulty to the same level as B2 marks using the percentile ranks of candidates’ performance in Part A and Part B1/B2

10. Equipercentile method of equating B2 candidates’ ranking in Part B2 40th% B2 candidates’ ranking in Part A 40th% B1 candidates’ ranking in Part A 50th% B1 candidates’ ranking in Part B1 50th% 40 36 36 32

11. Equipercentile equating http://www.hkeaa.edu.hk/DocLibrary/Media/Leaflets/HKDSE_SRR_A4_Booklet_Jun2011.pdf

12. Is there a fixed conversion factor? • The examination papers for each year will be different from the previous year’s • Therefore no fixed conversion factor • A conversion table will be produced every year for that particular examination paper based on the performance of the candidates in Parts A, B1 and B2

13. How to choose between B1 and B2? Levels 4-5 Parts A & B2 (more difficult part) Levels 3-4 Parts A & B1 or Parts A and B2 Levels 1-3 Parts A and B1 (easier part)

14. How is the paper mark calculated? • For Part B1 candidates: Part A marks + Part B2 equivalent marks • For Part B2 candidates: Part A marks + Part B2 marks • Therefore, choosing B1 or B2 makes no difference for mid-range candidates

15. Moderation of SBA Marks

16. Some queries/concerns Why moderate schools’ SBA marks? Is it fair to moderate students’ SBA marks based on exam results?

17. Why moderation? Teachers know their students well and thus are best placed to judge their relative performance within a school However, they are not necessarily aware of the standards of performance across all schools Moderation ensures the consistency of assessment standards across schools in order to achieve across-school comparability

18. What are the principles of SBA moderation? Accept teacher judgments of relative student performance within a school (internal standardisation, student queries) Moderation is done on school basis (not on individual basis) to standardise across schools

19. Methods of SBA moderation Statistical The distribution of a school’s SBA marks for a subject is re-shaped taking into consideration school’s pattern of mark distribution for that subject’s examination papers Expert judgment A team of experienced, subject experts reviews samples of schools’ SBA work for a subject

20. Moderation methods

21. Features of statistical moderation Internal ranking would not be changed The statistical moderation method would consider the empirical correlation between examination results and SBA results Eliminate the effect of strictness and leniency

22. Features of statistical moderation Sample review to help identify outlier schools For outlier schools, adjustment will be carried out, if necessary, by considering a number of factors, such as moderated marks based on examination results and sample review findings so as to reflect the actual SBA performance

23. Expertjudgmentmoderation Review of samples of students’ work Determination ofschool performance level on SBA and individual students’ moderated scores Detection of cases with extreme adjustment and follow-up actions

24. Review of samples of students’ work Stratified random sampling within each school District Coordinators / External Assessors review samples with reference to the stipulated assessment criteria to determine the group performance level More samples from a school may be collected if necessary

25. Result of moderation in 2012 HKDSE 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 2009CE 10.0% 2010CE 2012DSE 0.0% much lower lower slightly lower slightly higher much higher higher as expected Mean SBA Adjustment

26. Result of moderation in 2012 HKDSE 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 2009CE 10.0% 2010CE 2012DSE 0.0% much lower lower slightly lower slightly higher much higher higher as expected Mean SBA Adjustment

27. Result of moderation in 2012 HKDSE In general, most teachers are aware of the standards and their SBA marks need very little adjustment Small deviation from the standard is expected due to normal variation of experience and subjective judgment: no systemic bias observed Moderation provides assurance of fairness to students, parents, and schools

28. Reference HKDSE Examination: Moderation of School-based Assessment Scores http://www.hkeaa.edu.hk/DocLibrary/SBA/HKDSE/HKDSE-SBA-ModerationBooklet_r.pdf

29. Standards-referenced Reporting

30. Some queries/concerns Why the grade awards are different across subjects? Is this unfair to my subject?

31. ..... ..... .....

32. Standards-referenced reporting system24 NSS Subjects • Descriptors and exemplars to indicate standards required for each level • Facilitate users, including tertiary institutions and employers, to set appropriate entrance/ job requirements 32

33. Standards-referenced reporting system 24 NSS Subjects • Adopted to report candidates' assessment results in all 24 HKDSE Category A subjects • SRR adopted for HKCE English Language and Chinese Language since 2007

34. Grading in the HKDSE • Cut scores and grade levels determined so that each level represents a fixed standard of achievement from year to year • Methodology combines psychometric measurement and expert judgment • Psychometric measurements: i) Monitoring tests for the core subjects ii) Group ability index (GAI) for elective subjects

35. Grading in the HKDSE • Expert judgments made independently by experienced examiners and experts based on the actual performance of candidates on the exams, using the level descriptors and library scripts for reference • Psychometric cut score recommendations are derived for the reference of the expert panel • No pre-set quota for each level /subject

36. Standards maintenance in 2013 Four core subjects (CEML) Additional reference made to: statistical data from the monitoring tests which were administered to a representative sample of 2012 and 2013 DSE school candidates

37. Using monitoring tests to maintain standards Statistically-generated cut scores for reference of Expert Panel

38. Group Ability Index (GAI) for elective subjects • Different Category A elective subjects may attract students of different abilities • Compute an indicator (index) to reflect overall performance in core subjects (ability) of all candidates taking the subject (group), taking into account how the performance in the elective correlates with each of the core subjects • Provided as supplementary statistics, in addition to the level descriptors, library scripts and live scripts, to the subject experts for their expert judgment. Final decision based on performance as reflected in the candidates’ scripts

39. Group Ability Index The GAI of Subject X for a certain level (e.g. Level 3) or above is defined as: b Coefficients obtained by regressing scores of Subject X on scores of CEML n/N Proportionof candidates in a group attained a particular level (e.g. Level 3) for a core subject

40. Grading Methodology and Procedures

41. Who are involved in grading Each subject sets up its expert panel which comprises: the chief examiner(s) of different papers, assistant examiners/markers, if necessary; SBA supervisors/chief project examiner, if applicable, and subject manager

42. Overview of grading procedures Familiarisation with standards Sample script selection Standardisation Post-marking exercise Expert Panel Meeting Internal Meeting Public Examinations Board Meeting

43. 4. Post-marking exercise Determine preliminary cut scores ranges for all levels for each exam paper (incl. SBA) with reference to: the difficulty of live question paper and requirements of marking scheme inter-paper correlations, mean and standard deviation of current year’s papers in comparison to previous years item statistics, if available impression of candidates’ performance when marking live scripts or reviewing students’ SBA work level descriptors and borderline performance samples in the previous exams

44. 5. Expert panel meeting Objective: Apply expert judgment to arrive at a set of recommended cut scores for different levels for the subject and its components, if applicable Make reference to marked live scripts falling within / around the recommended cut score ranges and exam statistics

45. Expert panel meeting (cont’d) Also refer to cut scores suggested by statistical means: for core subjects, the suggested cut scores are derived from the results of the monitoring tests (2013 data in comparison with the 2012 data) for elective subjects, the suggested cut scores are derived from GAI

46. ..... ..... .....

47. Reference Grading Procedures and Standards-reference Reporting in the HKDSE Examinationhttp://www.hkeaa.edu.hk/DocLibrary/Media/Leaflets/HKDSE_SRR_A4_Booklet_Jun2011.pdf

48. Reporting of Results

49. Reporting of HKDSE results • For Category A subjects, results reported mostly on subject basis • Component results reported for: • Chinese Language • English Language • Combined Science

50. Reasons for component reporting • Allow end users to understand the achievement of candidates in • different areas of the subject e.g. reading, writing, listening and speaking skills in language subjects • different half subjects in Combined Science • For selection for studies or employment