Understanding the work of the HKEAA: Quality Assurance

1. Understanding the work of the HKEAA: Quality Assurance C.S. Tong Secretary General 7 June 2013

2. About HKEAA • Established in 1977 (independent statutory body) • 400 staff in 7 office locations • 8,000 exam personnel (seasonal) Mission: To provide valid, reliable and equitable examinations and assessment services in a professional, innovative, efficient and effective manner.

3. Quality Assurance

4. How are examination papers set? • Curriculum and Assessment (C&A) Guides • Subject Committees to oversee the continual development of each subject • Moderation committees to set papers with rigorous security and quality control mechanism • Moderation committee members are appointed on an annual basis subject to various security requirements

5. HKEAA Subject Managers Secondary School Teachers Subject Expert/ University Academics Who are Moderation Committee members? Open nomination for membership from schools as well as from the relevant Subject Committee (current S5 or S6 teachers not eligible)

6. How are scripts handled? • All scripts are scanned and then archived for safe-keeping: over 10,000,000 pages scanned for Onscreen Marking 6

7. How are Written Papers marked? Marking scheme Appointment of Markers Sample scripts selection Allocation of scripts to markers Markers’ meeting Standardisation meeting Marking Checkmarking Statistical analysis of marks

8. How are Written Papers marked? OSM facilitates real time, continuous monitoring of the quality of the marking: • Marking by questions to enhance consistency in applying the marking criteria • Double marking for open-ended questions • Use of control scripts to validate marking consistency • Scripts are selected for checkmarking by experienced Chief/Assistant Examiners 8

9. About 1,400 workstations Where are the Assessment Centres? Sheung Shui:185 Sha Tin Wai:92 Tai Hing:100 San Po Kong:73 Tsuen Wan:163 Tseung Kwan O:21 Lai King:376 Fortress Hill:382

10. Facilities at the Assessment Centres Workstations Control Room Lounge/Pantry Conference Room Reception Area 10

11. Markers’ meeting 1st checkmarking Training scriptsQualifying scripts Reference scripts Normal scriptsControl scripts satisfactory marking 2nd checkmarking satisfactory marking Complete OSM Procedures

12. What are the advantages of OSM? Security Data availability Efficiency Accuracy Marking quality

13. Markers’ meeting to clarify and finalize marking scheme Provision of Reference Scripts to illustrate the marking requirements and standard Training Scripts & Qualifying Scripts to ensure quality of marking before markers start marking ‘real’ scripts How does the Authority monitor Quality of Marking?

14. Monitoring Quality of Marking Selected scripts (including control scripts etc) have been discussed by a team including the Chief Examiner and awarded standardized scores Control scripts are randomly distributed to markers to ensure consistency of marking standards

15. Monitoring Quality of Marking Marker’s score can deviate from the standardized score to within a specified tolerance When the deviation exceeds the tolerance level, the marker will be contacted for clarification of marking scheme/ further training and qualifying/ or stopped from further marking and problematic scripts can be voided and redistributed to other markers for marking as necessary

16. Illustration Marker’s score agrees with Standardized Score Marker’s Score Standardized Score Tolerance level

17. Illustration Marker’s score agrees with Standardized Score Marker’s Score Marker’s score much lower than standardized score Marker’s score much higher than standardized score Standardized Score Tolerance level

18. Illustration Marker’s score agrees with Standardized Score Marker’s Score Marker’s score much lower than standardized score Marker’s score much higher than standardized score Standardized Score Tolerance level

19. Monitoring Quality of Marking Chief / Assistant Examiners will check each marker’s marking statistics (including marking speed, mark distribution, score deviation from control scripts etc) scrutinize markers’ marking records and select scripts for checkmarking contact markers and further clarify the marking scheme if necessary 19

20. Why Double Marking? • To address the issue of inter-rater variability for open ended questions, double marking may be adopted • Feasibility study carried out to assess the cost benefit between such factors as • Marking reliability • Markers’ standardization • Increased marking time • Marking cost

21. Big discrepancy between the two marks The discrepancy among marks is still out of range Marker 1 Marker 2 Marker 3 Discrepancy within range Discrepancy within range Candidate’s mark Candidate’s mark Candidate’s mark Double Marking: how? Chief Examiner or Assistant Examiner The Candidate’s final mark will be the sum of the two highest closest marks

22. How to ensure overall Quality? • Post examination review • External vetting of examination papers • Post examination review • Internal audit & risk monitoring • Examination Paper Quality Audit (EPQA) • Internal post-mortem meetings • Setting ofexamination papers • Curriculum and Assessment Guide & Assessment Framework • Examination paper development mechanism • Confidentiality and security measures Continuous Improvement HKDSE Recognition Processing & reporting of results • Conduct of examinations • Examination regulations • Public Examinations Information Centre (PEIC) • Public Examinations Communication and Support System (PECSS) • Oral Recording System (ORS) • On-site checking & audits • Examination logistics & security measures • SBA • Marking process • Mechanisms for handling irregularities, complaints and queries • Standards setting and maintenance • Grading procedures • Quality check of examination data • Feedback to schools Compliance & Quality of Delivery Safeguarding of Standards

23. Vetting of Examination papers • Different subjects are selected annually for vetting and peer review • Vetting reports provide useful feedback for subject managers and Subject Committees • Reports and responses from the vetting exercise as well as reports on examination irregularities are submitted to the Public Examinations Board for discussion

24. Discrepancy Checking Scripts are sampled for cross checking at various stages of marking and grading as part of our quality assurance framework: • Randomly selected candidates and all their scripts extracted for manual checking • Scripts of candidates with large discrepancies in the levels obtained in different subjects • Scripts of candidateswith large discrepancies in component levels (eg over 1000 scripts from Chinese Language checked in 2012) 24

25. Discrepancy Checking Performance at the school level are cross-checked across different core subjects: eg LS vs Chin Outlier schools to be manually checked 25

26. Discrepancy Checking Math vs English 26

27. Assessment Quality-Assurance Platform • Based on Rasch Model • Item-level and paper-level analysis to ensure quality: • Item difficulties • Discrimination power • Item reliability • Paper reliability (internal consistency)

28. AQP評核素質保證平台 • Item Level

29. AQP評核素質保證平台 • Paper Level

30. Supporting Quality Assessment at Schools

31. User-friendly version of AQP is available for schools to support Assessment for Learning

32. AQP評核素質保證平台 • Student-Level

34. Quality Assessment Management

35. International Recognition

36. International Recognition Over 150 overseas institutions responded to our survey proforma 70 institutions from the Mainland accept students from HK on the basis of HKDSE 149 institutions from Taiwan accept students from HK on the basis of HKDSE 36

37. International Recognition Comparison 145 37

38. 香 港 中 學 文 憑 考 試 一紙文憑 世界通行

39. Thank You! Delivering HKDSE with Pride 香港中學文憑 眾志昂首同行