Tuesday, 3 March 2015 Dr Mafu S Rakometsi

1. Tuesday, 3 March 2015 Dr Mafu S Rakometsi Portfolio Committee on Basic EducationReport on the Quality Assurance of the National Senior Certificate (NSC)

2. WHAT CAN WE LEARN FROM THE NSC RESULTS? Introduction to the context, principles, approaches and processes Dr Mafu S Rakometsi, Chief Executive Officer of Umalusi

3. Regulatory Framework • Quality Assurance of Assessment • NQF Act Section 27 (h) • The Quality Council (QC) must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.

4. Section17 of the GENFETQA Act • (5) The Council must, with the concurrence of the Director-General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has — • (i) conducted the assessment free from any irregularity that may • jeopardise the integrity of the assessment or its outcomes; • (ii) complied with the requirements prescribed by the Council for • conducting assessments; • (iii) applied the standards prescribed by the Council which a learner is required to comply with in order to obtain a certificate; and • (iv) complied with every other condition determined by the Council.

5. Framework for Quality Assurance of Learner Achievement • Based on established and existing practices in assessment for certification • Prescribed components of external assessment (examinations) and Site-based/ internal / continuous assessment • Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies

6. Framework for Quality Assurance of Assessment • Evaluation and /or accreditation of assessment bodies; • Periodic inspection of assessment systems; • Ongoing monitoring of assessment systems; and • Quality assurance of external examinations through: • Moderation of examination question papers • Monitoring and moderation of SBA • Monitoring the conduct of examinations • Moderation of marking • Standardization of assessment outcomes

7. Approval for the release of Results • Approval is based on the following requirements: • the examinations are conducted compliant to the applicable policies regulating the conduct and administration of the examinations; and • at the time of approval, there is no serious irregularity which could undermine the credibility of the examinations.

8. Quality Assurance of the DBE 2014 National Senior Certificate Examination Ms Faith Ramotlhale Senior Manager: Quality Assurance of Assessment

9. Moderation of question papers PURPOSE: • To ensure that the question papers are of the required standard - (CAPS and SAG’s); • To ensure that the question papers are relatively: - fair - reliable - representative of an adequate sample of the curriculum - representative of relevant conceptual domains - representative of relevant levels of cognitive challenge

10. Moderation of the question papers Approach: • Question papers set by panel of examiners – DBE • Internally moderated by DBE • Externally moderated by Umalusi • Subsequent moderations and approval

11. Moderation of the question papers Criteria: • Technical criteria • Internal moderation • Content coverage • Text selection, types and quality of questions • Cognitive demands • Marking memorandum/guidelines • Language and bias • Predictability

12. Moderation of the question papers Areas of Good Practice: • Percentage of question papers and memoranda approved after first and second moderation (Nov’ 2014 - 79% ; Mar’ 15 - 82%) • To ensure equivalence of standards, the final and supplementary examination question papers are moderated simultaneously. Areas of Concerns • 6 question papers were approved in August 2014 after the fifth moderation; • The use of analysis grids for content and cognitive levels; • (IsiXhosa FAL P1 &2; IsiXhosa HL P2; Nov’14 and March’15).

13. Moderation of the question papers Definition and Purpose: • Internal assessment refers to any assessment conducted by the provider , the outcome of which count towards the achievement of the qualifications • Ascertain the degree to which assessment bodies/DBE is attempting to ensure standardisationacross • Ascertain the standard and quality of the tasks • Establish the extent and quality of internal moderation and feedback • Verify the reliability and validity of the assessment outcomes

14. Moderation of Internal Assessment • Approach:

15. Moderation of Internal Assessment • Areas of good practice: • Internal moderation was conducted at three levels in some subjects, and well structured moderation process was in place • DBE moderation was found to be rigorous, brief, relevant and feedback was provided to teachers and learners • Generally, the SBA programme was managed appropriately, and the work schedule was adherence to • In some provinces marking was generally fair, marks were transferred accurately and marks were captured and calculated electronically

16. Moderation of Internal Assessment • Areas of concern: • Shadow marking, and incorrect allocation of marks have a negative effect on the validity, reliability and fairness of the SBA – at school and district level • Marking was too lenient , wide gap between marks scored in tests, examinations and practical tasks, and essay matrix levels were applied inaccurately • The use of rubrics still presents a major problem for teachers, and the marking rubrics for assignments, research projects, and practical investigations were poorly designed • Inability of learners to answer questions of a higher cognitive order, and poor research skills were noted in many subjects

17. Monitoring of Examinations • State of readiness • Umalusi shadowed the DBE in all provinces, and conducted independent verification of 82 district offices and a few schools • Findings: • Areas of Concerns: • Staff capacity is inadequate with a high number of vacant posts in PDE’s and districts • Inadequate resources at PDE’s and District offices • Co-ordination between PDE head office and District needs improvement

18. Monitoring of Examinations • State of readiness • High number of independent centres, lack of capacity to audit them may affect the credibility of examinations • Areas of Good Practice: • DBE standardized the turnaround times nationally for distribution of question papers and return of answer scripts • Most PED have the necessary security arrangements in place • Piloting of electronic locking system in one province • Most PDE’s have adequate to excellent printing, storage and distribution facilities

19. Monitoring of the writing phase Procedure: • Umalusi reviewed the sampling procedure • Centres offering SC and NSC had records of irregularities, high enrolments, record of performance, whistle blowers Areas of Good Practice: • In most provinces appropriate practices were followed and examination regulations were adhered to

20. Monitoring of the writing phase • Some examination centres had resident monitors assigned for the full duration of the examination. Areas of Concerns: • Delivery of examination material to rural nodal points was not always on time, and poor record keeping of examination material collected • Inadequate monitoring by assessment bodies in some provinces • Training of chief invigilators and invigilators should be intensified in certain provinces so that compliance to examination regulations is adhered to • Training of candidates on examination rules should be enhanced

21. Monitoring of marking phase • Areas of Good Practice • Marking centres were generally well organised and suitable for the task • Adequate and experienced security personnel at the gates and entrances to the venues in most provinces; and • Training of marking personnel was generally thorough and effective • Areas of Concerns: • Poor marking facilities, and shortage of ablution facilities • Late arrival of markers and load shedding hampered progress

22. Monitoring of Marker Selection PURPOSE: • To monitor the marker selection process with the view to establishing a benchmark for the practices employed in the selection and training of markers: Umalusi monitored the following processes: • Process of marker selection • Audit of marker selection • Training of various levels of markers • Criteria used for marker selection: Resolution 6 of 1997 as amended in 2001

23. Monitoring Marker Selection • Gauteng: (Performance of learners at 60%) • Shortage of applications for English HL – decrease number of Senior markers and extension of the marking period • Limpopo: • No documentation/ evidence to verify the list of applicants • Marker experience was captured incorrectly • List of markers declared incompetent during previous year was not available • Discrepancies in the appointment of marking personnel: Unqualified Senior Markers supervising highly qualified and experienced teachers • Unqualified markers with no subject specialization • Grade 10 teachers, principals, and teachers who were not teaching literature

24. Monitoring of Marker Selection • Mpumalanga • Each panel had the pass rates of schools for the previous year and verified against the application form • There were anomalies in the appointment of Physical Science P2 markers: teacher with PTD and ACE, incomplete forms accepted, Grade 10 teachers, incompetent teachers; inexperienced Senior markers • Placement battery Test administered • North West • The process was thorough, corroboration of application forms, school performance, and 2013 performance evaluation • The marker selection and appointment was not compromised, and training of markers was conducted satisfactory

25. Monitoring of Marker Selection • Kwa-Zulu Natal • Though pass percentage was captured it was not considered as a selection criteria , markers with 29% of learner performance were selected to mark Physical Science • Pass percentage was considered for Chief Marker positions • The monitoring of the process by Umalusi was not well received • Free State • The criteria were used without any enhancement, no deviation from PAM.

26. Monitoring of Marker Selection • Northern Cape • The markers were ranked according to ranking criteria designed by the PDE:- classroom performance, quality of marking in the previous marking sessions • Discrepancies in the appointment process were observed: • Inexperienced and unqualified chief markers and internal moderators were appointed in certain papers; • 2 markers without credentials were appointed in Life Sciences P1.

27. Monitoring of Marker Selection • Western Cape • Competency test administered in 11 gate way subjects • Markers were expected to obtain 60% • Two years teaching experience of Grade 12 between 2012-2014 • Consistency in the application of the selection criteria • Training of markers covered all aspects of marking • Eastern Cape: • Selection criteria -school performance resulted in the shortage of markers, thus more than 10% of novice markers were appointed

28. PURPOSE: • Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices • Umalusi engages with the following processes during the moderation of marking: • Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across provinces and marking centres • Moderation of marking (centralised and on-site) Verification of marking

29. Marking verification • Memo discussion meetings: • Areas of good practice: • The memo discussions for the approval of final memoranda went relatively well in 2014 • Improved training in most subjects was observed and the introduction of the tolerance range is highly commendable • Few changes were made to marking guidelines for enhancement of marking

30. Memo discussion meetings (cont) • Areas of concern: • Provincial representatives often experienced problems accessing a sample of scripts to pre-mark before coming for memo discussion meetings • The time between the marking dates and the memo discussions was too tight to allow pre-marking to take place • This was reported in several subjects, and seriously compromised the validity of the process, as meaningful discussion and consistency depends on the pre-marking of scripts

31. Memo discussion meetings (cont) • Areas of concern: • Some provinces sent only one representative or none at all to the memo discussions • The fact that the memo discussions for African languages, HL, SAL and FAL took place in the same time slot caused problems because these subjects share national examiners, internal moderators and external moderators • Where training occurred in FAL and SAL papers, it was compromised as the time allocated was too short • SeSotho HL P2 up to 60% of the memorandum was changed with many alternatives added

32. Verification of Centralised & on-site marking • Areas of good practice: • Computer Application Technology (CAT) consultation model with national internal and external moderators is commendable for ensuring consistency of marking across the provinces • The intensive and extensive training of markers had a positive effect on the quality of marking

33. Verification of Centralised & on-site marking (cont) Areas of concerns: • Candidates had difficulty in the interpretation of specific verbs that denoted to cognitive level of questions • Appointment of incompetent personnel as chief markers • The official rubric for creative writing needs further refinement as it has advantaged candidates in the higher categories • Appointment of incompetent internal moderators- unauthorised additions to the marking guidelines

34. Verification of Centralised & on-site marking (cont) Areas of concerns: • Inadequate number of internal moderators compromises the rigorous moderation of scripts • Mistakes in the marking may be attributed to non compliance to the marking memorandum by markers

35. Examination Irregularities • The marking moderation process was intensified by deploying external moderators to the 9 provinces for the extended period and independent sampling of scripts • Irregularities that occurred in Mpumalanga, KZN and EC were identified during the verification of marking process at the various marking centre • Subjects that were implicated were mostly high enrolment subjects, namely Mathematics, Physical Science, Life Sciences, English FAL, Geography, History, and Business Studies and Accounting • DBE was informed to conduct an audit in the affected subjects and centres- Umalusi oversight role

36. Examination Irregularities • DBE conducted an audit in the two provinces and presented the outcome of the audits involving irregular patterns in the learners’ scripts; 39 centres were identified in KZN and 19 centres in the Eastern Cape • Umalusi’s external moderators and internal staff were deployed to the provinces to monitor and verify the outcome of the audits • DBE recommended the clearance of additional centres to EXCO, and after Umalusi verification additional centres were cleared in KZN and in the Eastern Cape; • DBE and EC Chief Director presented reports to EXCO from their investigation (candidates, invigilators and chief invigilators), - Umalusi oversight role

37. Examination Irregularities • Methodology differed- EC invigilators, Chief invigilator and candidates were interviewed by one team on the same date; objectivity and professionalism was observed • KZN- invigilators and chief invigilators were interviewed separately from candidates with the difference of a week in between by different panels; - subjectivity and lack of professionalism observed • The number of implicated centres stood at 20 in KZN and 14 in the Eastern Cape, the results of the implicated candidates were blocked pending the outcome of the hearings with candidates

38. Examination Irregularities • Meetings were held between Umalusi and DBE Senior Management; Umalusi Chairperson of Council and the Minister to map out the way forward; and conflicting views presented by media • DBE will provide Umalusi with the plans for the conduct of the formal hearings in the two provinces- scheduled in March/April 2015

39. Examination Irregularities

40. Standardisation and verification of resulting • Provision of GENFETQA – Council may adjust raw marks • International practice – large scale assessment systems • Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance • Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

41. Verification of the Resulting Process

42. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS DBE NSC 2014

43. Statistical moderation Scope of standardisation 2014:

49. Statistical moderation Areas for Concern • Some recommendations for adjustments were not in line with the principles of standardization • Essay questions that were left out of English FAL remains a concern • Late submission and inaccurate standardisation data delays the approval of results

50. Quality Assurance of the Independent Examinations Board (IEB) 2014 National Senior Certificate Examination Ms Faith Ramotlhale Senior Manager: Quality Assurance of Assessment