Portfolio Committee on Basic Education Report on the Quality Assurance of the NSC

1. 18 February 2014 Dr Mafu S Rakometsi Portfolio Committee on Basic EducationReport on the Quality Assurance of the NSC

2. WHAT CAN WE LEARN FROM THE NSC RESULTS? Introduction to context, principles, approaches and processes Dr Mafu Rakometsi - CEO of Umalusi

3. Regulatory Framework • Quality Assurance of Assessment • NQF Act Section 27 (h) • The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.

4. Section17 of the GENFETQA Act • (5) The Council must, with the concurrence of the Director-General and • after consultation with the relevant assessment body or education • institution, approve the publication of the results of learners if the • Council is satisfied that the assessment body or education institution • has— • (i) conducted the assessment free from any irregularity that may • jeopardise the integrity of the assessment or its outcomes; • (ii) complied with the requirements prescribed by the Council for • conducting assessments; • (iii) applied the standards prescribed by the Council which a learner is • required to comply with in order to obtain a certificate; and • (iv) complied with every other condition determined by the Council.

5. Framework for QA of Learner Achievement • Based on established and existing practices in assessment for certification • Prescribed components of external assessment (examinations) and Site-based/ internal / continuous assessment • Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.

6. Framework for Quality Assurance of Assessment • Evaluation and /or accreditation of assessment bodies • Periodic inspection of assessment systems • Ongoing monitoring of assessment systems • Quality assurance of external examinations through: • Moderation of examination question papers • Monitoring and moderation of SBA • Monitoring the conduct of examinations • Moderation of marking • Standardization of assessment outcomes

7. Approval for the release of Results • Approval is based on the following requirements: • The examinations are conducted compliant to the applicable policies regulating the conduct and administration of the examinations • At the time of approval, there is no serious irregularity which could undermine the credibility of the examinations.

8. Quality Assurance of the DBE 2013 National Senior Certificate Examination Emmanuel Sibanda: Acting Sen. Manager : Quality Assurance of Assessments

9. PURPOSE: • To ensure that the question papers are of the required standard (standard captured in the NCS and SAG’s) • To ensure that the question papers are relatively: - fair - reliable - representative of an adequate sample of the curriculum - representative of relevant conceptual domains - representative of relevant levels of cognitive challenge Moderation of question papers

10. Moderation of the question papers Approach: • Question papers set by panel of examiners – DBE • Internally moderated by DBE • Externally moderated by Umalusi • Subsequent moderations and approval

11. Moderation of the question papers Criteria: • Technical criteria • Internal moderation • Content coverage • Text selection, types and quality of questions • Predictability • Cognitive skills • Marking memorandum/guidelines • Language and bias

12. Moderation of the question papers Findings: Areas of Good Practice • Percentage of question papers and memoranda approved after first and second moderation ( Nov 2013- 70% ; Mar 14 – 78%) • Simultaneous moderation of final and supplementary question papers

13. Moderation of the question papers Findings: Areas of Concern • Adherence to timeframes and impact on quality of setting and moderation • Question papers requiring more than four moderations 2 papers for November 2013 (Isizulu HP P1, Isixhosa HL P1) and 2 for March 2014 (Business Studies, Isizulu HL P1)

14. Definition: • Internal assessment refers to any assessment conducted by the provider , the outcome of which count towards the achievement of the qualifications • Umalusi appoints panels of moderators / subject specialist to carry out this mandate Moderation of internal assessment

15. Purpose of Umalusi’s verification: • To verify the rigour and appropriateness of the DBE moderation process – linked to DBE plans • Ascertain the degree to which assessment bodies/provinces are attempting to ensure standardisation across • Ascertain the standard and quality of the tasks • Determine the extent and quality of internal moderation and feedback. • Determine the reliability and validity of the assessment outcomes Moderation of internal assessment (cont.)

16. Moderation of Internal Assessment • Approach 1 (June/July 2013)- verifying the DBE SBA • moderation

17. Moderation of Internal Assessment • Approach 2 (June/July 2013)- Umalusi independent • moderation (own sample)

18. Moderation of Internal Assessment • Areas of good practice: • DBE conducted a very rigorous moderation and provided useful verbal feedback to PED at the end of each moderation session • Certain pockets of improvement wrt SBA implementation although far and wide • General adherence to policy in terms of number of tasks done, and presentation of learner evidence/portfolios.

19. Moderation of Internal Assessment • Areas of concern: • While internal moderation is being done in most schools, much of the focus is on compliance (monitoring) and not on the quality of assessment (actual moderation) • Teachers are still unable to develop tasks pitched at appropriate cognitive levels: focus is more on lower cognitive level • Assessment of practical investigations, research projects, assignments and simulations still a major problem • The use and development of rubrics is problematic: descriptors are unrealisable and vague

20. Monitoring of Examinations • “State of readiness” • Conduct of examinations • Marking

21. Monitoring of Examinations • “State of readiness” • Findings: • All provinces have working examination systems in place • Of concern: • In certain PDE’s closer monitoring of printing of question papers needs attention • Inefficient or lack of proper coordination with districts wrt exam related processes

22. Monitoring of the writing phase. Findings: • Generally examinations conducted in line with policy • Isolated instances of non-compliance (suitability of venue, identification of learners)

23. Monitoring of the marking phase

24. Monitoring of marking phase: • Findings: • Marking centres were generally well organised and suitable for the task • Inadequate and inexperienced security in some provinces • Absence of communication facilities reported at one centre in one province

25. PURPOSE: • Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices • Umalusi engages with the following during the moderation of marking • Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres • Moderation of marking (centralised and on-site) Verification of marking

26. Marking verification • Memo discussion meetings: • Areas of good practice: • The memo discussions for the approval of final memoranda went relatively well in 2013. Provision of an extra day of training in marking was very welcome

27. Memo discussion meetings (cont) • Areas of concern: • Not all representatives did pre-marking. Provincial representatives often experienced problems accessing a sample of scripts to pre-mark before coming for memo discussion meetings • The time between the examination dates and the memo discussions was generally far too short to allow pre-marking to take place. This was reported in several subjects, and seriously compromised the validity of the process, as meaningful discussion and consistency depends on the pre-marking of scripts

28. Memo discussion meetings (cont) • Areas of concern: • Some provinces sent only one representative or none at all to the memo discussions. • The fact that the memo discussions for African languages, HL, SAL and FAL took place in the same time slot caused problems because there was only one internal and external moderator for these levels in some languages.

29. Centralised & on-site marking verification • Areas of good practice: • Many external moderators expressed the opinion that the accuracy of marking had improved slightly. • External moderators were unanimous in their appreciation of the positive impact that the thorough training at the memo discussion meetings had had on the quality of marking.

30. Centralised & on-site marking verification (cont) • Areas of concern: • Markers still experience problems with regard to interpreting answers to open-ended and higher order questions • Use of rubrics continues to be a serious concern in particular the inappropriateness of the rubrics used for P3 of HL and FAL: the descriptors did facilitate good marking. This needs revisiting by the DBE as it has a bearing on the marks allocated • There are still markers marking literature questions, who do not have a thorough knowledge of the stories/dramas/novels/poems they are marking

31. Monitoring of the writing phase. Areas of good practice: The DBE exam was generally administered in line with policy. No major concerns were reported. One can see growth in the administration and conduct of the exams in PDEs.

32. Standardisation and verification of resulting • Provision of GENFETQA – Council may adjust raw marks • International practice – large scale assessment systems • Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance • Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

33. Statistical moderation Scope of standardisation 2013: • 59 standardised • Raw marks accepted: 38 subjects • Moderated Upward : 5 subjects • Moderated Downward : 16 subjects

34. Verification of the Resulting Process

35. Examination Irregularities • The majority of irregularities were of a technical nature and these were reported to Umalusi according to the established channels. • Some irregularities were as a result of registration-related problems, e.g. candidates nor appearing on mark sheets, some registered for incorrect subjects. • Umalusi represented on NEIC

36. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS DBE NSC 2013

47. Quality Assurance of the IEB 2013 National Senior Certificate Examination Emmanuel Sibanda – Sen. Manager : Quality Assurance of Assessments

48. Moderation of the question papers Areas of Good Practice: • Percentage of question papers and memoranda approved after first and second moderation ( Nov 2013- 95% Mar 2014 – 100%) • Examining panels must be acknowledged for their favourable response to the External Moderator’s requests for changes and recommendations made

49. Moderation of Internal Assessment • Term 4 moderation- focus on learner evidence and • teacher files

50. Moderation of Internal Assessment • Findings: Areas of Good Practice • IEB has very good systems and processes in place for SBA implementation and support of their educators. • This is borne out by the generally good quality of tasks set by the IEB educators. • Findings: Areas of Concern • The main challenge is how rubrics are developed: there is some element of vagueness and subjectivity of criteria – this leads to differences in interpretation by educators, resulting in differences in allocation of marks. • Moderation at school level can also be improved.