Portfolio Committee on Basic Education Report on the Quality Assurance of the NSC

1. 7 February 2012 Dr Mafu S Rakometsi Portfolio Committee on Basic EducationReport on the Quality Assurance of the NSC

2. WHAT CAN WE LEARN FROM THE NSC RESULTS? Introduction to context, principles, approaches and processes Dr Mafu Rakometsi - CEO of Umalusi

3. Regulatory Framework • Quality Assurance of Assessment • NQF Act Section 27 (h) • The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.

4. Section17 of the GENFETQA Act • (5) The Council must, with the concurrence of the Director-General and • after consultation with the relevant assessment body or education • institution, approve the publication of the results of learners if the • Council is satisfied that the assessment body or education institution • has— • (i) conducted the assessment free from any irregularity that may • jeopardise the integrity of the assessment or its outcomes; • (ii) complied with the requirements prescribed by the Council for • conducting assessments; • (iii) applied the standards prescribed by the Council which a learner is • required to comply with in order to obtain a certificate; and • (iv) complied with every other condition determined by the Council.

5. Framework for QA of Learner Achievement • Based on established and existing practices in assessment for certification • Prescribed components of External assessment (examinations) and Site based/ internal / continuous assessment • Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.

6. Framework for Quality Assurance of Assessment • Evaluation and /or accreditation of assessment bodies • Periodic inspection of assessment systems • Ongoing monitoring of assessment systems • Quality assurance of external examinations through: • Moderation of examination question papers • Monitoring and moderation of SBA • Monitoring the conduct of examinations • Moderation of marking • Standardization of assessment outcomes

7. Approval of Results • Compliance with minimum requirements • Shift from Moderation to Verification (Moderation for verification) • Emphasis on Assessment Body reports • The monitoring of the conduct of examinations and the moderation of marking as well as the monitoring of the implementation of internal assessment and the moderation of internal assessment is undertaken by Umalusi to verify the veracity of the assessment body reports in respect of these process. • Broadened definition of irregularities (including assessment system processes)

8. Approval for the release of Results • Approval is based on the following requirements: • The examinations are conducted compliant to the applicable policies regulating the conduct and administration of the examinations • At the time of approval, there is no serious irregularity which could undermine the credibility of the examinations.

9. Quality Assurance of the 2011 National Senior Certificate Examination Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments

10. Moderation of Question papers Number of NSC 2011 question papers moderate

11. PURPOSE • To ensure that the question papers are of the required standard (standard captured in the NCS and SAG’s) • To ensure that the question papers are relatively: - fair - reliable - representative of an adequate sample of the curriculum - representative of relevant conceptual domains - representative of relevant levels of cognitive challenge • External moderators Moderation of question papers

12. Moderation of the question papers Approach • Question papers set by panel of examiners – DBE • Internally moderated by DBE • Externally moderated by Umalusi • Subsequent moderations and approval.

13. Moderation of the question papers Criteria • Technical Criteria • Internal moderation – quality of internal moderator reports. • Content coverage • Cognitive Demand • Marking Guidelines • Language and Bias • Predictability

14. Moderation of the question papers Findings Areas of Good Practice • Additional back-up papers for CAT and IT • Simultaneous moderation of final and supplementary question papers. • Decrease in the number of question papers going beyond second moderation. • Standard of Internal moderation

15. Moderation of the question papers Findings Areas of Concern • Adherence to timeframes • Simultaneous moderation of final and supplementary question papers. • Predictability in Afrikaans SAL – prescribed short stories.

16. Internal assessment refers to any assessment conducted by the provider, the outcomes of which count towards the achievement of the qualification • panels of moderators / subject specialists • Moderation of internal /continuous assessment

17. Moderation of SBA • Previous model - moderation of learner and educator portfolios at the end of the year. • Umalusi implemented a new model in 2010 –focus on tasks rather than on portfolios • Ongoing – 3 moderations across 9 PDE’s • Feed back to educators • Improvement in SBA implementation • DBE implemented a similar model at a national level in 2011.

18. Purpose of Umalusi’s verification: • To verify the rigour and appropriateness of the DBE moderation process. • Ascertain the degree to which assessment bodies/provinces are attempting to ensure standardisation across • Ascertain the standard and quality of the tasks • Determine the extent and quality of internal moderation and feedback. • Determine the reliability and validity of the assessment outcomes Moderation of internal assessment (cont.)

19. Moderation of SBA - Scope

20. Moderation of SBA • Findings • Adherence to policy guidelines • Quality of Assessment tasks • Content Coverage • Cognitive demand and difficulty levels • Marking (tools, application and mark allocation) • Internal Moderation • Moderation instruments • Moderation at School level & district level • Feedback and Support

21. Moderation of SBA • Findings • Areas of Good Practice • DBE National Moderation • Provincially set preliminary examinations. • Increased use of common tasks

22. Moderation of SBA • Findings • Areas of Concern • Standard of tasks • Marking tools – use of rubrics • Life Orientation- Physical Education Tasks

23. Monitoring of Examinations • State of readiness • Conduct of examinations • Marking

24. Monitoring of Examinations • State of readiness • Findings • All provinces have examination systems in place • Vacant posts and use of contract staff is a major concern.

25. Monitoring of the writing phase. Findings • General Management of the examinations • Generally examinations conducted in line with policy • Isolated instances of non-compliance (suitability of venue, noise, security) • Shortage of question papers and answer books

26. Monitoring of the marking phase Findings • In general marking processes in line with policy • Concern with the appointment of markers • Concern with the quality and standard of marking • Large number of marking centres – security risk

27. Monitoring of the marking phase

28. PURPOSE • Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices • Umalusi engages the following during the moderation of marking 1. Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres 2. Off-site moderation of marking Verification of marking

29. Verification of marking • Memoranda discussion meetings • Approved and signed off finalized memoranda/ marking guidelines for all the NSC subjects • Concern with non attendance and pre- memo discussion preparations

30. Centralized verification of marking Findings • General improvement in standard of marking • Competency of markers to mark interpretation type questions. • Internal moderation • Unmarked questions • Unauthorised changes to marking guidelines.

31. Statistical moderation • Provision of GENFETQA – Council may adjust raw marks. • International practice – large scale assessment systems • Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance. • Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

32. Statistical moderation Standardisation • 56 subjects standardised • Raw marks accepted: 45 subjects • Moderated Upward : 3 subjects • Moderated Downward : 8 subjects

33. Verification of the Resulting Process

34. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS ERCO NSC 2011

40. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS IEB NSC 2011

50. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS DBE NSC 2011