THE STATE OF READINESS national senior certificate (NSC) examination

1. THE STATE OF READINESS national senior certificate (NSC) examination and annual national assessment (ANA) Portfolio Committee on Basic Education Presented by: Nkosinathi Sishi, PhD Chief Director: Educational Measurement, Assessment and Public Examinations RSA 31 AUGUST 2010

2. PURPOSE OF THE SOR The state of readiness visit in 2010 focused on the following key areas: • The readiness of the PEDs to administer the 2010 NSC examination. • The status of the monitoring and moderation systems relating to School Based Assessment (SBA) at the Grade 10, 11 and 12 levels. • The State of Readiness of the PEDs to administer the ANA in Grade 3, 6 & 9. • Readiness of learners to take the assessment

3. International comparability of QUALITY, STANDARDS, Systems, and practices Strategic goal: Credible educational measurement, assessment run with integrity Quality practices Strategic objectives National assessment and measurement of high international esteem Valid and fair educational judgments National historical learner records Integrity of Site-based assessment Recognition of prior learning Credible qualifications, certification and accreditation Educational excellence NQF Principles Principles Fair Assessment Developmental Assessment Assessment should be developmental rather than judgemental, Credibility of site-based assessment remains a big challenge. Fit for Purpose Assessment Valid Assessment, Reliable Assessment , Integration, Relevance Credibility and Standards Flexibility and Recognition of Prior Learning, Legitimacy, Access and Redress Articulation and Portability.

4. Methodology • Hypothesis • Self-evaluation • Meetings and interviews • Site visits: Provincial offices, printing and distribution centres, districts • Documents analysis • Sampling • Plenary • Shortcomings identified in the 2008 and 2009 monitoring process

5. Specific Aim • Establishing system preparation for the 2010 NSC examination • Combat readiness for the Annual National Assessment • Credibility and integrity of Site-based Assessment • Preparation of learners for the 2010 national assessment and public examinations

6. MAKING VALID EDUCATION ASESSMENT JUDGEMENTS World of work HE

7. Operational Objectives High quality question papers are set and moderated Candidates are accurately registered & resulted on the examination computer system Analytical reports are generated Question papers are accurately printed, securely packed and distributed to examination venues All candidates including special needs catered for Answer scripts accurately marked and moderated All examination processes are conducted with integrity Prioritisation of security and accountability in the distribution of scripts and the collection of answer books

8. Operational Objectives SBA is completed in accordance with policy prescripts and moderated. All examination and SBA marks are accurately captured. All examination irregularities are appropriately managed. All candidates are accurately and timeously resulted. Effective feedback provided to schools. All eligible candidates receive certificates.

9. National Senior Certificate Examinations 2010 School-based assessment is ongoing throughout the year. Practical examinations in Visual Art and Design began on 16 August 2010. Practical examinations in Computer Applications Technology and Information Technology to be conducted from 13 to 15 October 2010 The first written examinations on 25 October 2010: Information Technology Theory, and Computer Application Technology Theory paper. The last examinations to be written on 3 December 2010: Religion Studies Paper 2 Civil Technology, and Hospitality Studies

13. Integrated Examination Computer System The examination computer system is the backbone of the administration of the NSC - the system is still being enhanced. All candidates have been registered on the Integrated Examination Computer System (IECS). Registration data has been audited by the DBE. A third schedule of Entries has been sent to all schools to ensure accurate registration of all candidates. At the end of August 2010, mark sheets will be printed. Letters of admission to the examination will be sent to schools in September 2010. In 2010 a full parallel run on the old system will be conducted to improve confidence in the processing of results Both systems are maintained and supported by the State Information Technology Agency (SITA) 13

14. Question Papers Security surrounding all processes relating to preparation of question papers is satisfactory. All question papers are set by the Department of Basic Education to ensure a national standard. 260 question papers have been set by panels of examiners and moderated subject specialists. The question papers have been moderated by Umalusi to ensure appropriate standard The first batches of question papers have been handed to PEDs for printing and packing Final handover of question papers is 10 September 2010. Question papers are being brailled for blind candidates and adapted for deaf candidates.

15. Printing, Packing and Distribution All staff working in the printing and packing sections have security clearance and have signed confidentiality agreements. PEDs have received the first two sets of question papers printing. Question papers will be distributed to examination centres on the morning of the examination, except where extreme distances are involved. 15

16. Preparation for the examination • Invigilators who supervise the writing of exams will be trained across all PEDs in September. • The principal is usually the Chief Invigilator. • Detailed regulations govern the conduct of exams and the managing of irregularities – DBE has issued a Guideline document to assist this process • Markers and chief markers have been appointed/ will be appointed in September 2010. • Marking centres have been finalized and markers will be trained before marking commences.

17. Answer scripts • Each answer script is the product of a candidate’s 12 years at school and his/her performance is encapsulated in the script. • In 2010, emphasis is being placed on the return of answer scripts, and management of scripts during marking process: • All scripts to be returned to the district office the same day • Scripts to enjoy the same attention and security as the question paper. • All scripts correctly labelled with exam number and subject. • Scanning of scripts to be introduced by all PEDs as soon as possible. • All scripts to be checked at each stage of the process. • Implementation of sanctions where Chief Invigilators do not manage the return of scripts correctly. 17

18. Examination Centres • All examination centres are audited to ensure adequate resourcing. • All examination centres are monitored, with a special focus on examination centres with a history of irregularities. • Centres that do not satisfy national criteria are de-registered and alternative centres are arranged. • This includes private centres.

19. Marking • The quality of marking will be the focus in 2010. • Marking centres have been established across all PEDs based on the prescripts contained in the Policy. • Markers have been appointed based on the PAM criteria. • Training of markers is ongoing and specialised training sessions will be organised prior to the marking sessions. • National Marking Guideline discussions will be hosted in Pretoria to ensure standardisation of marking across all PEDs • Internal moderators to be appointed for each subject at each marking centre. • DBE will appoint external moderators to evaluate the marking at each centre.

20. School Based Assessment (SBA) • School based assessment constitutes 25% of the final promotion mark. • SBA is moderated at school, district and provincial level. • PEDs monitor the collection of marks from schools quarterly. • DBE issued a Guideline document to assist PEDs with the collection, control and capture of SBA marks to address shortcomings experienced in previous examinations, such as outstanding marks. • SBA marks are currently being moderated by the PEDs and capture will be completed by 15 November.

21. Processing and Release 2010 • Marking will be completed by 16 December across all PEDs. • Mark Capture will be completed by 18 December . • A technical team will prepare proposals in respect of adjustments for the pre-standardization meeting – 21 December. • Pre-standardization meetings will be held on 22 & 23 December. • Umalusi standardisation meeting to be held on 24 December. • Results will be thoroughly checked by DBE, PEDs and Umalusi on 28 – 30 December. • Umalusi approval meeting to be held on 31 December 2010. • Results will be released on 6 January 2011.

22. Analysis and Feedback • Detailed analysis of results per school, per district, per province, per subject will be made available. • Reports from chief markers/internal moderators will be consolidated for distribution to schools. • Outcomes of the Umalusi standardisation meetings will be made available to curriculum specialists. • Workshops will be conducted with teachers and subject advisors. • Under-performing schools will be brought to account and improvement plans will be designed. • Curriculum specialists to develop subject improvement plans.

23. Overview of State of Readiness • The monitoring exercise was severely impacted on by the public service strike - not possible to visit district offices and schools in North West & Northern Cape. • It was established that schools are filtering candidates in respect of: • progression to Grade 12 and • the subjects that they may offer. • School Assessment Irregularities Committees are not yet functional. • The collection and reconciliation of answer books from examination centres the exams needs attention. • At several district and circuit offices storage facilities were inadequate. • In the electronic age, security of examinations now also means securing electronic copies of the question papers – this needs attention. • Varying SBA moderation practices within and across the provinces do not give confidence that all candidates treated in a fair & equitable manner. • DBE is engaged in the process of developing guidelines for moderation which should resolve the current plethora of practices.

24. Progress in Mpumalanga • DBE is managing the NSC, SC and ABET examinations in the province and working collaboratively with the regional staff. • DBE staff are deployed to the province. • Certain functions have been decentralised to regions while DBE will manage centralised functions. • Focus is on establishing improved systems and processes and building staff capacity. • Supplementary, Senior Certificate and June ABET Level 4 examinations have been successfully administered. • All question papers to be printed and stored in Pretoria, and will delivered on a weekly basis to the regional offices or nodal areas and daily to the examination centres. • Scripts will be collected from schools daily

25. the black box • McKinsey report: quality of an education system cannot go beyond the effectiveness of class room interaction • Low impact of teacher human capital means our measures are inadequate • School organization

26. NSC CHALLENGES Five week world cup recess may have impacted on the teaching and learning momentum. The teacher strike has resulted in a school shut down. NSC policy is still not fully understood by learners, parents and educators. Security of question papers. Introduction of a new examination computer system. Moderation of the quality and standard of SBA needs attention. Conduct of examinations at private centres is not always satisfactory.

27. 2010 ANA STATE OF READINESS Awareness: • On the whole provinces ,districts and schools visited are aware of the continued implementation of ANA in 2010. • Plans are afoot to strengthen advocacy around awareness especially in districts and schools. Communication strategy: • Advocacy around ANA needs to be strengthened and monitored • its purpose • administration • capturing, analysis, reporting, and • post assessment improvement plans and support • Communication and flow of information needs to be improved to ensure a consistent message communicated at all levels.

28. 2010 ANA STATE OF READINESS Statistics: Learner statistics must be verified before sending to DBE • provinces will be requested to deal with shortages. Storage of test papers in districts: Inadequate space to store the large quantities of Gr 1 – 6 test papers in districts. PEDs will need to rent warehousing space in some districts.

29. Recommendations • Facilitate and maintain rigorous moderation systems for SBA. • Develop a policy for the moderation of SBA • Monitor provincial moderation systems. • Moderate sample assessment tasks and learner evidence in the case of Life Orientation – July and October 2010. • Moderate SBA tasks and learner evidence.

30. Recommendations • Provide diagnostic feedback to schools. • Develop comprehensive analysis (qualitative and quantitative) of data. • Detailed reports made available to provinces, districts and schools. • Conduct post test analysis of ANA and public examinations.

31. Overview • filtering candidates in respect of progression to Grade 12. It was also evident in visits to schools that restrictions were placed on certain learners in respect of the subjects that they may offer. 2. the establishment and functionality of the School Assessment Irregularities Committee is neglected

32. Overview • PEDs need to focus on the collection and reconciliation of answer books from examination centres to marking centres and storage as adequate systems are not in place in all provinces to ensure that all scripts are accounted for. • Generally storage facilities at Provincial offices are excellent but at several district/regional offices and circuit offices storage facilities were well below the expected norms and standards for a secure storage facility and provinces must improve storage facilities as a matter of urgency.

33. Overview • Security is always a concern in examinations but in the electronic age, security of examinations is fast moving away from merely securing physical copies of the question papers. There is now a need to secure the electronic copies of the papers. •  Moderation practices vary across the provinces, and even within provinces. The Department of Basic Education is engaged in the process of developing guidelines for moderation which should ensure that each candidate is being treated in a fair manner.

34. Overview • One province has been requested to visit their printing and packing procedures to improve security in this area. • The Annual National Assessments will be printed nationally this year. Provinces indicated that this would alleviate many problems. The key area of concern at the moment is that schools provide accurate statistics in respect of learner numbers and the Language of Learning and Teaching.

35. CONCLUSION • Based on the information that was made available to the monitoring team, it can be concluded that the Provincial Departments of Education have made significant strides towards readiness for the NSC examinations • However in respect to the annual national assessment, learner preparation has been disrupted by the strikes and it is recommended that these are shifted to February 2011. There are educational reasons to support that such tests are in future conducted earlier than later in the year

36. Conclusion Learner preparation: Registration, irregularities, subject changes, item analysis Stratification of the learner attainment strategy System administration Validation of standards Credibility of Site-Based Assessment Logistics regarding the ANA Analysis of ANA results Item writing, banking and discrimination Promotion requirements Release of results Top candidates