Quality Assurance: Lessons from NSC Results

1. Wednesday, 3 June 2015 Dr Mafu S Rakometsi Portfolio Committee on Higher EducationReport on the Quality Assurance of NC(V) Level 2-4; NATED N1-3 and GETC: ABET L4

2. WHAT CAN WE LEARN FROM THE NSC RESULTS? Introduction to context, principles, approaches and processes Dr Mafu S Rakometsi Chief Executive Officer of Umalusi

3. Regulatory Framework • Established through the promulgation of the General and Further Education and Training Quality Assurance (GENFETQA) Act, no. 58, 2001, as amended 2008; • Two predecessors, namely the Joint Matriculation Board - JMB (1918) and the South African Certification Council – (SAFCERT) (1986); and • Umalusi started work in 2001 having taken over from SAFCERT.

4. Regulatory Framework GENFETQA ACT, 2001 amended 2008 Section 17A. (1) The Council must assure the quality of assessment at exit points. (3) The Council must perform the external moderation of assessment of all assessment bodies and education institutions. (4) The Council may adjust raw marks during the standardisation process.

5. Section17 of the GENFETQA Act • (5) The Council must, with the concurrence of the Director-General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has— • (i) conducted the assessment free from any irregularity that may • jeopardise the integrity of the assessment or its outcomes; • (ii) complied with the requirements prescribed by the Council for • conducting assessments; • (iii) applied the standards prescribed by the Council which a learner is required to comply with in order to obtain a certificate; and • (iv) complied with every other condition determined by the Council.

6. Regulatory Framework National Qualifications Framework (NQF) Act, 2008 (Act No. 67 of 2008): • The (3) Quality Councils were established. • Umalusi - Quality Council for the General and Further Education and Training. • Development, management and promotion of the GENFETQS-f. • The capacity of private providers to offer qualifications that are registered on the GENFETQS-f. NQF Act Section 27 (h) • The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework

7. Framework for Quality Assurance of Assessment • Evaluation and /or accreditation of assessment bodies; • Periodic inspection of assessment systems; • Ongoing monitoring of assessment systems; • Quality assurance of external examinations through: • Moderation of examination question papers; • Monitoring and moderation of SBA; • Monitoring the conduct of examinations; • Moderation of marking; and • Standardization of assessment outcomes.

8. Quality Assurance of the DHET, IEB and BENCHMARK November 2014 examination Ms Faith Ramotlhale Senior Manager: Quality Assurance of Assessment

9. Moderation of question papers PURPOSE To ensure that the question papers are:- • of the required standard; and • are relatively: • fair • reliable • representative of an adequate sample of the curriculum • representative of relevant conceptual domains • representative of relevant levels of cognitive challenge

10. Approach • Moderation Process • Question paper development - examiners (assessment body) • Internal moderation - Internal moderators (assessment body) • External moderation - External moderators (Umalusi) • Subsequent moderations and approval

11. Moderation of the question papers Criteria • Technical criteria • Internal moderation • Content coverage • Text selection, types and quality of questions • Cognitive demands • Marking memorandum/ guidelines • Language and bias • Predictability

12. Question Paper Approval

13. Moderation of the question papers Areas of Good Practice • No question paper leakages( all GETC- BM,IEB & DHET) • Questions were not predictable- Creativity and innovation (GETC- BM,DHET & IEB) • The setting of an extra set of papers, first step towards eighteen month cycle, was a step in the right direction (DHET-NC(V)) • To ensure equivalence of standards, the November, March and back-up examination question papers were moderated simultaneously, (DHET-NC(V) and NATED)

14. Moderation of Question Papers Areas of concern • Improve the quality of internal moderation, (DHET, IEB, BM). • Training of examiners and internal moderators to enhance understanding of Subject and Assessment Guidelines, (GETC- DHET & Benchmark) • Focus areas of training: depth and breadth (ECD4, INCT4, LCZU4, LCSO4, LCXI4, TECH4, TRVT4 and WHRT4), (GETC-DHET) • Poor quality memoranda - over reliance on Memorandum Discussion process to improve quality of markingguidelines, (GETC-DHET)

15. Moderation of Question Papers Areas of concern • The question papers for A4TECH & A4HSSC did not meet the cognitive demands at first moderation (GETC-IEB) • Correct versions question papers, marking guidelines, internal moderator reports and assessment frameworks to be made available for external moderation, (DHET -NC(V)) • Resubmission of rejected question papers to Umalusi – turn around time to be improved, (DHET - NC(V)& NATED) • Standard of back-up papers to be the same as November and supplementary papers, (DHET-NC(V) • Quality and standard of internal moderation – a rigorous process should be evident, (DHET-NATED and NC(V))

16. Moderation of Question Papers Areas of concern • Outdated and/or underspecified curricula requires urgent revision (DHET-NATED); • Predictability of papers due to repetition of questions from previous question papers should be avoided; (DHET-NATED), and • Quality of assessment grids and marking guidelines should be improved (DHET-NATED, NC(V)).

17. Internal Assessment (SBA) Definition and Purpose • Internal assessment refers to any assessment conducted by the provider , the outcome of which count towards the achievement of the qualifications • GETC: 50%; NC(V) fundamental subjects 25%; NC(V) vocational subjects 50%; and NATED subjects 40% • Ascertain the degree to which the assessment body is attempting to ensure standardisation across providers • Ascertain the standard and quality of the tasks • Establish the extent and quality of internal moderation and feedback, and • Verify the reliability and validity of the assessment outcomes

18. Site-Based Assessment: GETC Areas of Good Practice • Year-on-year improvement regarding the administration compliance requirements, (DHET) • The SBA user guides developed by IEB were helpful in structuring the content of Common Assessment Tasks

19. Moderation of Internal Assessment - GETC

20. Moderation of Internal Assessment - GETC Areas of concern • No system to monitor the implementation of the CATs, (IEB) • The CATs did not adequately cover the specific assessment criteria as specified in the unit standards at first moderation, (BM) • The weightings of the CATs also did not comply with the SAG requirements, (BM) • Centres do not submit educator portfolios with sample requested – no evidence of assessment plans (IEB, Benchmark, DHET) • Assessors (educators) do not understand the use of rubrics, and marking guidelines (Benchmark, DHET)

21. Moderation of Internal Assessment - GETC • Areas of concern • Poor compilation of SBA portfolios and insufficient supporting documents provided, (Benchmark) • Very poor internal moderation system and processes (DHET) • Authentication is ad-hoc and sporadic (considering 50% weighting) (DHET) • Substantial differences between SBA and examination marks (DHET).

22. Moderation of Internal Assessment NC(V) • Areas of concern: ISAT • The quality of some tasks remains a cause for concern • Enrolments must be in line with available resources to effectively assess candidates’ acquisition of the practical skills required in the workplace • ISATs should be distributed to colleges in time for effective procurement and implementation • Care should be taken to prevent subjective and overly lenient marking • Moderation should be improved • Absenteeism among students affected the completion of ISATs,

23. Internal Assessment : ICASS Directives for Improvement (NC(V) • Improving and deepening the understanding of the latest ICASS Guidelines (NCV) • Development of assignments and tasks • Implementation of practical assessments is still problematic, • Implementation of analysis grids and Bloom’s taxonomy • Development of marking tools and rubrics • Assessment and moderation practices need improvement

24. Moderation of Internal Assessment ICASS Areas of good practice: (NATED) • There were pockets of excellence at some of the sites moderated. dedication of innovative lecturers Areas of concern • To ensure the availability and implementation of the latest version of ICASS guideline • Training of lecturers - limited experience in the industry • Teaching aids e.g. models are needed to improve learners’ understanding of the subject content • To address class attendance and retention rates at some colleges

25. Monitoring of Examinations: GETC • Areas of Concern (DHET) • Lack of storage rooms and inadequate security for question papers • Transportation of exam material using public transport (EC) • Candidates allowed to write without proper identification • Some centres keep answer scripts overnight - poor security

26. Monitoring of Examinations: GETC Areas of concern • Low number of candidates who wrote the exam, (IEB, BM) • Poor compliance with exam regulations (BM, IEB) • Inadequate storage facilities for QPs with poor security arrangements • Poor administration regarding exam regulations • Chief invigilators lack expertise to manage exams • Private exam centres often apply to the IEB to conduct the exam on alternative dates (IEB)

27. Monitoring of Examinations: TVET • Areas of good practice • General management of the examinations was good in terms of security of papers, appointment and training of invigilators • Areas of Concern • Irregularities during the writing phase of the examination, candidates were found with cell phones • Others entered the examination without identity documents or examination • Some Colleges monitored, papers were not sent to the marking centre within 24 hours, nor securely sealed

28. Monitoring Of Marking Phase: GETC • Areas of good practice • Marking centres were generally well organised and suitable for the task, (DHET,IEB,BM) • Adequate and experienced security personnel at the gates and entrances to the venues, (DHET,IEB, BM) • Training of marking personnel was generally thorough and effective, (DHET,IEB,BM)

29. Monitoring of Marking Phase: GETC • Areas of Concern • Shortage of markers in NC and KZN due to appointing markers for more than one learning area, (DHET) • Internal moderation model different and inconsistent across marking centres,(DHET) • Markers in LP disturbed the training process – apparently because of outstanding remuneration issues, (DHET) • The IEB marking centre management plan was not well documented – its based on experience and practice, (IEB) • There was no security in the building where the actual marking was done to control the flow of scripts, (BM)

30. Monitoring of Marking Phase: TVET • Areas of good practice • Safety and security of marking personnel and material was in general very good • Markers were aware of what constituted an irregularity and how it should be dealt with.

31. Monitoring of Marking Phase: TVET Areas of concern • Distribution incorrect versions of marking guidelines - English and Afrikaans • Quality of marking and internal moderation of low enrolment subjects • Inappropriate allocation of marking venues according to the nature of subject, and number of markers - drawing subjects

32. PURPOSE • Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices • Umalusi engages with the following processes during the moderation of marking: • Centralised memorandum discussions • Verification of marking (centralised and on-site) Verification of Marking

33. Areas of good practice • Good procedure observed: Panel discussions followed by marking of exemplar scripts – marking commences after markers have demonstrate an acceptable level of competence, (IEB,BM) • Analysis of evidence bound in educator portfolios were meticulous and contained all relevant documents as per policy requirements Areas of concern • Poor practice of pre-marking in 15/26 LAs – discussions very difficult and time-consuming, (DHET) • No pre-marking or preparing alternative responses in advance, (IEB) Memorandum Discussions

34. Memorandum Discussion TVET Areas of good practice • Sample marking improved Areas of concern • Appointments should be finalised well in advance • Marking personnel must attend marking guideline discussions, • Preparation of marking personnel for the marking process must be improved • Correct version of marking guidelines must be made available

35. Areas of Good Practice • Good internal moderation practices (DHET, IEB) • Good quality and standard of marking, (DHET, IEB) Verification of Marking: GETC

36. Standardisation and Resulting • Provision of GENFETQA – Council may adjust raw marks • International practice – large scale assessment systems • Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance • Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

37. Verification of the Resulting Process

38. Statistical Moderation Scope of standardisation 2014: GETC

39. Statistical Moderation Areas of good practice • Booklets and data sets were submitted according to the prescribed timelines (IEB, BM) • Standardisation and approval meetings went smoothly

40. Statistical Moderation Areas for Concern • Late submission and inaccurate standardisation data delays the approval of results, (DHET) • Late submission of qualitative input reports, (DHET) • Poor performance of learners,30-40% candidates do not write the exam, (DHET) • Non-adherence to the prescribed colours to draw the graphs (IEB) • Very low enrolment numbers (BM)

41. Statistical Moderation Scope of standardisation 2014: NATED / NCV

42. Statistical Moderation: TVET Areas of good practice • Standardisation booklets complied to the prescribed colours Areas of concern • Late submission of standardisation booklets delayed standardisation • Inaccurate data sets and low capture rate of data delayed the approval of results • Non submission of consolidated irregularity report to Umalusi for standardisation and approval of results

43. Conclusion • The quality assurance processes presented above were conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations • From the findings of the quality assurance processes many good practices were evident but on the other hand, there are still many challenges that need to be addressed • Significant interventions are required to improve the curriculum, teaching and learning, exposure of learners to a compulsory practical component and assessment for the NATED programmes to make it a meaningful qualification

44. Conclusion • The outdated and underspecified curriculum and lack of subject assessment guidelines for the NATED programmes creates problems for appropriate quality assurance to be conducted • Umalusi takes this opportunity to express appreciation to the Department of Higher Education and Training for their concerted efforts in delivering credible examinations in 2014

45. Thank you!