International Vocational Educational Organization (IVETA) Conference - December 3 & 4, 2013, Las Vegas, Nevada, U

1. International Vocational Educational Organization (IVETA) Conference - December 3 & 4, 2013, Las Vegas, Nevada, USA • QUALITY ASSURANCE IN TECHNICAL VOCATIONAL EDUCATION & TRAINING (TVET) HEART TRUST/NTA (JAMAICA) Approaches to Monitoring & Evaluating TVET Programmes Presenter: J Wallder

2. TVET – CARIBBEAN REGION IMPERATIVES • Jamaica • An island nation surrounded by the Caribbean Sea. • Located 90 miles south of Cuba; 118 miles west of Haiti. • 145 miles in length, up to 50 miles in width and 4,243 sq. miles in area. • Population – approx. 2.7 million. Dec. 2013

3. TECHNICAL VOCATIONAL EDUCATION & TRAINING (TVET) – DEFINITION • United Nations Educational, Scientific, and Cultural Organization (UNESCO) & International Labour Organization (ILO), 2002: • Those aspects of educational process involving the study of technologies and related sciences (including general education) and the acquisition of practical skills, attitudes, understanding and knowledge relating to occupations in various sectors of economic life. Dec. 2013

4. TVET – CARIBBEAN REGION IMPERATIVES • Caribbean Community (CARICOM) Council for Human and Social Development (COHSOD), at its 19th Meeting, April 2010, established plans to review the 1990 CARICOM Regional Strategy for TVET to consider new realities of the world of work. • CARICOM Secretariat stated “it has become appropriate at this time to review the strategy primarily because global trends have indicated a need for a knowledge-basedand skilled labour market”. Dec. 2013

5. TVET – CARIBBEAN REGION IMPERATIVES • Implications: • Education & training – more relevant to meet market needs • Emphasis on lifelong learning/continuous retooling • Renewed strategies to be aimed at workers at all levels of operations – operational, managers, professionals, technical experts. Dec. 2013

6. THE HEART TRUST/NTA - JAMAICA • The Human Employment and Resource Training Trust/National Training Agency (HEART Trust/NTA) • VISION STATEMENT: • The HEART Trust/NTA is committed to the systematic design, development and delivery of an integrated, flexible and responsive TVET system, enabling a productive workforce for national priorities and global competitiveness Established as a statutory body under the HEART Act in 1982 to govern the development and delivery of TVET in Jamaica. Dec. 2013

7. THE HEART TRUST/NTA Govt. of Jamaica (2030 Plan) Key Ministries: Education, Labour Board of Directors Key Drivers of the HEART Trust/NTA UNESCO ILO CARICOM Dec. 2013

8. THE HEART TRUST/NTA - JAMAICA • Core business - training and certification of Jamaican citizens for employment, while regulating and setting standards for the TVET system in Jamaica. • 30 Training institutions – current focus towards tertiary level status; integration of levels 1 & 2 programmes into secondary schools. • Partnering training institutions (CTIs: 79); • (Private Providers : 12) • 2010 – 2013: Approx. 100 skill areas offered (levels 1 – 5) • Average # certified per year: 18,300 Dec. 2013

9. ORGANIZATIONAL RESTRUCTURING APRIL 2013 - NEW MONITORING & EVALUATION DEPARTMENT ESTABLISHED • Key Objectives: • Framework to guide M&E activities • Performance standards for TVET design, operations and delivery • Monitoring and evaluation data collection and feedback mechanisms/tools to facilitate strengthening of processes • Monitoring and evaluation audits (across the TVET system) against the relevant standards towards programme accreditation and quality improvement • Evaluations to ascertain programme effectiveness, • impact and customer satisfaction within the TVET system • Provision of timely information to inform strategic decisions. Dec. 2013

10. SETTING UP THE M&E SYSTEMFACTORS TAKEN INTO ACCOUNT Dec. 2013

11. HEART TRUST/NTA – INTERNAL QUALITY SYSTEMS • The Organization • Guided by Act of Parliament/sound mandate; other key drivers • Planning system premised on Balanced Scorecard methodology • Comprehensive structures to support TVET programmes • Training Institutions • Required to establish comprehensive Policy and Procedures Manuals (PPM) • Management/administrative structures • Internal Quality Assurance Committees – monitors activities/alignment with PPM • The National Council for Technical Vocational Education & Training (NCTVET) • Responsible for quality assuring the TVET system • Established quality framework for developing competency standards, along with guidelines/procedures for delivery and assessment activities • Accredit TVET training providers and training programmes Dec. 2013

12. HEART Trust/NTA – Internal Quality Systems • Standards used by the NCTVET • Quality Assurance & Control/Governance • Staff Resources • Physical Resources • Client Services & Activities • Training • Assessment • Programme Evaluation Dec. 2013

13. HEART TRUST/NTA – EXTERNAL QUALITY SYSTEMS • The University Council of Jamaica (UCJ) • statutory body – established by the University Council of Jamaica Act, 1987. • Primary focus – tertiary educational institutions; Institutional registration and programme accreditation • Quality standards used by the UCJ: • Governance and mission • Academic programmes being offered • Admission policies and procedures • Staffing • Student assessment • Student support services • Physical accommodation • Library/resource centre • Laboratory/computer facilities • Finances Dec. 2013

14. MONITORING & EVALUATION – GLOBAL TRENDS • Globalization - increased pressures • on governments and organizations around • the world to be more responsive to the • demands of internal and external • stakeholders for: • Good governance • Accountability • Transparency • Greater development effectiveness • Delivery of tangible results. • Translated into attendant need for results-based monitoring and evaluation of policies, programmes and projects. Dec. 2013

15. MONITORING & EVALUATION – GLOBAL TRENDS • Definitions - Organization for European • Co-operation and Development (OECD) Monitoring Evaluation The systematic and objective assessment of an ongoing or completed project, programme, or policy, including its design, implementation, and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability. A continuous function that uses the systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress, and achievement of objectives in the use of allocated funds. Dec. 2013

16. Monitoring & Evaluation – Global Trends • Monitoring & Evaluation - Two Definitions Distinct but Complementary Monitoring Evaluation • Evaluation Gives Evidence of Why Targets and Outcomes are/not being Achieved: • Analyzes why intended results were or were not achieved • Explores unintended results • Provides lessons, highlights significant accomplishments or programme potential, and offers recommendations for improvement • Monitoring Gives Information on Where a Policy, Programme, or Project is at Any Given Time: • Clarifies programme objectives • Routinely collects data on indicators; compares actual results with targets • Reports progress to management and alerts them to potential problems Dec. 2013

17. Rationale for a Monitoring and Evaluation Framework for Jamaica’s TVET System • Training delivery of TVET programmes; • executed through all HEART training • institutions, partnering/Community • Training Institutions, • secondary school system, • private providers. • Must ensure standardization of inputs, • processes, outputs, outcomes and • impact across training entities. Why Monitor & Evaluate? Dec. 2013

18. RATIONALE FOR A MONITORING AND EVALUATION FRAMEWORK FOR JAMAICA’S TVET SYSTEM • The NCTVET (and other accreditation/ • certification bodies) - QA for accreditation. • M&E - critical support function – • seeks to monitor/track • progress of delivery entities. • Ensuring systems and processes are • established against quality/accreditation standards; • In state of readiness for accreditation/verification • activities. Why Monitor & Evaluate? Dec. 2013

19. A SYSTEMS APPROACH TO M&E • Monitoring and Evaluation Framework established, guided by a Systems Approach, to facilitate effective monitoring and evaluation activities of Jamaica’s • TVET system. Dec. 2013

20. SYSTEMS APPROACH TO MONITORING AND EVALUATION Inputs Financial, human, and material resources used for implementation of TVET programmes Technical Expertise, Approved Curricula, Equipment, Funds Systems established; work performed Strategic planning, Organization structure, Roles/responsibilities established; quality assurance systems; Training programmes implemented Processes Outputs The products and services that result from the TVET system Number of persons trained and certified Dec. 2013

21. SYSTEMS APPROACH TO MONITORING AND EVALUATION Outcomes • The likely or achieved short-term and medium-term effects or changes arising from persons trained and certified • Increased skilled labour force • More persons obtaining employment • The long-term results of the implementation of TVET programmes • Persons enjoying improved standard of living • Improved national productivity • Positive impact on GDP Impact Dec. 2013

22. A SYSTEM RESULTS-BASED APPROACH M&E MODEL • M&E aligned with goals/objectives of the Organization • Captures context of the TVET system • Tracks progress on inputs, processes, outputs and outcomes of the system • Conducts impact evaluations to measure actual results • Documentation & reporting mechanisms; inform stakeholders & mgtment decisions. Dec. 2013

23. ALIGNMENT OF OBJECTIVES WITH GOALS OF ORGANZATION/TVET SYSTEM Org. Goals: A Robust Quality Management System Implemented & Maintained • Provide ‘on the ground’ capacity building activities through guidance/coaching, sharing of best practices, sharing of resources, follow-up M&E activities Monitor and evaluate systems and processes of the TVET system to facilitate conformance with educational quality standards M &E Framework in accordance with international standards developed and implemented Facilitate preparation for accreditation activities Provide timely information to stakeholders of the TVET system Dec. 2013

24. DESIGN OF THE M&E SYSTEM: Context of TVET System, Stakeholders, Resources , Etc. Dec. 2013

25. KEY STAKEHOLDERS OF THE M&E SYSTEM; ENGAGEMENT • The Ministry of Education • The HEART Trust Board of Directors • The Senior Executive Team • Provide strategic direction and resources • Directors of the HEART Trust/NTA • Senior Director (Workforce Development & Employment Division) • Directors/Principals • Instructor/Teachers • Employees of training entities • Learners • Accreditation bodies: • NCTVET • UCJ Dec. 2013

26. KEY STAKEHOLDERS OF THE M&E SYSTEM; ENGAGEMENT • The TVET Development and Support Systems Division • Learning Resources Development (LRD) Department • TVET Consultancy and Institutional Capacity Building (TCICB) Department • Vocational Training Development Institute (VTDI) • Monitoring & Evaluation Department • The Monitoring and Evaluation Department – Process Owner • Its mandate: to effectively implement and maintain the M&E system • Responsible for engaging all stakeholders to facilitate successful implementation Dec. 2013

27. CRITICAL SUCCESS FACTORS FOR SUCCESSFUL IMPLEMENTATION • Sound understanding of organizational and wider TVET objectives. • Obtain requisite resources (through budgetary system) – staffing, funding for operations, accommodation & equipment. • Critical areas of competence of M&E staff - working knowledge of quality assurance systems and measures. • Familiarization with TVET training programmes, including competency-based delivery and assessment methodologies. Dec. 2013

28. CRITICAL SUCCESS FACTORS FOR SUCCESSFUL IMPLEMENTATION Development/imp. of macro plan for M&E f • Target training institutions for M&E • Timelines • M&E personnel assigned, including collaborations with partnering entities (e.g., CTIs and MOE schools) • Budget to facilitate M&E operations. Rationalize/dev./imp. appropriate M&E tools and strategies • Educational quality standards (consistent with NCTVET/UCJ) • Document review Portfolios of work • Questionnaires Interviews • Observation rating scale inventory check-list status reports Dec. 2013

29. CRITICAL SUCCESS FACTORS FOR SUCCESSFUL IMPLEMENTATION Documentation Facilities • Template (incorporating QA standards) for M&E audits & reporting • Results-based evaluations – standard report structure • Monitoring - via monthly score card reporting process; information is fed upwards from individual, to dept, to executive/board/Ministry levels. • Needs assessment currently underway to inform integrated IT system for M&E Dec. 2013

30. IMPLEMENTATION OF THE M&E SYSTEM: Performance Tracking; Use of QA Standards; Evaluation & Documentation Dec. 2013

31. PERFORMANCE TRACKING Performance Tracking Monitoring the TVET System Checking for consistency between what was intended against what is actually taking place. Tracking progress of the implementation process. Identifying system strengths and weaknesses. Documenting experience and appraising performances, e.g., assessing training performances against objectives; & making recommendations for adjusts where necessary Dec. 2013

32. IMPLEMENTATION - PERFORMANCE TRACKING Standards for Monitoring the TVET System NCTVET – primary QA body for TVET programmes - accreditation standards are adopted Provide general orientation/reinforcement on QM issues Conduct M&E audits to facilitate readiness for accreditation/NCTVET external verification evaluations. Reinforce quality-consciousness among TVET stakeholders. Dec. 2013

33. IMPLEMENTATION - PERFORMANCE TRACKING Example of Utilization of Standards in Reporting Template Dec. 2013

34. IMPLEMENTATION - PERFORMANCE TRACKING Types of Tools for M&E Dec. 2013

35. INFORMATION FROM M&E TRACKING PROCESS INFORMS FINAL REPORTS Directors, Principals, Managers of Training Insts. Ministry of Education Relevant Senior Directors Completed M&E Reports M&E Department Director, TCICB – for Capacity-buidling Issues Dec. 2013

36. M&E FOLLOW-UP PROCESS • Review content of report • Develop action plan with timelines to address gaps identified during the M&E audit • Agree on schedule to facilitate follow-up reviews • ensure action items are being addressed (inc. recommended capacity building issues) • M&E Officer required to follow-up with the training institution to: • Confirm receipt of M&E report • Establish convenient time to meet with the wider management team of the training institution Dec. 2013

37. EVALUATION Performance tracking/monitoring – not enough to measure real results. Evaluative information required to determine whether or not the intended outcome/result/impact is being achieved. Aims to determine the relevance and fulfillment of objectives, effectiveness, impact, and sustainability of the TVET system. Dec. 2013

38. EVALUATION • Sub processes to consider to inform Evaluation process: • Diagnosing needs, capabilities and capacities. • Determining baseline/starting-point performance. • Pilot-testing/trial-run to validate tools and strategies. • Formative evaluation of inputs, processes, outputs and impact. • Summative or outcomes evaluation. • Impact assessment (main effects and side effects). Dec. 2013

39. EVALUATION • Strategies for Evaluating the TVET System: • Directives from Senior Management – e.g., evaluations on the effectiveness of the restructured training system. • Evaluations on selected training interventions to analyse possible impact that has been made towards quality improvement. • Evaluations on certification rates. • Tracer studies. • Training impact evaluations. Dec. 2013

40. DOCUMENTATION AND DISSEMINATION OF M&E INFORMATION stakeholders are able to effectively apply information in their daily work for accountability, improvements in performance, decision making and learning. • Reports generated from M&E activities appropriately documented and disseminated • The M&E model - • serve to foster/promote learning and enhance knowledge management processes. Provide critical support for the TVET system’s strategic process thereby strengthening sub-processes to facilitate the achievement of objectives. Improve the overall performance and quality of results of ongoing and future programmes, projects and strategies. Dec. 2013

41. DOCUMENTATION & DISSEMINATION OF M&E INFORMATION Reports emanating from M&E activities – to stakeholders Updates via Balanced Scorecard reports: monthly/quarterlyannually . Examples of documentation and dissemination of M&E information Info/updates shared at meetings: dept.; div.; executive; Board; MOE Action plans generated via M&E audits – Follow-ups & Feedback Existing ICT infrastructure; NA currently being done to inform Integ. M&E system Quarterly progress reports (M&E audits) Dec. 2013

42. MONITORING THE M&E SYSTEM CONTINUOUS FEEDBACK • Continuous feedback from TVET system: • Training Institutions • Directors/Principals/Instructors/ • Learners • Senior Management, etc. • Comprehensive questionnaire administered annually • Information gathered - basis for learning/inform improvements in M&E systems/processes Dec. 2013

43. JAMAICA’S APPROCH TO MONITORING AND EVALUATION OF ITS TVET SYSTEM THANK YOU Dec. 2013