Queenie CC Lam Hong Kong Observatory 8 December 2011

1. Sharing of Experience from Hong Kong, China on the Implementation of AMP Competency Assessment System Queenie CC Lam Hong Kong Observatory 8 December 2011 WMO VCP Workshop on Implementing Competency Assessment for Aeronautical Meteorological Personnel Hong Kong, China, 5-9 Deceomber 2011

2. Outline • Implementation plan of Competency Assessment System (CAS) for Aeronautical Meteorological Observers (AMOB) and Aeronautical Meteorological Forecasters (AMF) • Implementation steps and tools used • Feedback from assessors and assessees

3. HKO’s Implementation plan of CAS • Oct 2010 Gap Analysis • Dec 2010 – Jan 2011 System and Tools Design • Feb 2011 Refresher Training Courses & Briefing on CAS for regular Aviation Forecasters & Observers • Mar 2011CAS Launch Assessment for regular Aviation Forecasters & Observers • Apr-May 2011 CAS assessment summary and follow-up action ; System review and refinement • Jun 2011 Briefing on CAS for non-regular Aviation Forecasters & Observers • Aug-Oct 2011 Assessment for non-regular Aviation Forecasters & Observers • Nov 2011 CAS assessment summary,system review and refinement • 2012 Integration of CAS into QMS

4. Implementation Steps Initial Review System Design Feedback Preparation of Documents System Implementation System Review

5. Implementation steps and tools used – Initial review • Scope and Target Assessees • In the CAS-AMOB, the target assessees for assessment include : • All Weather Observers • Scientific Assistants who are trained Weather Observers and may take up weather observation duties at the Airport Meteorological Office occasionally • In the CAS-AMF, the target assessees for assessment include : • All regular and occasional Aviation Forecasters • Experimental Officers who are trained Aviation Forecasters and may take up weather forecasting duties at the Airport Meteorological Office occasionally

6. Responsibilities • Top Management • Officer-in-Charge • Lead-Assessor of CAS-AMOB • Lead-Assessor of CAS-AMF

7. Initial Review – Gap Analysis • Gap analysis using cross-reference table method

8. Initial Review – Gap Analysis (2) • HKO’s gap analysis cross-reference table for AMOB

12. Initial Review – Gap Analysis (3) • HKO’s gap analysis summary for AMOB

13. Initial Review - Refresher Training • Organise refresher training courses to address the knowledge gap

14. System Design HKO’s Competency Assessment System (CAS) Overview

15. System Design – Assessment Tools • Direct observation • Oral/Experiential questions • Written assessment • Case study/simulation

16. System Design –Competency Assessment Matrix for AMOB

17. System Design –Competency Assessment Matrix for AMF (1)

18. System Design –Competency Assessment Matrix for AMF (2)

19. System Design –Selection of Assessors • Assessors • Staff responsible for the supervision of AMP and selected trainer(s) of the Weather Observation Course and Meteorology Course for Aviation Forecasters as appropriate • satisfy the qualification criteria for a trainer on the subject as specified in Quality System Procedure on training • Preferably possess knowledge of internal audit • For assessors of AMF – qualified as Meteorologist following WMO requirements

20. System Design – Officers • Lead-Assessor • prepare and conduct assessments in coordination with other assessors • recommend overall competency rating of AMP • Maintain individual AMP portfolio • Officer-in-Charge, Division Head of Aviation Weather Forecast and Warning Services • oversee the effective operation of the CAS • endorse the competency rating (either “Competent” or “Not Competent”) • Top Management, Assistant Director of the Aviation Weather Services Branch • approve the structure and implementation plan of the CAS • ensure the integrity of CAS as part of the QMS

21. System Design – Schedule & Validity • Assess every two years (similar to the frequency of revalidation test for ATC controllers in Hong Kong) • Guidelines for determining the schedule • Allow flexibility to cater for different types of weather situations in different seasons • Last around 1 day for each assessee • Agreed between assessor and assessee • Competence notification will carry a validity period of 24 months (in consistent with the assessment frequency) • revalidation of competence notification for another 24 months before expiry

22. Comments and engagement from assessors and assessees System Design – Staff Engagement

23. Preparation of Documents - Checklists Example : a sample of HKO’s checklists for Aeronautical Meteorological Observers

24. Preparation of Documents –Assessment Papers • Around 30 multiple choice questions • Around 10 short questions • Time allowed 2 hours • Passing mark 70

25. Preparation of Documents –Case Study/Simulation

26. Preparation of Documents – Assessment Procedures • Prior to assessment – prepare for assessment • During assessment – conduct and record assessment • Post assessment – feedback,document, evaluate, review and follow-up • Reassessment procedures

27. System Implementation • set implementation date • Conduct briefings to assessors and assessees • arrange assessment dates • Carry out assessment process using planned tools based on competency assessment matrix • Collect evidence and produce records • Keep records of training, qualification, years of forecasting experience, competency assessments in the Competency Portfolio of individual AMP

28. System Review (1) • Implementation plan on schedule • Feedback/Suggestions from assessors and assessees and observations • Observers see the assessment as a good training and learning mechanism • facilitate continuous improvement • suggest regular refresher courses • Suggest to allow more time for preparation by assessees in between refresher training courses and assessments • Suggest to record answers in softcopy form when taking written assessments especially the short questions to save time

29. System Review (2) • Minimal oral questions during direct observation assessment to minimise interference to operational work of the assesses • Combined testing of competency performance criteria as far as practicable • Make use of routine weather briefing (such as weather conference/teleconference, handover briefing, briefing to clients) to evaluate the assessee by direct observation • Refine CAS based on feedback/suggestions and observations • Implement suggestions made in the review

30. Sharing of Implementation Plans • Webpage on CAT developed by CAeM TT-CAT on the CAeM training website(http://www.caem.wmo.int/moodle) “CAT” under “Regulatory and Reference Material” Login as a guest

31. Q&A