Exams and assessing competence

1. Exams and assessing competence

2. Part A/Paper 1 changes • OSPHE • ARCP • Assessing competence in HK training – the role of exams

3. Part A changes -What • Paper I - ‘Knowledge Paper‘ – No change • Paper II - ‘Skills Paper‘ Proposed change - 2b to become a short answer paper – 90 minutes, five sections (10 marks per section) – short questions with short answers (numerical or no more than a paragraph) - no choice – questions will not be published on the website

4. Part A – Why change? • PMETB - direction of travel for work that must be done to ensure that the MFPH is part of an assessment programme that enables the public, the profession and the candidates/trainees to have confidence in those decisions taken on the basis of examination results • Changes in response to long term aspirations of Southgate review

5. When • Needs PMETB approval before final confirmation (PMETB panel meeting January 2009) • Commencement set for Jan 2010

6. The OSPHE • “Objective Structured Public Health Examination”

7. Why the need? • Validity of the General Viva • Training issues with time spent on thesis • Issues around the limited competencies assessed by thesis • Issues around the narrow range of assessment available with thesis marking and viva

8. OSPHE –part B MFPH • After Part A (Knowledge & Knows How) • OSPHE (Shows How) • Reliable • Valid • Will encourage participation in service training opportunities during training as this will be the best form of preparation.

9. Milestones • Part A – 16m after starting Masters • Part B – 6-9m wte after Part A

10. What is assessed in the OSPHE? 1. Presenting communication skills in a typical Public Health setting (presenting to a person or audience). 2. Listening communication skills in a typical Public Health setting (listening and ascertaining key information). 3. Demonstrating ascertainment of key Public Health points from the material provided and using it appropriately and in relation to wider Public Health information sources. 4. Giving a balanced view/explaining appropriately key Public Health concepts in a Public Health setting. • Appropriately and sensitively handling uncertainty, the unexpected, conflict and/or responding appropriately to challenging questions. ALL the above are part of Curriculum

11. The OSPHE • It is a test of 5 competencies – NOT performance at individual stations • It is possible to not do very well at 2 out of the 6 stations – and pass • Feedback is given as a score for each competency overall - not overall station scores.

12. FORMAT • 12 Stations of 8 mins in each. • 6 of them are briefing/preparation time for the following assessed stations • Giving a short presentation • Explaining something verbally to a health professional , lay person , representative , manager etc • Media Interview role play • Phone conversation • Small meeting

13. OSPHE - format • Candidates have 6 stations which equally and independently tests all five competencies • Each station lasts 17 mins • 8 mins of briefing/preparation time • 1 min changeover • 8 mins assessment time

14. Marking • Examiner watching and marking during exam • 5 grades • Weightings • Automated scoring

15. The OSPHE • Taken around mid-point stageof training • We do not require candidates to be familiar with NHS procedures or any particular culture. • Question & Marking examples on Faculty website www.fph.org

16. Advice to post part A trainees • “Spend as little time as possible at desk using PC” • “Spend as much time as possible doing active service PH ; meetings , discussions with other professionals , managers , lay representatives of interest groups , discussions with journalists , radio interviews etc

17. As the Trainer • Ensure their service work/projects are in keeping with lots of talking & interaction • Every week before trainer-trainee meeting think of something which has happened over last week suitable for a mini-OSPHE • Role play it with your trainee – then give them constructive feedback • Try to be able to observe your trainee in meetings etc – and give them feedback

18. 350 Euston Road

19. Waiting Area

20. Briefing Room

21. Exam Corridor

22. Prep Room

23. Exam Room

24. Assessment • Exams • Work based assessments Includes in-work and real-time assessments, directly observed procedures (DOPs), case-based discussions (CBDs), multi-source feedback (MSF)

25. Evidence • Presented through portfolio • In a variety of formats (including supervisor reports) Assessed at ARCP

26. ARCP - Annual Review of Competency Progression • Deanery process • Requires reports from Educational Supervisor & Academic Supervisor • Based on documentation only • Sign-off sheets • Sample of workplace assessments • Sample of log-sheets • Other records (MSF, exam etc)

27. Documents Required for the Review by the ARCP Panel • Educational Supervisors reports to include any identified problems with competencies • Copies of all WBAs completed in that year • Updated CV – to include a sheet indicating new achievements since last review such as: examinations, audits, presentations and posters, research, record of CPD for that year, details of placements for that year. • Personal development plan for the year • Results of MSF • Printout of log book if relevant

28. Educational Supervisor Report

29. Academic Supervisor Report

30. Workplace Assessment Record Sheets

31. ARCP Panel • Consider the evidence presented • Consider progress against time-line • Decide on outcome • Provide feedback (Involves lay representation)

32. How the Panel Works • Convened by the deanery - a minimum of 5 members • Chaired by the chair of the STC, TPD or Associate Dean • Process is an assessment of documentation and submitted evidence • Trainee should not normally attend the panel meeting • Panel does not plan remedial programme • Other issues may be discussed by the panel but assessment of evidence and judgement must be kept separate

33. ARCP Outcomes • 1 – Satisfactory progress • 2 – Specific development needed • 3 – Inadequate progress • 4 – Left programme • 5 – Incomplete evidence • 6 – Completed training

34. Outcomes from the ARCP panel • Satisfactory progress – achieving progress and competences at expected rate. • Unsatisfactory progress – further development of specific competencies required but additional training time NOT required i.e. acceptable progress overall and not expected to delay training. TPD and ES will make sure the trainee and employer know what must be done.

35. Continued…... 3. Inadequate progress and additional training time required – panel identifies need for additional training time and recommends what is required and the level of supervision BUT the deanery determines the details depending on local circumstances and resources. The total overall extension should be no more than 1 year. An absolute maximum of 2 years during the total of training can only be granted by the Postgraduate Dean. Time can be made up of shorter periods of time divided over the course of training.

36. Continued…... 4. Released from programme with or without specified competencies. The panel can recommend a release from programme if insufficient and sustained lack of progress is seen despite additional training time. Any achieved competencies are recorded and documented. The trainee must give up their training number and advice on career options should be made available.

37. Continued…... • 5.Incomplete evidence presented – additional training time may be required. The trainee supplies incomplete or insufficient information such that the panel cannot make a statement about progress. • 6.Completion of training – all necessary competencies have been achieved

38. Comments on ARCP • Just started • Systematic and uniform for all specialities • Moving to electronic portfolio

39. e-portfolio • Currently being piloted with a group of trainees & trainers • Planned to be up and running for all in January 2009 A • Allows every functionality encompassed by paper based portfolio • Gives interaction between educational supervisor and trainee • Evidence can be stored either confidentially or shared • Able to support any file format (pdf excel word ppt music files etc) • MSF incorporated

40. Front page for a trainee

41. Front page as a trainer

42. View portfolio

43. Assessments

44. Diary

45. Activities e.g. presentations

46. Add evidence

47. Benefits e-portfolio • Hold record of trainee work, trainers’ assessments • Avoid time wasted on formatting • Avoid time wasted on re-typing • Allow trainers and trainees to keep track of their communications • Summarise progress (for trainees, trainer and TPD) • Summarise what still needs to be done

48. Assessing competence in HK training – the role of exams • Part A/Part 1 • Part B (OSPHE)/Part 2 • ?Portfolio assessment

49. Possible scenarios • Do nothing • Replace Part 2 General Viva with OSPHE and keep thesis examination • Replace Part 2 General Viva with OSPHE and replace thesis examination with formal and structured portfolio examination

50. Replacing general viva with OSPHE • OSPHE structure validated, tried and tested • Scenarios will need to be HK relevant • Examiner training