How to understand the National Training Survey

1. How to understand the National Training Survey

2. The main problems interpreting the GMC Survey apply to any large dataset Deciding what you want to find out is the first part of the battle! Select one area of interest and focus on it. Leave the other data to one side and concentrate on one theme at a time. Later, you may need to cross-compare to see broader themes

3. Decide what you wish to investigate. Decide what you want to discover. • Follow the path! If you’re interested in ophthalmologists at Sunderland Eye Infirmary, ignore paediatricians at James Cook! • Concentrating on one area at a time will allow you to see the key themes for that group. Once you have seen those themes you can look at other data to ascertain whether it’s a problem to just that group or more widespread. • Lies, damn lies! • The phrase “Lies, damn lies and statistics” isn’t to suggest that statisticians deliberately mislead, more that the person using the data must appreciate that what is shown (or not shown) can greatly affect the interpretation of the information. If an indicator score is not what you expected or hoped, check to see how many respondents were involved and how cohesive those responses are. • Read the instructions • The GMC have given detailed guidance on how to interpret the survey results: • http://gmc-onlineeducationreports.org/ReportsGuide.aspx

4. The box chart The box chart represents a number of statistical techniques graphically. It need not be scary. Just break them down into the pieces. Two types of average are show – Mean and Median Interquartile Ranges are represented by different shades of green Confidence Intervals (Error bars) show how cohesive the responses are

5. The box chart This score is 0-100 Unless specified, it’s not a percentage! This is the MEAN score for the group These are the “interquartile ranges” These are the ERROR BARS (confidence interval) This is the MEDIAN score for all trainees nationally

6. Mean and Median The mean is all the scores added up, then divided by the number of scores The median is the middle value when all scores are arranged in order 2, 2, 8, 2, 7, 9, 2, 6, 3, 8, 1 Mean = 2 + 2 + 8 + 2 + 7 + 9 + 2 + 6 + 3 + 8 + 1 = 50. 50 ÷ 11 = 4.5 Median - sort the numbers into ascending order 1, 2, 2, 2, 2, 3, 6, 7, 8, 8, 9

7. Interquartile ranges 1, 2, 2, 2, 2, 3, 6, 7, 8, 8, 9 Interquartile Range Top quartile Bottom quartile The quartile ranges are 25% (a quarter!) of the NUMBER of scores when arranged in ascending order. The MEDIAN score is precisely in the middle of the 2nd and 3rd quartiles.

8. Error Bars 1, 2, 2, 2, 2, 3, 2, 2, 1, 3, 79 The mean score for the above series is 9. Is that a true average? The confidence bars allow you to see if everyone feels the same If there are long error bars, there is a wide range of opinion If there are short (or no) error bars, it shows everyone thinks the same

9. Outliers A red or a green triangle to the right of the box chart denotes an outlier Essentially, an outlier is a MEAN score for the group which sits in either the top or the bottom quartiles and the error bars do not extend into the interquartile range. The easiest way to look at these is to regard green triangles as showing good practice and red triangles as showing areas of concern. It’s not fool proof – if there’s little confidence (long error bars) in the results, a very high or very low score may not be flagged. This could suggest that some very satisfied (or dissatisfied) doctors are being “lost” amongst the data.

10. Outliers A red or a green triangle to the right of the box chart denotes an outlier Essentially, an outlier is a MEAN score for the group which sits in either the top or the bottom quartiles and the error bars do not extend into the interquartile range. The easiest way to look at these is to regard green triangles as showing good practice and red triangles as showing areas of concern. It’s not fool proof – if there’s little confidence (long error bars) in the results, a very high or very low score may not be flagged. This could suggest that some very satisfied (or dissatisfied) doctors are being “lost” amongst the data.

11. Indicators Now we understand how to use a box chart, we can read the indicator scores The trainee survey uses 22 indicator scores Each indicator score is “fed” by the responses to a series of questions The “Overall Satisfaction” indicator (for example) is fed by the questions: How would you rate the quality of teaching [informal (and bedside teaching) as well as formal and organised sessions] in this post? How would you rate the quality of supervision in this post? How would you rate the quality of experience in this post? How would you describe this post to a friend who was thinking of applying for it?

12. Groups All the results for the indicators are collated nationally (hence the national average) Scores are generated for smaller groups: All trainees at a given deanery All trainees at a given hospital, trust or GP training scheme All trainees in a given specialty All trainees in a given specialty at a given hospital

13. Groups and Indicators are the criteria that should be used to “follow the path” It’s useful to compare different groups How does ACCS at the RVI compare to ACCS at UHND? How does UHND compare to the RVI? It’s useful to compare different indicators Overall satisfaction is low Workload scores badly

14. Following the path – tell a story When you follow the path, a story can emerge. What do you think these results tell you about the unit? Overall satisfaction Clinical supervision Work load Work intensity (appropriateness)

15. “Damn Lies” This is the caveat – although the results of the survey are statistically valid, that does not necessarily mean that they are important Always remember the error bars – they show how useful the data is Bear in mind that the smallest reporting unit for the GMC Survey is three. From a group of 100 trainees, if three are very unhappy it doesn’t look like a big problem. If it’s a group of five trainees, it looks like a huge problem. Do not forget that even from 100, three unhappy trainees is three people’s working lives, and the lives of the hundreds of patients they treat.