“Lies, damned lies and ...” a guide to dodgy data Neil Sheldon Royal Statistical Society

1. “Lies, damned lies and ...” a guide to dodgy data Neil Sheldon Royal Statistical Society Centre for Statistical Education Manchester Grammar School

3. The recommended daily calorie intake varies depending on how old you are. For the average adult this is about 2,000 per day (women) and 2,500 per day (men). NHS website

4. The recommended daily calorie intake varies depending on how old you are. For the average adult this is about 2,000 per day (women) and 2,500 per day (men). NHS website • We are told that recommended daily intake varies with age – but we are not told how it varies so that information is of no use.

5. The recommended daily calorie intake varies depending on how old you are. For the average adult this is about 2,000 per day (women) and 2,500 per day (men). NHS website • We are told that recommended daily intake varies with age – but we are not told how it varies so that information is of no use. • Neither are we told how much recommended calorie intake varies with factors such as height, body weight, metabolic rate, lifestyle, or levels of physical activity – all of which will affect energy needs and so the number of calories required.

6. The recommended daily calorie intake varies depending on how old you are. For the average adult this is about 2,000 per day (women) and 2,500 per day (men). NHS website • We are told that recommended daily intake varies with age – but we are not told how it varies so that information is of no use. • Neither are we told how much recommended calorie intake varies with factors such as height, body weight, metabolic rate, lifestyle, or levels of physical activity – all of which will affect energy needs and so the number of calories required. • A better way to present the figures would be as a range. For example, "many women need between 1850 and 2150 calories per day, though for some the figure may be as low as 1550 or as high as 2450".

7. The recommended daily calorie intake varies depending on how old you are. For the average adult this is about 2,000 per day (women) and 2,500 per day (men). NHS website • We are told that recommended daily intake varies with age – but we are not told how it varies so that information is of no use. • Neither are we told how much recommended calorie intake varies with factors such as height, body weight, metabolic rate, lifestyle, or levels of physical activity – all of which will affect energy needs and so the number of calories required. • A better way to present the figures would be as a range. For example, "many women need between 1850 and 2150 calories per day, though for some the figure may be as low as 1550 or as high as 2450". • Statistics are often presented in the form of averages, but it is important to think about the variation from the average too

8. Rubbish dump UK: we drop 30 million tonnes of litter a year in public. Reclaim Our Streets websiteOn average, each person in the UK throws away about 500 kg in rubbish every year. Wasteonline website

9. Rubbish dump UK: we drop 30 million tonnes of litter a year in public. Reclaim Our Streets websiteOn average, each person in the UK throws away about 500 kg in rubbish every year. Wasteonline website • In a population of about 60 million people, 30 million tonnes is about 500 kg per person.

10. Rubbish dump UK: we drop 30 million tonnes of litter a year in public. Reclaim Our Streets websiteOn average, each person in the UK throws away about 500 kg in rubbish every year. Wasteonline website • In a population of about 60 million people, 30 million tonnes is about 500 kg per person. • But one of these claims is about litter and the other is about rubbish – so they can’t both be correct.

11. Rubbish dump UK: we drop 30 million tonnes of litter a year in public. Reclaim Our Streets websiteOn average, each person in the UK throws away about 500 kg in rubbish every year. Wasteonline website • In a population of about 60 million people, 30 million tonnes is about 500 kg per person. • But one of these claims is about litter and the other is about rubbish – so they can’t both be correct. • 500 kg per year is more than a kilogram per day. Do you drop that much litter? Do you know anyone who does?

12. Rubbish dump UK: we drop 30 million tonnes of litter a year in public. Reclaim Our Streets websiteOn average, each person in the UK throws away about 500 kg in rubbish every year. Wasteonline website • In a population of about 60 million people, 30 million tonnes is about 500 kg per person. • But one of these claims is about litter and the other is about rubbish – so they can’t both be correct. • 500 kg per year is more than a kilogram per day. Do you drop that much litter? Do you know anyone who does? • It seems likely that the figure quoted for litter is really the figure for rubbish.

13. Rubbish dump UK: we drop 30 million tonnes of litter a year in public. Reclaim Our Streets websiteOn average, each person in the UK throws away about 500 kg in rubbish every year. Wasteonline website • In a population of about 60 million people, 30 million tonnes is about 500 kg per person. • But one of these claims is about litter and the other is about rubbish – so they can’t both be correct. • 500 kg per year is more than a kilogram per day. Do you drop that much litter? Do you know anyone who does? • It seems likely that the figure quoted for litter is really the figure for rubbish. • It is important to say exactly what you mean in statistics • Often you can check a statistical claim against other things you know

14. People who regularly put in overtime and work 10 or 11 hour days increase their heart disease risk by nearly two-thirds, research suggests.BBC website

15. People who regularly put in overtime and work 10 or 11 hour days increase their heart disease risk by nearly two-thirds, research suggests.BBC website • There is a link – a correlation – between overtime and heart disease, but does overtime cause the increased risk?

16. People who regularly put in overtime and work 10 or 11 hour days increase their heart disease risk by nearly two-thirds, research suggests.BBC website • There is a link – a correlation – between overtime and heart disease, but does overtime cause the increased risk? • The story identifies possible causes: People who spend more time at work have less time to exercise, relax and unwind. They may also be more stressed, anxious, or have depression.

17. People who regularly put in overtime and work 10 or 11 hour days increase their heart disease risk by nearly two-thirds, research suggests.BBC website • There is a link – a correlation – between overtime and heart disease, but does overtime cause the increased risk? • The story identifies possible causes: People who spend more time at work have less time to exercise, relax and unwind. They may also be more stressed, anxious, or have depression. • But it also sounds a note of caution, quoting the lead researcher as saying: more research is needed before we can be confident that overtime work would cause coronary heart disease.

18. People who regularly put in overtime and work 10 or 11 hour days increase their heart disease risk by nearly two-thirds, research suggests.BBC website • There is a link – a correlation – between overtime and heart disease, but does overtime cause the increased risk? • The story identifies possible causes: People who spend more time at work have less time to exercise, relax and unwind. They may also be more stressed, anxious, or have depression. • But it also sounds a note of caution, quoting the lead researcher as saying: more research is needed before we can be confident that overtime work would cause coronary heart disease. • A correlation between two things does not always imply that one causes the other • Good statistics will always be cautious about drawing conclusions

19. Pay for those at the top is increasing 17 times faster than average pay.For every £100 earned by a top company director in 2000 they earn £205 in 2006, while ordinary employees have had a £6 increase in every £100 they earned six years ago after allowing for inflation. TUC 2007

20. Pay for those at the top is increasing 17 times faster than average pay.For every £100 earned by a top company director in 2000 they earn £205 in 2006, while ordinary employees have had a £6 increase in every £100 they earned six years ago after allowing for inflation. TUC 2007 • An increase from £100 to £205 is, indeed, about 17 times as big as an increase from £100 to £106.

21. Pay for those at the top is increasing 17 times faster than average pay.For every £100 earned by a top company director in 2000 they earn £205 in 2006, while ordinary employees have had a £6 increase in every £100 they earned six years ago after allowing for inflation. TUC 2007 • An increase from £100 to £205 is, indeed, about 17 times as big as an increase from £100 to £106. • But we don’t normally measure pay increases ‘after allowing for inflation’.

22. Pay for those at the top is increasing 17 times faster than average pay.For every £100 earned by a top company director in 2000 they earn £205 in 2006, while ordinary employees have had a £6 increase in every £100 they earned six years ago after allowing for inflation. TUC 2007 • An increase from £100 to £205 is, indeed, about 17 times as big as an increase from £100 to £106. • But we don’t normally measure pay increases ‘after allowing for inflation’. • And we normally compare annual pay increases, not pay increases across six years.

23. Pay for those at the top is increasing 17 times faster than average pay.For every £100 earned by a top company director in 2000 they earn £205 in 2006, while ordinary employees have had a £6 increase in every £100 they earned six years ago after allowing for inflation. TUC 2007 • An increase from £100 to £205 is, indeed, about 17 times as big as an increase from £100 to £106. • But we don’t normally measure pay increases ‘after allowing for inflation’. • And we normally compare annual pay increases, not pay increases across six years. • So why have the TUC presented the statistics in this way?

24. Pay for those at the top is increasing 17 times faster than average pay.For every £100 earned by a top company director in 2000 they earn £205 in 2006, while ordinary employees have had a £6 increase in every £100 they earned six years ago after allowing for inflation. TUC 2007 • An increase from £100 to £205 is, indeed, about 17 times as big as an increase from £100 to £106. • But we don’t normally measure pay increases ‘after allowing for inflation’. • And we normally compare annual pay increases, not pay increases across six years. • So why have the TUC presented the statistics in this way? • Doing the calculations the usual way, pay for those at the top is increasing about 4 times (not 17 times) faster than average pay.

25. Pay for those at the top is increasing 17 times faster than average pay.For every £100 earned by a top company director in 2000 they earn £205 in 2006, while ordinary employees have had a £6 increase in every £100 they earned six years ago after allowing for inflation. TUC 2007 • An increase from £100 to £205 is, indeed, about 17 times as big as an increase from £100 to £106. • But we don’t normally measure pay increases ‘after allowing for inflation’. • And we normally compare annual pay increases, not pay increases across six years. • So why have the TUC presented the statistics in this way? • Doing the calculations the usual way, pay for those at the top is increasing about 4 times (not 17 times) faster than average pay. • If statistics have been calculated in a non-standard way, ask why • Good statistics are independent of vested interests

26. In a poll of 3,000 people, researchers found that the average British man tells three lies every day; that's equivalent to 1,092 a year. However the average woman appears more honest, lying 728 times a year – around twice a day. Science Museum

27. In a poll of 3,000 people, researchers found that the average British man tells three lies every day; that's equivalent to 1,092 a year. However the average woman appears more honest, lying 728 times a year – around twice a day. Science Museum • The figures were arrived at by asking people how often they told lies ... But did they answer the question truthfully?

28. In a poll of 3,000 people, researchers found that the average British man tells three lies every day; that's equivalent to 1,092 a year. However the average woman appears more honest, lying 728 times a year – around twice a day. Science Museum • The figures were arrived at by asking people how often they told lies ... But did they answer the question truthfully? • These figures may be intended as a bit of fun, but there is a serious point here. Some people may not want to admit to lying; other may wish to exaggerate how much they lie; most people probably have very little idea how many lies they tell.

29. In a poll of 3,000 people, researchers found that the average British man tells three lies every day; that's equivalent to 1,092 a year. However the average woman appears more honest, lying 728 times a year – around twice a day. Science Museum • The figures were arrived at by asking people how often they told lies ... But did they answer the question truthfully? • These figures may be intended as a bit of fun, but there is a serious point here. Some people may not want to admit to lying; other may wish to exaggerate how much they lie; most people probably have very little idea how many lies they tell. • The data are presented in a way that suggests far more accuracy than is realistic: e.g. ‘lying 728 times a year’.

30. In a poll of 3,000 people, researchers found that the average British man tells three lies every day; that's equivalent to 1,092 a year. However the average woman appears more honest, lying 728 times a year – around twice a day. Science Museum • The figures were arrived at by asking people how often they told lies ... But did they answer the question truthfully? • These figures may be intended as a bit of fun, but there is a serious point here. Some people may not want to admit to lying; other may wish to exaggerate how much they lie; most people probably have very little idea how many lies they tell. • The data are presented in a way that suggests far more accuracy than is realistic: e.g. ‘lying 728 times a year’. • It would be better to say: on average, men admit to lying about three times a day, and women admit to lying about twice a day.

31. In a poll of 3,000 people, researchers found that the average British man tells three lies every day; that's equivalent to 1,092 a year. However the average woman appears more honest, lying 728 times a year – around twice a day. Science Museum • The figures were arrived at by asking people how often they told lies ... But did they answer the question truthfully? • These figures may be intended as a bit of fun, but there is a serious point here. Some people may not want to admit to lying; other may wish to exaggerate how much they lie; most people probably have very little idea how many lies they tell. • The data are presented in a way that suggests far more accuracy than is realistic: e.g. ‘lying 728 times a year’. • It would be better to say: on average, men admit to lying about three times a day, and women admit to lying about twice a day. • People are not always reliable (or honest) when talking about themselves • Good statistics take into account uncertainty by avoiding being too precise

32. An inquiry into the accuracy of death certificates looked at cases of cancer of the oesophagus. In a group of 96 patients, doctors diagnosed 74 cases of the cancer; post mortem examinations found 73 cases.British Medical Journal

33. An inquiry into the accuracy of death certificates looked at cases of cancer of the oesophagus. In a group of 96 patients, doctors diagnosed 74 cases of the cancer; post mortem examinations found 73 cases.British Medical Journal • These figures seem reassuring: post mortems seem to confirm the diagnoses.

34. An inquiry into the accuracy of death certificates looked at cases of cancer of the oesophagus. In a group of 96 patients, doctors diagnosed 74 cases of the cancer; post mortem examinations found 73 cases.British Medical Journal • These figures seem reassuring: post mortems seem to confirm the diagnoses. • However, a breakdown of the figures tells a different story.

35. An inquiry into the accuracy of death certificates looked at cases of cancer of the oesophagus. In a group of 96 patients, doctors diagnosed 74 cases of the cancer; post mortem examinations found 73 cases.British Medical Journal • These figures seem reassuring: post mortems seem to confirm the diagnoses. • However, a breakdown of the figures tells a different story. • Post mortems found 22 cases of cancer of the oesophagus that doctors had not diagnosed, and 21 cases that were not cancer of the oesophagus and had been wrongly diagnosed.

36. An inquiry into the accuracy of death certificates looked at cases of cancer of the oesophagus. In a group of 96 patients, doctors diagnosed 74 cases of the cancer; post mortem examinations found 73 cases.British Medical Journal • These figures seem reassuring: post mortems seem to confirm the diagnoses. • However, a breakdown of the figures tells a different story. • Post mortems found 22 cases of cancer of the oesophagus that doctors had not diagnosed, and 21 cases that were not cancer of the oesophagus and had been wrongly diagnosed. • So the errors in diagnosis (called false negatives and false positives) almost exactly balanced out.

37. An inquiry into the accuracy of death certificates looked at cases of cancer of the oesophagus. In a group of 96 patients, doctors diagnosed 74 cases of the cancer; post mortem examinations found 73 cases.British Medical Journal • These figures seem reassuring: post mortems seem to confirm the diagnoses. • However, a breakdown of the figures tells a different story. • Post mortems found 22 cases of cancer of the oesophagus that doctors had not diagnosed, and 21 cases that were not cancer of the oesophagus and had been wrongly diagnosed. • So the errors in diagnosis (called false negatives and false positives) almost exactly balanced out. • Beware of ‘headline figures’ – often there is hidden information in the detail • What is true of the total is not always true of the parts

38. Asked to choose from a list of common worries, 30 per cent of parents stated abduction or murder, with only 5 per cent fearing for their child's health in later life due to current physical activity levels.From a survey of 1000 parents

39. Asked to choose from a list of common worries, 30 per cent of parents stated abduction or murder, with only 5 per cent fearing for their child's health in later life due to current physical activity levels.From a survey of 1000 parents • Home Office statistics show that there is about a one in a million chance of a child being killed by a stranger.

40. Asked to choose from a list of common worries, 30 per cent of parents stated abduction or murder, with only 5 per cent fearing for their child's health in later life due to current physical activity levels.From a survey of 1000 parents • Home Office statistics show that there is about a one in a million chance of a child being killed by a stranger. • NHS figures show that almost one in three children is overweight or obese by the age of 11.

41. Asked to choose from a list of common worries, 30 per cent of parents stated abduction or murder, with only 5 per cent fearing for their child's health in later life due to current physical activity levels.From a survey of 1000 parents • Home Office statistics show that there is about a one in a million chance of a child being killed by a stranger. • NHS figures show that almost one in three children is overweight or obese by the age of 11. • Childhood obesity increases the risk of severe health issues and early death.

42. Asked to choose from a list of common worries, 30 per cent of parents stated abduction or murder, with only 5 per cent fearing for their child's health in later life due to current physical activity levels.From a survey of 1000 parents • Home Office statistics show that there is about a one in a million chance of a child being killed by a stranger. • NHS figures show that almost one in three children is overweight or obese by the age of 11. • Childhood obesity increases the risk of severe health issues and early death. • This information was published as part of a campaign to get children walking to school so as to become fitter.

43. Asked to choose from a list of common worries, 30 per cent of parents stated abduction or murder, with only 5 per cent fearing for their child's health in later life due to current physical activity levels.From a survey of 1000 parents • Home Office statistics show that there is about a one in a million chance of a child being killed by a stranger. • NHS figures show that almost one in three children is overweight or obese by the age of 11. • Childhood obesity increases the risk of severe health issues and early death. • This information was published as part of a campaign to get children walking to school so as to become fitter. • Many people are not good at assessing risk – and they worry about the wrong things • Good statistics can help to evaluate risk and promote better decision making

44. “On many occasions I have praised students for an excellent performance, and in general when they try again, they do worse. On the other hand, I have often screamed at students for doing very badly, and in general they do better the next time.” – a teacher

45. “On many occasions I have praised students for an excellent performance, and in general when they try again, they do worse. On the other hand, I have often screamed at students for doing very badly, and in general they do better the next time.” – a teacher • The teacher went on to argue that this shows that praise and reward don’t work, but criticism and punishment do.

46. “On many occasions I have praised students for an excellent performance, and in general when they try again, they do worse. On the other hand, I have often screamed at students for doing very badly, and in general they do better the next time.” – a teacher • The teacher went on to argue that this shows that praise and reward don’t work, but criticism and punishment do. • But if you have just done something unusually well, chances are that next time it won’t be as good.

47. “On many occasions I have praised students for an excellent performance, and in general when they try again, they do worse. On the other hand, I have often screamed at students for doing very badly, and in general they do better the next time.” – a teacher • The teacher went on to argue that this shows that praise and reward don’t work, but criticism and punishment do. • But if you have just done something unusually well, chances are that next time it won’t be as good. • And if you have just done something unusually badly, chances are that next time you will do better.

48. “On many occasions I have praised students for an excellent performance, and in general when they try again, they do worse. On the other hand, I have often screamed at students for doing very badly, and in general they do better the next time.” – a teacher • The teacher went on to argue that this shows that praise and reward don’t work, but criticism and punishment do. • But if you have just done something unusually well, chances are that next time it won’t be as good. • And if you have just done something unusually badly, chances are that next time you will do better. • The praise and the criticism do not cause the changes.

49. “On many occasions I have praised students for an excellent performance, and in general when they try again, they do worse. On the other hand, I have often screamed at students for doing very badly, and in general they do better the next time.” – a teacher • The teacher went on to argue that this shows that praise and reward don’t work, but criticism and punishment do. • But if you have just done something unusually well, chances are that next time it won’t be as good. • And if you have just done something unusually badly, chances are that next time you will do better. • The praise and the criticism do not cause the changes. • The changes happen because, in the long run, most things tend towards the average.

50. “On many occasions I have praised students for an excellent performance, and in general when they try again, they do worse. On the other hand, I have often screamed at students for doing very badly, and in general they do better the next time.” – a teacher • The teacher went on to argue that this shows that praise and reward don’t work, but criticism and punishment do. • But if you have just done something unusually well, chances are that next time it won’t be as good. • And if you have just done something unusually badly, chances are that next time you will do better. • The praise and the criticism do not cause the changes. • The changes happen because, in the long run, most things tend towards the average. • Good runs and bad runs generally come to an end – this is called ‘regression to the mean’ • This happens naturally, and it can be very misleading to identify a cause