What counts as good evidence?

1. What counts as good evidence? Dylan Evans Department of Mechanical Engineering University of Bath

2. It’s hard to know what is bogus! • NASA invests $1m in antigravity machine • US Patent 6,362,718 issued for a motionless electromagnetic generator • Both of these machines are physically impossible • But scientists have advocated both!

3. Seven warning signs • Robert L Park is a Professor of Physics at the University of Maryland • He has identified seven warning signs that indicate that a scientific claim is bogus. • Remember, these are only warning signs – they are not infallible.

4. 1. Pitching to the media • The integrity of science rests on the willingness of scientists to expose new ideas and findings to the scrutiny of other scientists before anyone else. • This is called ‘peer review’. • Going to the media first suggests that the work is unlikely to stand up to peer review. Eg. Pons & Fleischmann, 1989.

5. 2. Conspiracy and cover-ups • The discoverer says that a powerful establishment is trying to suppress his or her work. • Mainstream science is made out to be a part of a larger conspiracy that includes industry and the government. • Examples: GM potatoes, cold fusion.

6. 3. At the limit of detection • The scientific effect is said to be very hard to detect. • All scientific measurements must contend with some level of background noise. • But if the signal-to-noise ratio cannot be improved, even in principle, the effect is probably not real. • Examples: parapsychology.

7. 4. Evidence is anecdotal • Anecdotes and narrative have a very strong emotional impact. • Statistics have a weak emotional impact. • Frauds tend to use anecdotes and not statistics such as RCTs. • Scientists have to learn to put statistics above anecdote. Eg. bloodletting.

8. 5. Ancient wisdom • The discoverer says a belief is credible because it has endured for centuries. • ‘Our ancestors possessed wonderful advanced knowledge that has been forgotten.’ Eg. crystal healing. • Science was born when people began to reject blind tradition in favour of experiment and observation (eg. Bacon)

9. 6. Splendid isolation • The image of the lone genius who struggles in secrecy in an attic laboratory is rarely accurate. • Scientific breakthroughs nowadays are almost always syntheses of the work of many scientists.

10. 7. New laws of nature • The discoverer proposes a new law of nature that conflicts with what is already well-established. • Weigh up the evidence for each alternative: eg. homeopathy and the meta-analysis in the BMJ.

11. On the other hand… • Sometimes a scientist MAY go to the media first! • Sometimes there ARE powerful forces trying to suppress a good idea! • Some scientific effects ARE very hard to detect!

12. On the other hand… (contd) • Anecdotal evidence is better than NO evidence at all! • Some ancient knowledge HAS been forgotten! • Some scientists DO work in isolation! • Sometimes a new law of nature IS correct!

13. So how can we tell? • No ‘recipe’ or ‘algorithm’ for assessing science. • No substitute for thinking! • But there are some general guidelines on how to be critical. Apply them to your own work and to that of others.

14. Whenever you read a paper…..(even your own!).. • What is the hypothesis being tested? • Read the methods section – is there enough information to allow the experiment or observation to be repeated? • Read the results section – is there a proper statistical analysis?