1 / 46

Quality of Information

Quality of Information. Research Methodology Colin Legg. SWINE FLU: 50,000 'DEAD IN MONTHS'. SWINE FLU: 50,000 'DEAD IN MONTHS '. NEARLY 50,000 people in Britain could die of swine flu before a vaccine becomes available, it emerged today.

ira
Download Presentation

Quality of Information

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Quality of Information Research Methodology Colin Legg

  2. SWINE FLU: 50,000 'DEAD IN MONTHS'

  3. SWINE FLU: 50,000 'DEAD IN MONTHS' • NEARLY 50,000 people in Britain could die of swine flu before a vaccine becomes available, it emerged today. • Dr Margaret Chan, head of the World Health Organisation, today rubbished claims by the Government that a reliable flu vaccine would be ready within weeks. • The Government predicts 100,000 new cases-a-day by the end of August. • Chief Medical Officer for England, Sir Liam Donaldson, refused to rule out a death rate of one in 200 - the figure calculated by experts at Imperial College, London yesterday July 15 2009

  4. Mass swine flu coffins warning • MINISTERS have been urged to have thousands of cardboard coffins ready for mass deaths should swine flu take a turn for the worse. • By David Maddox • news.Scotsman.com • Published Date: 17 September 2009

  5. “H1N1 is already killing Americans”

  6. Authority • Who is the author? • What is the source of the information?

  7. Context • A variant of H1N1, Spanish flu, killed 50 million to 100 million people between 1918 and 1919 • What is normal ‘Seasonal flu’ death rate? • “Disease control experts say the death rate from H1N1 is similar to the death rate from seasonal influenza, which kills anywhere between 250,000 and 500,000 people globally each year. “

  8. Case Fatality Ratio • Estimated in Mexico as 0.4% • Multiplier method • Community survey method • Method extrapolating from seasonal influenza mortality • Method extrapolating from a more ‘mature’ epidemic • The four methods produced a wide range of estimates … from 0.0004% to 0.06% Wilson & Baker 2009

  9. Context • H1N1 (swine flu): • has mild symptoms; • low lethality rate; • spreads easily between humans • H5N1 (bird flu): • hardly transmitable between humans; • the lethality rate is over 50%. • What will happen when someone with bird flu catches swine flu as well?

  10. Methodology • Based on real data or model predictions? • What are assumptions of models? • Well established protocols forEstimating infection ratesEstimating cause of deathData handling

  11. Precision • Scope of survey: • Representativeness • Geographic range • Sample size • What level of confidence in the estimates?

  12. Motivation • Are there any financial interests? • Are there any political interests?

  13. Information has huge powerNeed to get your data right Need to be critical of other’s data

  14. The flow of information • Real world • Observation • Notebook • Database • Statistical summary • Interpretation • Publication • Citation • Impact

  15. Observation • Errors of measurement • Noise (sampling error) • Bias

  16. Antarctic ozone hole 3/09/2000 Observation • Errors of measurement • Noise (sampling error)BiasUnrepresentative sample • Machine error • Elite data Use of ‘Indicators’

  17. Sunday Times 10/10/04 “MEN are far less capable than women of retaining self-control when they have been drinking, scientists have found. Their loss of inhibition is more than three times as great as that of women with the same concentration of alcohol in the blood.”

  18. Notebook - database • Copying errors • Selective reporting • Fraud • Fabrication • Falsification (including by omission) • Plagiarism

  19. Statistical summary • Simplification - loss of detail • Statement of precision or confidence • Statistical significance • Consider accuracy

  20. Scientific explanations: • Empirical • controlled conditions of observation • capable of verification by others • Rational • no known false assumptions • no logical errors

  21. Scientific explanations: • Testable • verifiable by direct observation • or make predictions which can be tested • Parsimonious • accept simplest explanation with fewest assumptions

  22. Explanations for the existence of ghosts • Membrane theory • The membrane between this world and the next occasionally wears thin • Stone tape theory • Building materials have a magnetic property that ‘records’ people and events • 3. Deranged attention-seekers theory

  23. Scientific explanations: • Parsimonious • accept simplest explanation with fewest assumptions • General • broad explanatory power in wide range of circumstances

  24. Scientific explanations: • Tentative • willing to accept alternative explanations when assumptions proved false • or more parsimonious explanations when found • Rigorously evaluated • substantial evidence, frequently re-evaluated

  25. Extrapolation • Are your data representative of the population about which you wish to make inferences?

  26. Presentation • ‘How to lie with statistics’ • D Huff (1954, 1991) Penguin Books Ltd

  27. From Financial Times Web site

  28. Presentation • ‘How to lie with statistics’ • Present data to best illustrate your point • Remember others are doing the same!

  29. Publication • Typographic errors • Nature and BMJ: • 11.6% & 11.1% errors in stat tests • An error in 38% & 25% of papers • 12% of such errors change P by an order of magnitude • Garcia-Berthou & Alcaraz 2004 • BMC Medical Res Methodology

  30. Funding bias • Funding control • Fashion • Political correctness

  31. Funding bias • Funding control • Fashion • Political correctness • Economic / financial interests

  32. Publication bias Effect of nitrogen fertilizer on concentration of Carbon-based secondary compounds in trees Journal impact rating 0 Leimu & Koricheva 2004 Proc. Roy. Soc. Lond. B Effect size

  33. BBC Panorama 10/10/04 “I believe that it would be impossible to produce an unbiased report when the source of funding came from groups with clearly vested interests.” Professor Jim Mann Committee member

  34. Publication • Publish or Perish • Fabrication, Falsification and Plagiarism • “One in three scientists confesses to having sinned” • Of 3,247 respondents • 1.5% admitted falsification or plagiarism • 15% admitted changing design, methods or results in response to pressure from funding source • 12% overlooked other’s use of flawed data • Nature (2005) 43519: 178-719

  35. Citation process Original data • Kaibab deer Visitor’s estimates Permanent rangers

  36. Citation process Rasmussen 1941 • Kaibab deer Visitor’s estimates Permanent rangers

  37. Citation process Leopold 1943 • Kaibab deer Visitor’s estimates Permanent rangers

  38. Citation process Davis & Golley 1963 • Kaibab deer Visitor’s estimates Permanent rangers

  39. Original data Davis & Golley 1963

  40. Citation process • Kaibab deer • Abstracting journals

  41. Impact • Political decisions • Misuse of data • Social responsibility of science

  42. “The pill carries increased risk of blood clots” • Over 6 month period: • 7-8% rise in rate of conception • 10-13% rise in abortions • = 30,000 extra pregnancies • = 10,000 extra abortions

  43. Conclusion • Information is powerful and valuable,but fragile • Handle with care

More Related