Competências Básicas de Investigação Científica e de Publicação

1. CompetênciasBásicas de InvestigaçãoCientífica e de Publicação Lecture 2: Hypotheses, reproducibility and experimental design Ganesha Associates CC BY 3.0

2. The scientific process involves making models of how things work • These evolving models are described in the scientific literature • Sometimes the models are wrong, often they are incomplete • Scientific progress is driven by the communication and publication of the results of new research, and the reinterpretation of older work • The tool which makes all of this possible is the hypothesis Ganesha Associates CC BY 3.0

3. The scientific process involves making models of how things work Ganesha Associates CC BY 3.0

4. Ganesha Associates CC BY 3.0

5. Ganesha Associates CC BY 3.0

6. Ganesha Associates CC BY 3.0

7. A visual abstract from Cell Ganesha Associates CC BY 3.0

8. Experimental vs. Observational studies No modification of experimental variables Useful to discover trends and associations Cannot directly be used to infer causality Compare responses different treatments Designed to avoid misleading results e.g. randomisation Can be used to infer cause and effect Ganesha Associates CC BY 3.0

9. Mainlearning points • Studentprojectsfallintothreecategories • No hypothesis, i.e. observational • Weakhypothesis • Strong hypothesis • The workwillbepublished in a • Nationaljournal • Lowimpactfactorjournal • High impactfactorjournal • Startingwithstronghypothesis improves your chances ofgettingpublished in a goodjournal Ganesha Associates CC BY 3.0

10. Ganesha Associates CC BY 3.0

11. What is a strong hypothesis ? • A strong hypothesis is based on a series of premises – things that are already known with some certainty • Each premise must be supported by references back to the (international) primary literature • So a strong hypothesis will be backed by references to recent papers in high quality journals Ganesha Associates CC BY 3.0

12. Ganesha Associates CC BY 3.0

13. Why did the Titanic sink? It was Captain Smith's fault This was Captain E. J. Smith's retirement trip. All he had to do was get to New York in record time. Captain E. J. Smith said years before the Titanic's voyage, "I cannot imagine any condition which would cause a ship to founder. Modern shipbuilding has gone beyond that." Captain Smith ignored seven iceberg warnings from his crew and other ships. If he had called for the ship to slow down then maybe the Titanic disaster would not have happened. It was the shipbuilder's fault About three million rivets were used to hold the sections of the Titanic together. Some rivets have been recovered from the wreck and analysed. The findings show that they were made of sub-standard iron. When the ship hit the iceberg, the force of the impact caused the heads of the rivets to break and the sections of the Titanic to come apart. If good quality iron rivets had been used the sections may have stayed together and the ship may not have sunk. It was Bruce Ismay's fault Bruce Ismay was the Managing Director of the White Star Line and he was aboard the Titanic. Competition for Atlantic passengers was fierce and the White Star Line wanted to show that they could make a six-day crossing. To meet this schedule the Titanic could not afford to slow down. It is believed that Ismay put pressure on Captain Smith to maintain the speed of the ship. It was Thomas Andrews' fault The belief that the ship was unsinkable was, in part, due to the fact that the Titanic had sixteen watertight compartments. However, the compartments did not reach as high as they should have done. The White Star Line did not want them to go all the way up because this would have reduced living space in first class. If Mr Andrews, the ship's architect, had insisted on making them the correct height then maybe the Titanic would not have sunk. It was Captain Lord's Fault The final iceberg warning sent to Titanic was from the Californian. Captained by Stanley Lord, she had stopped for the night about 19 miles north of Titanic. At around 11.15, Californian's radio operator turned off the radio and went to bed. Sometime after midnight the crew on watch reported seeing rockets being fired into the sky from a big liner. Captain Lord was informed but it was concluded that the ship was having a party. No action was taken by the Californian. If the Californian had turned on the radio she would have heard the distress messages from Titanic and would have been able to reach the ship in time to save all passengers. Ganesha Associates CC BY 3.0

14. Coin-tossing - an example • IwonderhowmanyheadsortailsIwillgetifItossthiscoin 100 times • No model • The frequencydistributionofheadsandtailswillbeapproximatedby a binomial distributionwithn=100 andp=0.5 • Simplemodel, basedonsymmetry • A detailedanalysisofthe dynamics revealsthattheprobabilityof a headis 0.51 • Complexmodel, basedonasymmetry, aerodynamics, etc Ganesha Associates CC BY 3.0

15. Coin-tossing – impact on CV 1. None, or possibly negative 2. R. A. Fisher and others did perform this experiment in the early days of biological statistics, before the advent of computers, as a proof that the binomial distribution tended towards a normal one at high levels of n. Interestingly they all found that the probability of a head p was usually slightly higher than 0.5, but this difference was ignored. 3. PersiDiacusis, Susan Holmes and Richard Montgomery (Stanford, 2004) publish a paper on the ‘Dynamical bias in the coin toss’ proving that the lack of total symmetry in a coin means that the probability of a head will always be slightly greater than 0.5. Ganesha Associates CC BY 3.0

16. Coin tossing - relevance • Ithinkthattherewillbeanassociation (+ or -) betweenmutations in gene xandsusceptibilitytodiseasey • No causal basis for arelationshipgiven • Ipredictthatmutations in gene xwillincreasesusceptibilitytodiseaseybecausepatientswithdiseaseyoftenhavelowlevelsof gene productx. • Built-in control, patientswith normal levelsofthe gene productshouldnothavethedisease. • Ipredictthatchemically non-neutral mutations in gene xwillincreasesusceptibilitytodiseasey in patientswithlowlevelsof gene productx. • Secondlevelofcontrol – neutral mutationsshouldbeasymptomatic Ganesha Associates CC BY 3.0

17. Coin-tossing – moral ofthestory • With a stronghypothesis, you: • Avoidfollowing leads which go nowhere – false positives, failearly • Avoidignoringunexpectedobservationsthat are of high interest – false negatives • May needto do lesswork ! • Will getpublished in betterjournals ! Ganesha Associates CC BY 3.0

18. Testing a hypothesis - 1 • The first stage of any scientific analysis is to define the null hypothesis that is to be tested. • This makes the prediction that our chosen variable has no impact on the outcome we are measuring. • As an example, the null hypothesis could be that “there is no difference in rainfall levels between urban and adjacent rural areas”. • The null hypothesis is denoted symbolically as H0 Ganesha Associates CC BY 3.0

19. Testing a hypothesis - 2 • It is also necessary to state the alternative hypothesis (H1) – that there is an increase in precipitation levels in urban areas relative to adjacent rural areas because of the heating differences of the two surface types (the urban area heats up more and has increased convective uplift). • Notice, it is a lot easier to specify the H0 version.  H1 requires an understanding of convection, thermodynamics and so forth.  Ganesha Associates CC BY 3.0

20. Testing a hypothesis - 3 • The H0 hypothesis protects us from trying to explain results that have no significance. • The H1 hypothesis challenges us to make sure we have thought of all of the other factors that could cause the effects we observe and excluded them by appropriate experimental controls or via additional experiments.   Ganesha Associates CC BY 3.0

21. Testing a hypothesis - 4 • You can often spot problems with project design by analyzing the Introduction to a research proposal. • If you edit out repetitive statements and non-essential background information, you should be left with a logical sequence of statements which leads to a clear H1 hypothesis.   • Unlike H0 hypotheses – there is no treatment effect – you have to have a mechanism in mind in order to formulate the H1 hypothesis. • Without this you have no logical structure to connect your results back to a known process or mechanism. Ganesha Associates CC BY 3.0

22. Testing a hypothesis - 5 • For example, this is the edited version of the introduction to a scientific paper.  Other than remove a couple of repetitious statements, I have made no changes. • Ectomycorrhizal fungi (EMF) have a fundamental role in nutrient absorption of many plant species. 2.  Tree species of ecological and economic relevance in reforestation programs depend on ectomycorrhizal symbiosis, especially in soils contaminated by mining activities. 3.  The ability of EMF to reduce the toxicity of heavy metal ions (e.g. copper) in their host plants is accompanied by the decrease of metal concentrations in the aerial part of the plant. 4.  Due to increased absorption of these metals by the EMF roots and the accumulation in the extra radical mycelium, greater tolerance to such elements is achieved by the host. 5.  Research has been conducted to determine the sensitivity of EMF to a variety of potentially toxic metals to understand the diverse mechanisms through which the fungi may tolerate heavy metals. 6.  Enzymatic activity is important for the mobilization and transference of soil nutrients through EM fungi towards the host plant. 7.  In this study we investigate the effects of copper and phosphorus concentrations on mycelial growth and enzymatic activities of the EM fungi Pisolithusmicrocarpus, Chondrogasterangustisporus and Suillus sp. in two growth experiments. Ganesha Associates CC BY 3.0

23. Testing a hypothesis - 6 • The logical discontinuity occurs between statements 6 and 7.  Since copper is toxic and phosphorous a fertilizer, we are almost certainly going to see effects.  But what do they mean ? • There is actually quite a large literature on copper toxicity in plants.  Several mechanisms have been identified and it would have been possible by choosing the experimental conditions carefully to distinguish which ones were operating under this particular set of experimental conditions. • Hence the introduction should have continued after point 6 to develop reasons for a more selective approach.  This would form the basis for the missing H1 hypothesis. Ganesha Associates CC BY 3.0

24. Case study: Hummingbird territorial behaviour Ganesha Associates CC BY 3.0

25. Hummingbird territorial behaviour Most hummingbird species demonstrate strong territorial behavior If a bluffing charge attack does not work, the resident may engage the trespasser in a brief but intense physical battle So why do hummingbirds defend territories ? H0: Hummingbirds are randomly distributed in space and time. Ganesha Associates CC BY 3.0

26. Hummingbird territorial behaviour H1 If territory = F(energy), then behavior not species-dependent If territory = F(mating), then behaviorshould be species and sex dependent If… If… Ganesha Associates CC BY 3.0

27. Territorial behaviour in 1971 • Time, Energy, and Territoriality of the Anna Hummingbird (Calypteanna) Science 173 (1971) 818-821. • When territory quality decreases defenders may switch to less expensive forms of defense because the energy savings outweigh the loss of resources • Augmented territorial defense during the breeding season is made possible by increased feeding efficiency due to the availability at this time of very nectar-rich flowers. • Individuals with large territories are more successful reproductively. Ganesha Associates CC BY 3.0

28. Hummingbird territoriality since • Digestive physiology is a determinant of foraging bout frequency in hummingbirds. Nature. 1986 Mar 6-12;320(6057):62-3. • Mitochondrial respiration in hummingbird flight muscles. ProcNatlAcadSci U S A. 1991 Jun 1;88(11):4870-3. • Cloning and analysis of the gene encoding hummingbird proinsulin. Gen Comp Endocrinol. 1993 Jul;91(1):25-30. • Flight and size constraints: hovering performance of large hummingbirds under maximal loading. J Exp Biol. 1997 Nov;200(Pt 21):2757-63. Ganesha Associates CC BY 3.0

29. Hummingbird territoriality since • Hovering performance of hummingbirds in hyperoxic gas mixtures. J Exp Biol. 2001 Jun;204(Pt 11):2021-7. • Adipose energy stores, physical work, and the metabolic syndrome: lessons from hummingbirds. Nutr J. 2005 Dec 13;4:36. • Neural specialization for hovering in hummingbirds: hypertrophy of the pretectal nucleus Lentiformismesencephali. J Comp Neurol. 2007 Jan 10;500(2):211-21. • Three-dimensional kinematics of hummingbird flight. J Exp Biol. 2007 Jul;210(Pt 13):2368-82. Ganesha Associates CC BY 3.0

30. Case Study One • Observations • There is evidence that the central nervous system can influence bone metabolism • Neurotransmitter (serotonin)receptors are found in bone • Drugs that inhibit neurotransmitter release can affect bone growth in humans • Hypothesis • If serotonin plays a role in regulating bone growth, then reducing serotonin release in bone will slow growth • Experiment • Inject mice with drugs (SSRI’s) that reduceserotonin releaseand measure impact on growth of femur over time • Result • Small but significant reduction in bone growth observed Ganesha Associates CC BY 3.0

31. Case Study One • Observations • There is evidence that the central nervous system can influence bone metabolism • Neurotransmitter (serotonin) receptors are found in bone • Drugs that inhibit neurotransmitter release can affect bone growth in humans • Hypothesis • If serotonin plays a role in regulating bone growth, then reducing serotonin release in bone will slow growth • Experiment • Inject mice with drugs (SSRI’s) that reduce serotonin release and measure impact on growth of femur over time • Result • Small but significant reduction in bone growth observed • Referees comments • Is this a direct or an indirect effect ? Ganesha Associates CC BY 3.0

32. Case Study Two • Observations • A plant extract (E) is used totreat diabetes • Prenatal malnutrition may cause diabetes • Hypothesis • If E plays a role in regulating blood sugar levels then the responses to E will be affected by dietary history • Experiment • Measure acute and chronic effects of E on glucose and insulin levels in normal and prenatal malnourished rats fed either on a standard or high glucose diet • Result • E does appear to reduce blood sugar levels but with no significant differences between the experimental groups of animals Ganesha Associates CC BY 3.0

33. Case Study Two • Observations • A plant extract (E) is used to treat diabetes • Prenatal malnutrition may cause diabetes • Hypothesis • If E plays a role in regulating blood sugar levels then the responses to E will be affected by dietary history • Experiment • Measure acute and chronic effects of E on glucose and insulin levels in normal and prenatal malnourished rats fed either on a standard or high glucose diet • Result • E does appear to reduce blood sugar levels but with no significant differences between the experimental groups of animals • Referees comments • Statistical complexity of experimental design prevents any clear conclusions being drawn about mechanism Ganesha Associates CC BY 3.0

34. Case Study Three • Observation • Fungal and other microbial symbionts can improve nutrient uptake by plant roots • Hypothesis • Soil microbial diversity is adversely affected by conventional farming techniques • Experiment • Compare microbial diversity associated with root soil in vines cultivated using either conventional or organic techniques • Result • Diversity and metabolic vigor higher in soils associated with the roots of organically cultivated plants Ganesha Associates CC BY 3.0

35. Case Study Three • Observation • Fungal and other microbial symbionts can improve nutrient uptake by plant roots • Hypothesis • Soil microbial diversity is adversely affected by conventional farming techniques • Experiment • Compare microbial diversity associated with root soil in vines cultivated using either conventional or organic techniques • Result • Diversity and metabolic vigor higher in soils associated with the roots of organically cultivated plants • Referees comments • What was the control for this experiment ? Maybe the differences in the two soil areas has nothing to do with agricultural methods. Ganesha Associates CC BY 3.0

36. Case Study Four • Observation • A patient suffers from an unusually aggressive form of retinoblastoma • Premise • Patients who develop retinoblastoma also have a particular type of genetic defect which can be identified cytogenetically • Hypothesis • This patient hasan unusual cytogenetic pattern which could be useful diagnostically • Result • The patient does indeed have a unique cytogenetic profile, but there are many differences from the pattern normally seen in retinoblastoma cases Ganesha Associates CC BY 3.0

37. Case Study Four • Observation • A patient suffers from an unusually aggressive form of retinoblastoma • Premise • Patients who develop retinoblastoma also have a particular type of genetic defect which can be identified cytogenetically • Hypothesis • This patient hasan unusual cytogenetic pattern which could be useful diagnostically • Result • The patient does indeed have a unique cytogenetic profile • Referees comments. • In this patient there are many differences from the normal pattern seen in retinoblastoma cases. How can we chose which differences are causal? Ganesha Associates CC BY 3.0

38. Case Study Five • Observation • Infection by certain human papilloma virus serotypes can lead to cervical cancer. One study has shown that the prevalence of HPV 31 and 33 is much higher in Recife than elsewhere in Brasil • Mutations to human P53 are often associated with the more severe consequences of HPV infection • Hypothesis • There is a strong correlation between patients with advanced HPV 31 and 33 disease and a new p53 mutation which is found mainly in the NE of Brasil • Result • Patients with HPV 31 and 33 infections have higher levels of cervical cancer and the new P53 mutation compared to other regions in Brazil. Ganesha Associates CC BY 3.0

39. Case Study Five • Observation • Infection by certain human papilloma virus serotypes can lead to cervical cancer. One study has shown that the prevalence of HPV 31 and 33 is much higher in Recife than elsewhere in Brasil • Mutations to human P53 are often associated with the more severe consequences of HPV infection • Hypothesis • There is a strong correlation between patients with advanced HPV 31 and 33 disease and a new p53 mutation which is found mainly in the NE of Brasil • Result • Patients with HPV 31 and 33 infections have higher levels of cervical cancer and the new P53 mutation compared to other regions in Brazil. • Referees comments • Isn’t this what we would expect from the existing epidemiological data? Ganesha Associates CC BY 3.0

40. Case study summary • These five hypotheses failed • Either because they were not based on a rigorously researched mechanism of action • Or because the experimental design was weak • As a consequence, important variables were not controlled for • And the results could not be explained within the context of the model • Moral: Hypotheses are often wrong first time around • Moral: Strong hypotheses usually require less work to prove than weak ones ! Ganesha Associates CC BY 3.0

41. Hypothesis lecture learning points • Good hypotheses build directly onto previous work • So they need to become technically more sophisticated over time moving from the general to the particular • A given problem can be associated with a number of very different hypotheses – your experiments should include tests to exclude these alternative explanations Ganesha Associates CC BY 3.0

42. Hypothesis lecture learning points • Hypotheses can be weak (observational) or strong (mechanism-based) • For example, a hypothesis which predicts that a tossed coin will end up ‘heads’ 50% of the time is much weaker than one that can predict the exact sequence of ‘heads’ and ‘tails’ • So hypothesis ‘quality’ is important Ganesha Associates CC BY 3.0

43. Break Ganesha Associates CC BY 3.0

44. Reproducibilty • No research paper can ever be considered to be the final word • Replication and corroboration of research results is key to the scientific process. • In studying complex entities, such as animals, ecosystems and so forth, the complexity of the system and of the techniques can all too easily lead to results that seem robust in the lab, and valid to editors and referees of journals, but which do not stand the test of reproducibilty. • Journals, research laboratories and institutions and funders all have an interest in tackling issues of irreproducibility. Ganesha Associates CC BY 3.0

45. “Why Most Published Research Findings are False” • Original paper by John Ioannidis published in PLoS Medicine in 2005 led to much debate. • Small sample sizes, weak hypotheses and experimental design, and bias mean that most positive findings are incorrect. • Commentary by Steven Goodman, ibid, 2007. • Agreement that many medical research findings are less definitive than readers suspect • P-values are widely misinterpreted. • Bias of various forms is widespread • Multiple approaches are needed to prevent the literature from being systematically biased • Need for more data on the prevalence of false claims. • Sources: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 and http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1855693/ Ganesha Associates CC BY 3.0

46. Case study: Amgen retrospective preclinical study • Fifty-three papers were identified as ‘landmark’ studies in preclinical cancer research. • However, scientific findings were confirmed in only 6 (11%) cases. • Source: www.nature.com/nature/journal/v483/n7391/full/483531a.html Ganesha Associates CC BY 3.0

47. Amgen retrospective preclinical study • The limitations of preclinical cancer models have been widely reviewed and are largely acknowledged by the field. • They include: • the use of small numbers of poorly characterized tumour cell lines that inadequately recapitulate human disease • an inability to capture the human tumour environment • a poor appreciation of pharmacokinetics and pharmacodynamics • the use of problematic endpoints and testing strategies • in addition, preclinical testing rarely includes predictive biomarkers that, when advanced to clinical trials, will help to distinguish those patients who are likely to benefit from a drug. • Learning point: Unfortunately, much published research ignores these problems • Read “Confiabilidadeemcrise” and “Nature Special: Challenges in irreproducible research”. Ganesha Associates CC BY 3.0