Unit 4: Poor Experimental Design, Community of Scientists, Ethics, Critical Thinking

1. Unit 4: Poor Experimental Design, Community of Scientists, Ethics, Critical Thinking Core 270 Spring 2008 Dr. Sharon Fredericks

2. Experimental Design • Flaws in the design of an experiment will cause the results to be _______________ and lead to ________________conclusions. • i.e., not taking into account confounding variables, poor sample selection, inappropriate statistical analysis • The unethical scientist will cover up errors and continue to publish, causing mistrust in his work and in the field itself.

3. Sampling Bias • Differences between a sample and the population it represents should result only from random chance. • When selecting the group to be studied, it must be done with no bias. • ___________________________in selecting all subjects • __________________________________dividing the subjects into the control and experimental groups • Otherwise, one is favoring the outcome. Sampling bias can be minimized with a ______________ group. http://edweb.sdsu.edu/Courses/Ed690DR/Class04/samplingbiasbackground.html

4. Sources of sampling bias • Failure to adhere to the random sampling procedures. • Omission of specific subgroups of the population from the sampling frame and therefore from the sample. • Faulty measuring devices • this may be in terms of the specific questions used in a questionnaire • may also arise in a survey that involves taking physical measurements, when the measuring device is incorrect, e.g., using a tape measure that has been stretched, so that all measurements are too small • Violations of equal probability of selection principles because of duplicate listings in the sampling frame, or other causes. • Non-response to a survey by specific subgroups of the population that are relevant to the measures of concern in the survey. http://npts.ornl.gov/npts/1995/courseware/Useable_Nav3_19_27.html

5. Examples of Sampling Bias • Example given in text1: Diet pills were found to be more effective because the experimental group had a higher percentage of healthier, more active, and more motivated people. • A web-site poll or survey is inherently biased and inaccurate because • Only a ______________________________, motivated people, will log on • A person may be able to vote multiple times

6. Example in the News • The Union of Concerned Scientists conducted a scientific survey of more than 1,600 federal climate scientists. • This was presented as evidence that the Bush administration was engaged in “wide-ranging political interference in research related to global warming.” http://www.foxnews.com/printer_friendly_story/0,3566,251119,00.html

7. Poor Sample • Low response rate of 17 percent. • “This means we don't know the views and experiences of the other 83 percent...” • ______________________________of the group • “Many peer-reviewed academic journals will not accept papers relying on samples smaller than 50 percent.” • The survey also may be infected with a “selection bias”. • Scientists most upset about perceived interference filled out the questionnaire.

8. Faulty Measuring Device (survey) • Lumped into the same category scientists who said they actually experienced the alleged tampering and scientists who simply “perceived” that it happened to someone else. • 43% of the respondents reported they had perceived or personally experienced changes or edits during review of their work that changed the meaning of their scientific findings • In the study's appendix, and only 15 percent of the respondents said that they had actually experienced such interference. • Similarly, 43% perceived or experienced “fear of retaliation for openly expressing concerns about climate change outside my agency.” • The question didn't ask how many actually experienced retaliation, instead of just fearing it. Actually, only 14% personally harbored such a fear. The other 29% perceived it in others.

9. Faulty Measuring Device (survey) • “The researchers assume that all these responses refer to officials' efforts to alter certain kinds of findings about global warming. But that is __________________ in the questionnaire.” • For example, a scientist could find a change in working conditions without it having anything to do with global warming.

10. Misinformation reported by the Media • NBC's Andrea Mitchell reported that “nearly half [the scientists] were pressured to eliminate the words ‘climate change’ or ‘global warming’…” • The New York Times reported that 60% of the scientists “personally experienced” interference. • ABC's Jake Tapper said, “scientists say their work on global warming has been watered down and twisted by the White House...”

11. Lack of Knowledge Regarding Subjects • Determine if changes in the subject is due to the independent variable being tested or if it is a by product of the ____________________________ • Examples in text1: • Study on the physiological effect of LSD on an elephant. Researchers based the dosage on weight and not on metabolism, causing permanent damage to the animal and resulting in it being euthanized. • Wild prairie flower could only thrive on grazed land due to competition with grasses.

12. Another Example • Passive sample collectors such as Minnow traps and Breder traps are simple, inexpensive and easily replaceable. • They are used to collect fish for experiments. • Species specific characteristics of the fish such as behavior and size will affect what is caught in minnow traps. • So, the contents of a trap may not reflect the population of fish or a _________________sample. http://www.yale.edu/eeb/layman/pdf/Layman%20and%20Smith%202001.pdf#search='sampling%20bias'

13. Lack of Statistical Significance • The _______________and more ____________ the group, the more significant is the statistical analysis. • Larger groups are expensive, so usually small pilot studies are done first. However, in drug studies, not all negative side effects may be encountered. • Need to be sure that statistical, not anecdotal evidence, supports findings. • Anecdotal evidence is the experience of a single person or a few people.

14. Statistical Analysis • Scientists use more than one method and the most appropriate should be used, not the one that supports the hypothesis the best. • Need to be sure the statistical analysis is done correctly. • All analytical methods used should be discussed in the research paper.

15. Example • A study collected data from 51,603 women. • It reportedly showed that the 1,007 women who increased their consumption of regular soft drinks over a period of four years from less than one per week to one or more per day gained an average of 10.3 pounds. • Among the approximately 16,600 women who consumed more than one soft drink per day, the researchers reported 83 percent more cases of type 2 diabetes. http://www.foxnews.com/story/0,2933,130263,00.html

16. Example Continued • When the researchers ___________________ adjusted their results for bodyweight (a risk factor for diabetes) and for caloric intake (a proxy measure for consumption of sweetened foods other than soda), the 83 percent increase dropped to a 32 percent increase. • That result is of the same magnitude as the study’s reported 21 percent increase in diabetes among consumers of more than one diet soft drink per day. • Diet drinks do not contain any sugar at all.

17. Failure to Control Variables • Results of a study should be due to changes in the variable of interest, and not due to a confounding variable. • ________________________________is critical. • Example in text: Study of the effect of personal interaction, bright colors and mobiles on infant development was confounded by the night cleaning staff playing and interacting with both the experimental and control groups of infants.

18. Another Example • Early studies of the effects of alcohol on heart disease were _____________________________. • Alcohol actually prevents heart disease; while smoking causes heart disease. • Unfortunately for investigators, smoking and alcohol exposure are positively correlated (smokers drink more than non-smokers). • Early studies appeared to show that alcohol caused heart disease; when actually the small protective effect of alcohol was masked by the large causative effect of smoking. • Once smoking behavior was taken into account, the apparent negative effect (heart disease) of alcohol disappeared. http://hta.uvic.ca/3lecturesample.html#ethics

19. Final Example • In August of 2001, an article was published in the Journal of the American Medical Association (JAMA) stating that soy-based infant formula was not harmful and as good as cow milk-based formula • Critics say that the study: • Was poorly designed • Did not ask the right questions – failure to control variables • Did not use all data equally • Lacked statistical significance http://www.mercola.com/2001/sep/15/soy_formula.htm

20. Poor Design • Telephone interviews with 282 adults fed soy formula and 563 adults fed milk formula during controlled feeding studies at the University of Iowa between 1965-1978. • "Data derived from telephone interviews, particularly interviews that ask a lot of embarrassing questions, cannot be used to draw any meaningful conclusions," said Dr. Naomi Baumslag, Professor of Pediatrics at Georgetown University.

21. Failure to Control Variables & did not use data equally • The questions were geared to assess reproductive disorders and age of maturation. • The study provided no information on dose length or quantity, nor on the ages at which ingestion ended, all vital in a study on toxicity. • Dr. Mary Enig, President of the Maryland Nutritionists Association, stated, "The research team glossed over negative findings and ________ them from the Abstract and Conclusions..."

22. Lacked Statistical Significance & Financial Bias • Many of the negative findings for the soy-fed group were not "statistically significant." (according to the study) • But critics point out that the group of 282 soy-fed individuals was _________________________ for statistical significance to be achieved. • Dr. Samuel Fomon, the main researcher of the paper, played an important role in the development of soy infant formula.

23. Disregard for the Condition of Subjects • In this age, it is very difficult to do experiments on animals. It’s a long process to get permission. There are strict guidelines on care, treatment, etc. • Rules for housing • Veterinarian oversees care • Scientists must treat their subjects humanely in order to determine whether the results are due to the variable studied and not due to stress, poor health, and poor living conditions.

24. Should Some Experiments ever be done? • With the present economic environment, funding is tight. Research groups must give strong support for their studies. • In addition to the practical aspect, e.g., How will the results help mankind or produce a product?, there are the ethical questions: • Should we be trying to do this? E.g., cloning humans • Is the study not putting unnecessary harm to the subjects? E.g., clinical drug trials

25. Community of Scientists2,3 • Scientists rarely work alone. • Scientists, being human beings, are subject to biological and psychological factors.3 • Well-respected scientist can withhold invitations to present papers at conferences or reject papers in the peer review process • Balance between _____________ and sharing of knowledge and ____________ for fame and prizes. • Science is a “social institution”.2 Social aspects influence the actual practice of science. • E.g., allocation of resources is strongly influenced by politics3

26. Community of Scientists3 • Scientists may work in 3 areas • Scientists live within a society composed of many cultures and create their own cultures within their institutions. • Within these communities, ideas progress from initial proposals to paradigms and may undergo a scientific revolution.

27. Scientific Norms2 • Also called “Ethos of science” • By Merton and Ziman • Rules and foundations on how the community of scientists are expected to __________________. • CUDOS • Communism • Universalism • Disinterestedness • Originality • Skepticism

28. Communism2 • Also called scientific communalism • Scientific knowledge _______________________. • Scientists are obligated to publish their results and share with the community. • Reward is recognition and promotion. • Exceptions • Military research • Commercial research • Computer security research

29. Universalism and Disinterestedness2 • Work of scientist should be judged on competence and merit and not on any ____________________________________ • Nationality, race, gender, religion, political persuasion • Work should not be judged on the past accomplishments of the scientist • Disinterestedness is not the same as uninterestedness. • Scientists should conduct work without __________ and not for the purpose of gaining fame and fortune or to advance a political cause. • Scientists’ purpose should be to advance knowledge.

30. Originality & Skepticism2 • Work should be original: new research, new ideas, new insight on old problems, new techniques • Skepticism in the form of careful, __________________________________ of one’s work and the work of others is required.

31. Becoming a Scientist2 • Children are natural scientists because they are curious and want to understand what they observe. • If this interest is maintained, then they may decide to become a scientist. • Undergraduate degree, usually in a scientific field, where _______________________________________is learned. • Bachelor of Science • Graduate degree in a specialty of a natural science • Ph.D. (doctor of philosophy) earned • Post-doctoral position • Additional skills in a related field • Usually still under the supervision of a more experienced scientist • Job in of the 3 areas mentioned earlier.

32. Questionable Practices of Scientist2 • Honorary authorship • Inadequately supervising subordinates as Ph.D. advisor, post-doc advisor, department and laboratory heads • Interfering in another scientist’s research • Self-plagiarism • Misrepresentation of one’s own record • Conflict of interest

33. Reward System2 • Higher salary or company sponsored “chairs” • Profits from patents • Cash awards (e.g., Nobel prize) • _________________________________recognition • Eponymy (having something named after you – element, unit of measure, period of time/age, law) • Nomination into a society • Citation in other people’s papers • Being hired at a famous university or lab • Being published in a highly selective journal • Receiving grant money from a prestigious agency

34. Nobel Prizes1 • Alfred Nobel (born 1833) invented blasting powder and dynamite. Because of the military applications of dynamite, he became a millionaire • He left $9 million in a trust to establish the Nobel prizes. Winners’ works benefit mankind. • Established in 1900; first prizes given out in 1901. • 5 categories: • Announced in the fall every year by the Royal Swedish Academy of Sciences • 34 women: 11 in Literature, 12 for Peace, 7 in physiology/medicine, 3 for chemistry, 2 in physics with Marie Curie with 2 prizes (chemistry and physics) Sources: www.nobelprize.org and http://nobelprize.org/nobel_prizes/lists/women.html

35. Nobel Prize • Laureates: 777 individuals and 20 organizations • The youngest winner was Lawrence Bragg (25 years old, physics, 1915) and the oldest was Leonid Hurwicz (90 years old, economics, 2007). • Linus Pauling is the only person to receive 2 unshared Nobel prizes (chemistry, peace). 3 other individuals (J. Bardeen, physics; Marie Curie, physics & chemistry; F. Sanger, chemistry) and 2 organizations (Red Cross & UNHCR) have received multiple Nobel prizes. • 2 people have refused the Nobel prize • Jean-Paul Sartre (1964, Literature) had consistently declined all official honors. • Le Duc Tho (1973, Peace) jointly with US Secretary of State Henry Kissinger. He state that he was not in the position to accept due to the situation in Vietnam. http://nobelprize.org/nobel_prizes/nobelprize_facts.html

36. 2007 Nobel Prize Winners • Physics – Albert Fert (France) and Peter Grünberg (Germany) ‘for the discovery of Giant Magnetoresistance’. • Created technology to read data on hard disks of computers • Allowed for smaller hard disks by using nanotechnology and creating more sensitive read-out heads

37. 2007 Nobel Prize Winners • Chemistry– Gerhard Ertl (Germany) ‘for his studies of chemical processes on solid surfaces’ • Developed techniques for surface chemistry • Important for industrial processes • Production of artificial fertilizers • Cleaning up exhaust emissions in cars • Determine how fuel cells function • Use of catalysts on surfaces to speed up reactions

38. 2007 Nobel Prize Winners • Physiology/medicine – Mario R. Capecchi (U. of Utah), Sir Martin J. Evans and (United Kingdom), and Oliver Smithies (UNC, Chapel Hill) ‘for their discoveries of principles for introducing specific gene modifications in mice by the use of embryonic stem cells’ • Led to the method called gene targeting in mice which has been used in basic research and therapies • Basically, a single gene can be knocked out and made inactive to determine the role of that gene in health and disease • Use of homologous gene recombination to repair defective genes • Combination of the 2 techniques

39. 2007 Nobel Prize Winners • Literature – Doris Lessing (United Kingdom) ‘that epicist of the female experience, who with scepticism, fire and visionary power has subjected a divided civilisation to scrutiny’ • Peace – Intergovernmental Panel on Climate Change (IPCC) (Switzerland) and Al Gore (USA) ‘for their efforts to build up and disseminate greater knowledge about man-made climate change, and to lay the foundations for the measures that are needed to counteract such change’

40. The Bank of Sweden Prize in Economic Sciences in Memory of Alfred Nobel • Not originally in the Nobel’s will • Called by most the Nobel prize in Economics • 2007 Winner: Leonid Hurwicz (U. of MN), Eric S. Maskin (Inst. For Advanced Studies, Princeton), Roger B. Myerson (U. of Chicago) ‘for having laid the foundations of mechanism design theory’

41. Scientific Commandments2 • Be honest • Never manipulate data • Be precise • Be fair with regard to priority and ideas • Be without bias with regard to the data and ideas of your rivals • Do not make compromises in trying to solve a problem

42. A Matter of Trust • Most science progresses by building and expanding on previous work • Researchers must trust that the work is valid • If the work can not be independently reproduced, using a different technique if possible, doubt is cast on all that research group’s work. • May spread to the whole field, causing funding to dry up. • May tarnish image of all scientists

43. Misconduct in Science2 • Fabrication : • Falsification: _____________________ data or results, like deleting data that does not support hypothesis • Plagiarism: use of someone else’s words or ideas without giving proper credit; • Self-plagiarism: publishing the same work in more than one journal

44. Falsifying Documentation • This is best caught through __________________. • Scientific work is formally reviewed by other experts in the field before being published. • Same type of process is used for grant money applications • Not perfect because of competition • If a group does not want to openly share the evidence and results of a study, one should be suspicious. • Another check:

45. Related Questionable Practices • Failing to retain significant research data for a significant period of time • Maintaining inadequate research records • Refusing to give peers reasonable access to unique research materials or data that support papers

46. Reasons for Bias • _____________is usually the reason for bias. • Also fame, prestige, awards • A research group may want continued or more funding, so the results need to be favorable or progress needs to be demonstrated. • A company sponsors a study in order for the results to favor the sale of a product.

47. Research with Human and Animal Subjects • Limitations on what type of research is allowed • Projects must undergo scrutiny and pass review boards • Must weigh potential knowledge versus pain and suffering on the part of the subjects • Strict rules on the care of animal subjects to avoid ______________due to stress in living conditions • Examples of unethical studies: • Tuskegee syphilis study in 1932

48. Informed Consent • Most ________________________________ ethical aspect with human subjects • Subjects are adequately informed of • Aims, methods, anticipated benefits, and potential hazards of experiment • Option to not participate or stop participating at any time • Subjects should agree without coercion • Scientists need to document consent in writing

49. Informed Consent continued • Deception in consent is only allowed if • Risk to the subjects is minimal or non-existent • Rights & welfare of the subjects are not affected • It is necessary for the experiment to be conducted without ____________________ • True nature of the study is explained afterwards

50. Example #1 • In February 2000, a promising young physicist named Jan Hendrik Schön published some startling experimental results. • Schön and his partners had started with molecules that don't ordinarily conduct electricity, and claimed they had succeeded in making them behave like semiconductors, the circuits that make computers work. • In a field where publishing 2-3 articles a year makes you productive, Schön was the lead author on dozens of articles, most of them appearing in the industry-leading journals. http://www.salon.com/tech/feature/2002/09/16/physics/print.html