Week 1. Scientific method, research methodology and experimental design

1. Week 1. Scientific method, research methodology and experimental design MScMethodologySeminar I Dr. Felipe Orihuela-Espina

2. Contents • Common excuses toattemptnotabidingbythescientificmethod • Scientific method • Researchmethodology • Experimental design INAOE

3. COMMON excuses toattemptNOT abidINGbythescientificmethod INAOE

4. Excuse No. 1 • Thearchetypicalspeech: • “Theresearchmethodwasconceivedforothersciencese.g.physics. Computersciencecannotbeenforcedtofitintothescientificmethod.” • Thepointattempted: • CS isdifferent, and shallbetreateddifferently • Theantidote: • Be kindtoyourself and read “Theneedfor a hypothesis in informatics” by Prof. Alan Bundy. • http://www.inf.ed.ac.uk/teaching/courses/irm/notes/hypotheses.html • ALL sciencesmustabidebythescientificmethod; CS is no exception. • …particularities are dealtwith in theresearchmethodology. INAOE

5. Excuse No. 2 • Thearchetypicalspeech: • “My researchistheoretical, not experimental ergo thescientificmethod, whichputsexperimentation in itscore, doesnotapply in my case.” • Thepointattempted: • Theoryis so far superior toexperimentationthat I cannotbothertoexpress my research in thosemeaninglessterms • Theantidote: • Dropyourcynicism • Theoryisabsolutelynecessary and central tothescientificmethod • …in fact; itwas Galileo whofirstintroducedmathematicaldemonstrationtothescientificmethod. • …so if Galileo could, so couldyou… • Moreover; mindyoubutyou“will eventually have to show that these theoretical predictions are borne out in practice” [Bundy, “Theneedfor a hypothesis in informatics”] INAOE

6. Excuse No. 3 • Thearchetypicalspeech: • “My researchisveryspecific, thereissimply no wayitshallfitintothescientificmethod.” • Thepointattempted: • I’mtoobusy; I don’thave time forthisrubbish • Theantidote: • Stop victimizingyourself. • Allscientificworkisworth and has particularities… yoursis no exception. • …bytheway; Profis more busythanyou, and he isnotmoaning. INAOE

7. Excuse No. 4 • Thearchetypicalspeech: • “Nobodyfollowsthescientificmethod. Whyshould I?” • Thepointattempted: • Option 1: I’mbl…y lazy, and I don’twanttowaste my time onit • Option 2: I couldn’tcareless of thescientificmethod • Theantidote: • Checkanygroundbreaking/breakthroughpaper in yourarea. • Seeany traces of thescientificmethodonit? INAOE

8. Excuse No. 5 • Thearchetypicalspeech: • “All I needto do forpublishingistorepeattheprocedureotherpeoplehavepublishedbefore” • Thepointattempted: • Scienceisirrelevant. Publishing istheonlythingthatmatters • Theantidote: • Ifyouonlycareforthepublication…youmayenjoyoneortwosuccesses • …ifyoujusthappentoabidebythescientificmethod, you’llenjoyfar more success INAOE

9. Excuse No. 6 • Thearchetypicalspeech: • “I’m a hands-onguy; oldProfs in theirofficeshaveforgotwhat real scienceisallabout.” • Thepointattempted: • Thescientificmethodisjustforoldpeople in theiroffices and isolatedfromthe real world; real scienceis in thelab • Theantidote: • Whileyou are absolutelyrightthatsciencemustbeproved in thelab… • Youmaybesurprisedtoknowthat “some” of thoseoldfellowshadbeenyoung once. • Itistheirexperience and hardworkwhogetthemwherethey are (not chance)… • …nottomentionthatbeforetheygotthere, theyhavehadtoprovedthemselves in thelab! INAOE

10. Thescientificmethod (CLASSIC) INAOE

11. Scientificmethod • In general, science aims at providing explanations of phenomena observed in nature and society • This explanation is offered in terms of relations between an observed phenomena (effect) as a consequence of its possible origin (cause). • These relations are established by means of experimentation following the scientific method. • i.e. experiments are central to science INAOE

12. Scientificmethod • Purpose: • Tobuild a representation of theworld/nature/societythatisaccurate, reliable, consistent and notarbitrary INAOE

13. Objectivism • Thereexistanobjectiverealitywhichisthesameforeveryone [Cotton and Sekula]. • Realityexists as anabsolutegoal: facts are factsregardless of feelings, desires, hopes orfears of people. Ayn Rand, “mother” of objectivism • “If your own private reality includes a law of gravity that is different from Newton's, any predictions you make with it are not going to match reality.” [http://www.physics.smu.edu/pseudo/SciMeth/] INAOE

14. Objectivism • Thereexist [unchanging]lawsbywhichtheuniverseworks [Cotton and Sekula] • Theselaws can bediscovered (notinvented) throughexperimentation • Theselawsmayhowevernotbedeterministic; theymightbestochastic. • Unchangingdoesnot mean static! Thisbecomesespeciallyclear in social research as societyevolveswith time. INAOE

15. Scientificmethod • Fact: • A factisanobservationthat has beenconfirmedrepeatedly and thatforallpracticalpurposesitisaccepted as true. • DefinitionbytheNationalAcademy of Sciences • "Everyone is entitled to his own opinion, but not his own facts.” • Daniel Patrick Moynihan (1927-2003), Senador de los EEUU • "Facts do not cease to exist because they are ignored.“ • Aldous Huxley INAOE

16. A brief (notexhaustive) history of thescientificmethod 384-322 BC: AristotelicEmpiricism ~700-1000. Arabs Ibn al-Haytham , AbūRayhān al-Bīrūnī and Ibn SinaAvicennaamongothersdevelopdifferentforms of experimentation and quantificationtodiscriminateamongcompetingtheories Roger Bacon Aristoteles 1214-1294. Roger Bacon describes thecycle of observation, hypothesis, experimentation and verification 1561–1626. FracisBaconincorporatesinduction as a rationalmethodtoreachcausality 1564-1642. Galileo uses mathematicaldemonstration as a formtoobtainvalidscientificresults Galileo Galilei Francis Bacon 1791-1867. Faradaydemands intelectual honesty and criticismfrom peer (peer review) togetherwithscrupulousdocumentation of experiments so thatthey can bereproduced. S XIX-XX. SeveralcontributionsfromHume (inductivereasoning), Mill (knowledgebasedonexperience), Popper (necessityforfalsability), Peirce (schemeforhypothesistesting and randomization) Michael Faraday INAOE

17. Scientificmethod • Thescientificmethod in a nutshell • Observation of a phenomenon • Formulation of a hypothesisor plausible explanation of thephenomenonthatmightexplaintheobservations • Tocarryoutanexperimentalteringtheconditions and measuring/observingthephenomenonunderthechangingenvironment • Confirmation (orrefutation) of thehypothesisbasedonevidence (observations) collectedduringtheexperimentation INAOE

18. Scientificmethod Figure from: [www.studyblue.com] INAOE

19. Scientificmethod • Thescientificmethodreviewed: • Observation and description of a phenomenon • Formulation of a researchhypothesisor plausible explanation of thephenomenontoexplaintheobservations as a causal mechanism (induction) • I really mean causal • Use thehypothesistopredicttheexistenceorocurrence of otherphenomena, ortoquantify new observations (deduction) • Tocarryoutseveralexperimentsalteringtheconditions and measuring/observingthephenomenonunderthechangingenvironment • Confirmation (orrefutation) of thehypothesisbasedonevidence (observations) collectedduringtheexperimentation INAOE

20. Scientificmethod • Tohave positive evidencesupporting a hypothesisisnotequivalenttodemonstrate a hypothesis, letalonetoconfirm a fact. • …with positive evdienceyouonlyincreaseyourconfidence in thehypothesis • The more experiments are madethatresult in evidencesupportingyourhypothesis, thebiggercertaintyyouhaveonyourhypothesis ¡YOU CAN’T DEMONSTRATE THAT A HYPOTHESIS IS CORRECT/TRUE! INAOE

21. Scientificmethod • Thescientificmethodisaneffort: • Collective of allscientists • Individual researchisunavoidablyinfluencedby personal and cultural bias • Standardizedtominimizebias • Consensusamongthescientificcommunityis a central demand in empiricalresearch [SwanbornPG1996] INAOE

22. Scientificmethod • Thescientificmethodisaneffort: • Dynamic (time) • Itissubjecttocontinuousrevision • Thehypothesisthat has beenholdacrossmanyobservationsmightstillberefuted at any time by new evidence (facts) • "When the facts change, I change my mind. What do you do, sir?" • Atribuido a John Maynard Keynes • Scienceisself-corrective INAOE

23. Scientificmethod • Thescientificmethodisaneffort: • Critic and creative • Criticalthinkingiscapable of decidingwhetheranstatementisalways true, partially true or false. • Creativethinkingpermitsdevelopingunique and useful ideas worth of consideration • http://www.umich.edu/~elements/probsolv/strategy/crit-n-creat.htm • Withoutthesetwothinkingswewouldbeunabletoformulate a hypothesisnortoconfirm/refute them INAOE

24. Scientificmethod • Thescientificmethodisaneffort: • Skeptical • Itrequiresexperimental evidencebeforedoingoracceptinganstatement • …butitadmitsassumptions (lackingevidence) basedonpreviousknowledgeaboutthephenomenon • Skepticaldoesnot mean denier (availableevidenceisneverenough) • Trustis of utmostimportanceamongscientist. • In principle, you trust thatothersresearchers are beinghonest • …yetyouhavetotest/review/challengetheirevidence. INAOE

25. Fromhypothesistofacts Hypothesis Model - Scientificthoery Evidence Law Fact + INAOE

26. Scientificmethod • “In science, you encounter the disturbing fact that, if your "point of view" does not agree with reality as determined by experiment through the scientific method, then your point of view is simply wrong.” • [Cotton y Sekula, http://www.physics.smu.edu/pseudo/SciMeth/] • Havingsaidthat: • Itisacceptablethatourpoint of viewchanges as sciencesgathers more reliableevidence • …butthatchangemustbeguidedbyevidence. • If a establishedtheory (onethat has passedmanyexperiments) is in disagreementwith new evidence, thetheory has tobereviewedmaybeevendiscarded as a description of reality • Itmaystillbevalidwithincertainrestrictions • Classicalexamples: Newton’sgravitylaws (reviewed), Copernicus’sheliocentricmodel (discarded). INAOE

27. Researchmethodology INAOE

28. ResearchMethodology • Thescientificmethodgiveus a general frameworktoexertexperimentation. • …however, itdoesnotgetintodetails of howtocarryoutexperiments • …withoutit, wecan’tmakevalidobservations • Theresearchmethodologyindicateshowtheexperiments are tobeconducted. • …ittellsusthedetails of howtocarryouttheseexperiments • …withoutitwe are likelytofailtocomplywiththescientificmethod. INAOE

29. ResearchMethodology • A researchmethodology ALWAYS has 3 mainelements: • ResearchQuestions (RQs)indicatewhatphenomenonisbeingresearched. • Aims/Goalsindicatewhataspect of thephenomenon are youfocusing • Hypothesis (bothresearch and experimental) representsyourexpectedexplanationforthephenomenon • Hypothesis are “tentative, intelligent guesses” [NentyHJ2009] • Thereis no thesiswithout a previoushypo-thesis!!! INAOE

30. ResearchMethodology • A goodresearchmethodologyensuresthat: • Researchquestions (RQs) are correctlystated • Researchhypothesisoughttobefalsifiable • Aims/Goals are boundedbytheRQs • Experiments are correctlydesignedtoanswertheRQs • Experimental hypothesis are formulatedaccordinglytothedesign INAOE

31. ResearchMethodology • A researchmethodologyoften has 2 parts: • A description of yourexperiments • Byfarthemostimportantpart • Remember: thescientificmethoddemandsexperimentation! • A step-by-step plan of action • Analgorithmtoachievetheabove. • Hopefully, itincludesalso a plan B • …and perhapsalso a plan C • Just a collateralnecessity… INAOE

32. Researchmethodology • Description of yourexperiments: • Foreachexperiment, theresearchmethodologyshouldaimtoinclude: • Itsclear and univocalrelationwiththeRQs • Itsclear and univocalrelationwiththeAims • Itsresearchhypothesis • Its experimental hypothesis • Its experimental design • inc. units, factors, sessions, groups, etc • Variables (dependent, independent and controlled) • Possiblesources of bias • And effortsto reduce them • Theintendedanalysisstrategy • Thevalidationefforts • Bothtypes and mechanisms INAOE

33. A templatefor a thesisresearchmethodology Each RQ results in one experimental chapter. Thethesisisoftencompletedwithanintroductorychapterincludingotherimportantelements; e.g.justification, significance, scope, etc, a chapterwiththeliteraturereview, and a chapterwithconclusions and futureresearch Thesistopic (1-3 wordsmax) In additiontoyour experimental publications, youmayalsopublishyourliteraturereview Oftenaddresses a particular aspect of themaintopic RQ1: ResearchQuestion? RQ2 RQ3 RQ4 Associatedgoal Associatedgoal Associatedgoal Associatedgoal RH1: ResearchHypothesis RH2 RH3 RH4 EX2.1 EX1.1: Experiment EX3.1 EX3.1 EH2.1 EH3.1 EH3.1 EH1.1: Experimental Hypothesis EX2.2 P3.1 P4.1 P1.1: Publication HE2.2 P3.2 Indicates a target conferenceorjournal P2.1 Dr. Felipe Orihuela Espina

34. Experimental DESIGN INAOE

35. Experimental design • “It can be proven that most claimed research findings are false.” • [Ioannidis JPA, PlOS Medicine, 2005 2(8):e124] • >2000 citas (Google scholar) INAOE

36. Contents • Classical scientific method • Experimental elements • Caracteristics of a good experimental design • Randomization • Classicaldesign • Quantitative and qualitativeobservation • Aproximation • Data analysis • Statisticalpoweranalysis • Interpretationofresults • Causality (Presentaciónaparte) INAOE

37. ExperimentalElements INAOE

38. Experimentation • Thestatisticaldesign of a studyconsists of elaboratingan experimental plan, methodicallyorganized so thatthequality and amount of informationobtainedfromitismaximized • [Madero R 2006, An PediatrContin. 2006;4(6):401-4] INAOE

39. Experimentation INAOE Figure from: [https://www.moresteam.com/toolbox/design-of-experiments.cfm]

40. Experimentation • Experiment: • In general: Test involvingreplicatingorobserving a certainphenomenonunderconstrainedcircumstances, oftencontrolled, so thatitseffects can beanalyzed and a hypothesismaybeverifiedorrefuted. • In statistics: A processwhichoutcomesmightbeidentifiedbeforeitsexecution (notnecessarilyforeseenorpredicted) • In ComputerScience: Thatcollection of relatedsimulationsthatyouactuallycallexperiments. • Outcome (a.k.a.result): • The “result” of anexperiment. INAOE

41. Experimentation • Anexperimentalways: • Aims at answering a researchquestion • Has anassociatedgoal • Occasionally, theresearchquestion and thegoal are thesamething • Itisdesignedtoverifyor decide overthevalidity of anhypothesis • Thegoal of a study has tobeexpressed in terms of thehypothesis y determines theparticulars of allotherstepsinvolved in thestudydesign. • [Madero R 2006, An PediatrContin. 2006;4(6):401-4] INAOE

42. Experimentation • Factor • A controlledorindependent variable in anexperimentwhosevalueshavebeenchosenbytheresearcher. • http://www.stats.gla.ac.uk/steps/glossary/anova.html#factor • Anexplanatory variable manipulatedbytheinvestigator • http://stattrek.com/statistics/dictionary.aspx?definition=treatment • Each of thesubdivisions of the factor are referredto as levels. INAOE

43. Experimentation • Factor • A factor A isnested in another factor B ifallthelevels of A are differentforeachlevel of B. • Nested factor havehierarchicalrelations. INAOE

44. Experimentation • Treatment • A combination of specificvaluesacrossthefactors • Note that a treatmentis NOT a case; thetreatmentisappliedtoseveralobjects, and each of theseis a case. INAOE

45. Experimentation • Experimental unit • Eachone of therecipients (objects, algorithms, subjects, etc) overwhichan experimental treatmentisapplied. • Whentherecipients are peopleoranimals, oftenthey are referredto as participantsorsubjects. • In clinicaltrials, sometimesthey are alsocalledpatients. INAOE

46. Experimentation • Example: • A researcher in agriculturewantstoknowtheeffect of a certainfertilizer in corn. • He designsanexperiment in which he controlstheamount of fertilizer (factor) that he will use in eachplant (experimental unit). • He divides theplot of land in 4 areas, and in eacharea he uses a differentdose of fertilizer (treatment). Figure from: [bulletin.ipm.illinois.edu] INAOE

47. Experimentation • Ejxample: • A researcher in pharmacoloywantstoknowtheoptimaldosefor a new drug. • He designsanexperiment in which he controlsthedose of thedrug (factor) • He splitstheparticipants (experimental units) into 3 groups; onegroupreceives a placebo, whilsttheothertworeceivethedrug in differentconcentrations (treatments). INAOE Figure from: [ZhuZ et al, 2011, Journal of Carcinogenesis, 10:17]

48. Experimentation • Example: • A researcher in publicpolicywantstoknowaboutwhichpolicy of watermanagement (factor) is more efficient. • He observes (measureindicators) of a total of 50 municipalities (experimental units) • Then he considersthedifferentpoliciesbythesize of thepopulations in thosemonicipalities; below 10k, between 10k and 50k and over 50k (treatment). Imagen de [www.ec.gc.ca] INAOE

49. Experimentation • Example: • A researcher in computersciencewantstoknowhowtogetthebetterclassification of a dataset (experimental unit) describedby a number of features. • He designsanexperiment in which he classifiesthedatasetaccordingtodifferentfeatureselectiontechniques (factor 1) and differentclassifiers (factor 2). • Eachsimulation, he choosesa pair <featureselectiontechnique, classifier> with a specificparameterization (treatment). INAOE Figure [Self elaborated; project LACCIR]

50. Experimentation • Designmatrix: • Thedesignmatrixis a matrixwhoserowsrepresent experimental units and whosecolumnscorrespondtothedifferentfactors (bothcontrolled and independent) of the experimental design. • Itisoftennamed X. • Thedesignmatrixpermitsrepresentation of most experimental design in a conciseform • Moreover, itisexplicitlyused in severalstatisticalmodels (e.g.the general linear model). INAOE