Experimental methods in empirical research The Rolls-Royce of scientific research? Daniel Gile

Experimental methods in empirical researchThe Rolls-Royce of scientific research?Daniel Gile daniel.gile@yahoo.com www.cirinandgile.com D.Gile Experimental res

What is good scientific research? (In one easy lesson – satisfied or no money back) Many possible definitions, but even the most demanding researchers would probably accept the following: Good scientific research consists in exploring reality rigorously, cautiously, in full awareness of human limitations inherent to such exploration, in compliance with social, editorial and institutional norms of science taking care to avoid and/or limit and/or measure the probability and/or magnitude of possible errors D.Gile Experimental res

Is a specific research method or paradigm part of that definition of good scientific research? D.Gile Experimental res

Is experimental research a prerequisite for scientific research? Japanese psychologist Bears in the mountain Tuna fish migration Astronomy Paleontology… D.Gile Experimental res

Theoretical vs. Empirical methods Theoretical research → ideas: Analysis of facts Analysis of theories Empirical research is centered → data: Collection Analysis (Most often for the sake of developing or testing theories) D.Gile Experimental res

Naturalistic vs. Experimental research Naturalistic research: collecting data about phenomena as they occur ‘naturally’ Experimental research: Creating ‘controlled’ environments to collect data about phenomena in these environments D.Gile Experimental res

Why experimental research? Phenomena which occur naturally are influenced by many factors so It is difficult to trace the influence of one particular factor For example : The success of a learning process May depend on a teaching method (in which the investigator is interested) But could also be influenced By the teacher’s personality By environmental conditions By the student’s motivation By the time available to study… D.Gile Experimental res

Why experimental research? The principles (1) F1 Something happens over time (from t1 to t2) to the object of study O, which turns into O’ We want to know why and how We think factor F1is the cause of this change But perhaps it is F2? Or F3? How do we find out? O O’ F2 t1 t2 F3 D.Gile Experimental res

Why experimental research? The principles (2) F1 We think factor F1 is the cause of this change But perhaps it is F2? Or F3? One possibility is to create an environment Where O is shielded from F2 and F3 So if O changes, it can only be due to F1 This is one type of experimental setup O O’ F2 t1 t2 F3 D.Gile Experimental res

Why experimental research? The principles (3) F1 O O’ Another possibility is to keep F2 and F3 constant and suppress (or change) F1 If nothing happens to O, then it must be F1 which was causing the change F2 F3 ? O F2 F3 D.Gile Experimental res

Why experimental research? The principle (4) F1 In all these (and other) scenarios, We create a ‘controlled’ environment in which we shield the object from outside factors or ‘control’ these factors So as to be able to isolate the influence of whatever factor (a ‘variable’) in which we are interested from the influence of ‘confounding’ factors O O’ D.Gile Experimental res

Experimental research - An example Is translator training method A an efficient booster of translation skills? If we ‘just’ compare the work of 40 translators, 20 of whom have been trained with A and 20 without it, Differences in translation performance may be due to A, but also to: - characteristics of the texts, - age, - experience, - languages involved, etc. If we experiment with the same text and two groups with similar characteristics (age, experience, languages etc.) If differences found, probably due to A D.Gile Experimental res

Experiments for hypothesis-testing This is a prototypical model of hypothesis-testing experiment The behavior of 2 or more groups is observed Each is made to be similar to the others except for the value of the variable to be examined Comparisons of output variables (‘dependent variables’) are made with a view to determine whether differences are likely to be due to chance or not If not, they are called ‘significant’. D.Gile Experimental res

‘What if’ experiments Hypothesis-testing experiments are the best known But not the only ones Exploratory experiments are also used: What if…? (No special hypothesis) What if I gave a translator this type of reference document? What if I made a translator work in a team with an expert without any knowledge of the source language? What if I doubled his/her pay when the customer is happy with the product? What if… D.Gile Experimental res

‘What if’ examples Caveman – biological clock Isolated Community of volunteers Big Brother  Agriculture Children in a room Sometimes with hypotheses or theories (formalized) Sometimes without D.Gile Experimental res

Advantages of experiments Eliminate the ‘confounding’ influence of ‘parasitic’ variables Often allow to measure some indicative value more accurately than in naturalistic studies (precise corrections made during translation, pauses etc.) Allow to create interesting situations which do not occur or occur very seldom in reality and therefore could not be studied naturalistically Make it possible to study situations with small investments as opposed to huge investments which studying the same phenomena with naturalistic methods would entail D.Gile Experimental res

Why do experiments have such a high status in science? - Because of the advantages outlined in the previous slide - Because often used to test theories and therefore take place at an advanced phase of the research process on a given phenomenon - Because they often involve sophisticated thinking and design - Because they most often involve statistics, which is a highly sophisticated tool BUT DOES ALL THIS MAKE THEM MORE ‘SCIENTIFIC’ THAN NATURALISTIC STUDIES? D.Gile Experimental res

Limitations of experiments: ecological validity Most often cited limitation of experiments in the social sciences has to do with ecological validity: Is the task and are the measurements a valid model for real life? For example: A translator who translates in a particular room under the observation of cameras or an experimenter or on a specific computer with a specific type of software or under specific time constraints or with specific instructions as to the use of data or with the instruction to think aloud while translating … Does s/he perform the same translation task as in real life? If not or if the answer is uncertain, can conclusions on real life be drawn from the experiments with the caution prescribed by science? D.Gile Experimental res

Limitations: neglected confounding variables In experiments, confounding variables are controlled. But are all of them controlled? - Potentially relevant of which the investigator is not aware? - Variables which cannot be controlled because there is no way to control them or controlling them is too expensive/impractical? (family traditions, some genetic factors, some personality traits, economic situation and history, history of social interaction…) - Variables which cannot be controlled without reducing the sample to an insignificant size? (only subjects aged 20 to 25, right-handed, with a given personality profile, with parents from a particular background and having a specific economic status, who attended particular schools with particular curricula, who read at least 20 books a year…) D.Gile Experimental res

Neglected confounding variables F5 F4 F1 If we control F2 and F3 but not F4 and F5 Can we safely assume that the effect measured in the experiment is only due to F1? The answer is clearly No O O’ F2 t1 t2 F3 D.Gile Experimental res

Limitations: experiments as a case study By ‘controlling’ relevant variables and parameters, For instance all subjects must be of the same age Or right-handed, or work on one particular source text Or have 5 to 10 years of professional experience Experimenters deliberately exclude some naturally occurring variability And it is difficult to say with the appropriate caution prescribed by science Whether the results would also be replicated with other values If there are many replications of the experiments with different values (and without bias), fine But if there aren’t? D.Gile Experimental res

Limitations: statistical significance (1) Inferential statistics (statistical tests) are used to determine whether differences are ‘significant’ or not Variables in a population have certain values and a ‘distribution’ (How tall, how many times go to church, how much money spend on Belgian chocolate…) In order to check whether some factor makes a difference (more training, more work, better food…) Check whether the population with the ‘extra’ has the same distribution of values than the population without the ‘extra’ (or with a certain value in that variable and another value) D.Gile Experimental res

Limitations: statistical significance (2) How will a statistical test check this? You will measure the values of a relevant output variable (for instance some metric of translation quality) On samples from the two populations (the two conditions) You will find some difference between the mean values The test will tell you how likely it is that the difference is compatible with the idea that the two populations have the same distribution for the relevant variable If it tells you this is unlikely, you will say the difference is ‘significant’ D.Gile Experimental res

Limitations: statistical significance (3) Actually, it will determine, on the basis of measurements made on the sample means A certain distribution And the proportion of values which lie between a lower and an upper limit if the two populations are not different If your value is below a threshold or above another threshold, It will tell you that it corresponds to only 5% or 1% of the cases still compatible with the idea that the two populations are the same “significant at .05” “significant at .01” D.Gile Experimental res

Limitations: statistical significance (4) ) D.Gile Experimental res

Limitations: statistical significance (5) ‘Significance’ is determined at a certain level: the experimenter allows him/herself a certain probability of ‘false positive’ results (of wrongly concluding that the difference is not produced by chance) Significance at 5% (.05) means that the experimenter allows him/herself to draw the wrong conclusion that the difference is ‘real’ once every 20 times If you were to decide that something is true or not, would you say that it is true if you have one chance out of 20 to be wrong? (Significance at 1% means that you could be wrong once in 100 times) D.Gile Experimental res

Limitations: statistical significance (6) Statistical significance tests rely on a number which is calculated from values obtained from the samples being examined, with a threshold For instance, if you get 5,62 the difference is significant, if you get 5,63 it isn’t. How do you like this transition from ‘yes’ to ‘no’? Does it make sense to you? Statistical significance says the difference is likely to be due (or not to be due) to chance. It does not say how large the difference is on average. In TS, how useful is knowing that there is some difference, but without knowing how large it is? D.Gile Experimental res

So are experiments not a good research paradigm? Experimental research can be very useful: It is potentially powerful in eliminating (some) confounding variables It does allow observation of situations which would not occur naturally (but in the social sciences, what about ecological validity?) It does allow precise measurements which would be difficult or impossible to carry out in real life It can take on board variability, measure it and overcome it But it has its limitations, and works best under certain conditions D.Gile Experimental res

Conditions for powerful experimental research Proper design, and in particular proper sampling and piloting Cautious inferencing from the results Correct statistics (test that conditions for the use of tests are met) Many replications with various values of the independent variable (so that you really cover a range of values and move away from the case-study scenario) How often are such conditions met in your field? D.Gile Experimental res

If conditions are not met Still useful, but for ideas and tentative results Experimental results are by no means the final ‘scientific criterion’ for a decision In particular, ‘significance’ is just a tentative direction Naturalistic findings can be more powerful Especially with the use of corpora Because they allow the use of much authentic data Can even test hypotheses, more powerfully than experiments! D.Gile Experimental res

Conclusion Experiments are a tool A powerful tool under certain conditions (and in particular in cognitive psychology) But only a tool Not very powerful under other conditions NOT THE ULTIMATE IN SCIENCE Don’t let scientific status symbols fool you Science is in the human mind When you explore reality with a rigorous, skeptical, cautious, systematic mindset, You are ‘scientific’ whatever the tool D.Gile Experimental res

Experimental methods in empirical research The Rolls-Royce of scientific research? Daniel Gile