How Research Methods Textbooks Mislead Psychologists About Scientific Method

1. How Research Methods Textbooks Mislead Psychologists About Scientific Method Brian Haig University of Canterbury 5 April 2013

2. Introduction 1. Textbooks are the major source of formal learning in undergraduate education, and they are prominent in graduate education as well. 2. In psychology, students take more “methods” courses than any other type of course (e.g., University of Canterbury: Psyc206; Psyc344; Psyc460/Psyc461 are all compulsory for graduate research students). 3. All undergraduate students take a generic second- or third-year course on research methods. 4. There are dozens of textbooks available for adoption in such courses.

3. 4a A Sample of Textbooks Gravetter, F. J., & Forzano, L-A. B. (2009). Research methods for the behavioral sciences (3rd ed.). Belmont: Wadsworth. Leary, M. R. (2012). Introduction to behavioral research methods (6th ed.). Boston: Allyn & Bacon. Mitchell. M. L., & Jolley, J. M. (2013). Research design explained (8th ed.). Belmont: Wadsworth. White, T. L., & McBurney. D. H (2013). Research methods (9th ed.). Belmont: Wadsworth.

4. 5. Thomas Kuhn (1970) argued that textbooks in the various sciences contribute in a major way to the dogmatic initiation of learners into established research traditions. 6. Psychology’s research methods textbooks are largely uncritical in the presentation of their subject matter. 7. It follows that they cannot be a source for a genuine education in research methods. 8. I identify a number of deficiencies in the way psychology’s research methods textbooks deal with scientific method. 9. Method is central to science, and scientific method is, therefore, important. Most of what we know in science is acquired through application of its methods.

5. The Typical Research Methods Textbook 1. The standard textbook possesses a common structure (psychology as a science, research questions, ethics, writing, validity, experimental and nonexperimental research, data exploration and inference, … .). 2. The first chapter (approx. 35 pages) is often dedicated to scientific method, broadly understood. 3. Cameron Ellis and I did a content analysis of a representative sample of 16 current undergraduate research methods textbooks (Haig & Ellis, 2013). 4. All textbooks have been through multiple editions (range=2-11; median=5.5; latest edition=10/16).

6. A Bill of Indictment 1. Most textbook authors are not methodologists by speciality (only 6 of 28 in the sample, with ‘methodologist’ construed generously). 2. The most serious limitation of textbooks on research methods is that the methods dealt with are not informed by their accompanying methodology (Methodology is the interdisciplinary field that studies methods. It includes statistics, philosophy of science, and cognitive science). This results in an impoverished understanding of the methods.

7. 3. Rarely are methodological references on scientific method provided. i) few citations of professional methodologists and their work (including philosophers of science) ii) heavy reliance on secondary sources (when citations provided). Example: Stanovich (1998), How to think straight about psychology (5th ed.) iii) methodological points about scientific method often made by appealing to substantive research (particularly in social psychology), which is referenced much more than methodological research.

8. 4. The changes in successive editions of books are often minimal. a) revisions are driven more by attempts to capture, or retain, market share than present new methodological developments. b) publishers encourage authors to produce new editions regularly in order to encourage sales of new books (the “problem” of second-hand book sales).

9. Theories of Scientific Method Ignored An Argument: P1 Credentialed theories are the major vehicles of scientific knowledge. P2 This holds for method as well as matter. P3 Major theories of scientific method are not presented in textbooks. Cl Texts books do not present genuine knowledge about scientific method. This state of affairs is both embarrassing and unacceptable.

10. Major Theories of Scientific Method 1. Inductive Method - 10/16 texts mention induction, but: - Most characterize inductive inferencewithout relating it to inductive method (e.g., radical behaviorist method, Bayesian confirmation theory). - Induction characterized as enumerative (or generalizing) induction. But there are different types of induction (e.g., enumerative, eliminative, probabilistic). These are not mentioned or distinguished in any of the 16 texts. - Inductive nature of constructive replication not explicitly stated in any text.

11. 2. Hypothetico-Deductive Method - The most popular account of scientific method - Claims a hypothesis or theory is evaluated in terms of its test predictions - 9/16 texts neither mention nor describe it. - 5/16 vaguely describe it without naming it. - 2/16 name it and describe it. - 1/16 says it is controversial. - No text talks about its limitations or its proper scope of application.

12. 3. Inference to the Best Explanation - Concerned with explanatory reasoning. In brief, it captures the idea that theories are evaluated according to their explanatory worth. - Frequently used in the evaluation of scientific theories (e.g., Darwin’s theory of natural selection) - Not mentioned or referenced in any of the 16 texts - Methods of inference to the best explanation are available (e.g., the theory of explanatory coherence). - Structural equation modeling that combines model fit with model parsimony can be regarded as a hybrid account of inference to the best explanation.

13. 4. Bayesian Method - Widely regarded as the leading theory of scientific confirmation in contemporary philosophy of science - Not mentioned, referred to, or indexed in any of the 16 textbooks (as ‘Thomas Bayes’, ‘Bayes’ theorem’, ‘Bayes factors’, ‘Bayesian statistics’, or ‘Bayesianism’) - Note that Bayesian statistics is beginning to be promoted by a small group of behavioral science methodologists. Conclusion: One cannot learn much of worth about scientific method from methods textbooks that claim to deal with scientific method! (Additionally, one can learn a good number of things that are not so).

14. Psychologists on Scientific Method A few psychologists have written about scientific method in a serious vein: Cattell, R. B. (1966). Psychological theory and scientific method. In R. B. Cattell (Ed.), Handbook of multivariate experimental psychology (pp. 1–18). Chicago, IL: Rand McNally. Haig, B. D. (2005a). An abductive theory of scientific method. Psychological Methods, 10, 371-388. O’Donohue, W., & Buchanan, J. A. (2001). The weaknesses of strong inference. Behavior and Philosophy, 29, 1-20. Rozeboom, W. W. (1997). Good science is abuctive, not hypothetico-deductive. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 335-391). Hillsdale, NJ: Erlbaum. Skinner, B. F. (1956). A case history in scientific method. American Psychologist, 11, 221-233. None of these references are cited in any of the 16 texts.

15. What Do Methods Textbooks Say About Scientific Method? 1. “Scientific Method is the Scientific Method” - 11/16texts spoke of the scientific method (with no agreement on what it is!). - 3/16 spoke about a scientific method. - 2/16 didn’t mention scientific method at all. There is no such thing as the scientific method. Instead, we have a plurality of scientific methods.

16. 2. “Scientific Method is a Series of Steps” -9/16 texts described scientific inquiry as a series of steps (as few as 4 steps, as many as 11) - the steps listed vary greatly: Example1: Gravetter & Forzano, Research methods for the behavioral sciences (2009). Step 1: Observe behavior or other phenomena Step 2: Formulate tentative explanatory hypothesis Step 3: Use the hypothesis to make a testable prediction Step 4: Evaluate the prediction via planned observations Step 5: Support, refute, or refine the original hypothesis … and so on in a circle or spiral

17. Example 2: ‘HOMER’: An acronym for the five steps of the scientific method “ Each letter of Homer stands for one of the five steps in the scientific method : hypothesize, operationalize, measure, evaluate, and replicate/revise/report.” (Lakin, et al., 2007) The propensity to characterize scientific method as a series of fixed steps is probably due to John Dewey’s (1910) five-step account of scientific method, which had an enormous influence on American education (Rudolph, 2005). Dewey later revised his formulation to do away with fixity of order and spoke of phases, not steps.

18. 3. “Scientific Method is a Process of Falsification” a) Falsification is the process of showing a claim to be false. b) 13/16 texts mention ‘falsification.’ c) All texts fail to distinguish between the process of falsification and Popper’s theory of falsificationism with its attendant ideas of conjectures and refutations, corroboration, and demarcation. d) 3/13 texts mention that falsification(ism) is controversial, but none of them deal with the well- known criticisms of it in the philosophy of science.

19. Phenomena Detection and Theory Construction 1. The methodological distinction between phenomena detection (e.g., the discovery of empirical regularities) and theory construction (the generation, development, and appraisal of explanatory theories) is of fundamental importance to science. 2. These are two quite different sorts of undertaking and they employ different research methods (Haig, 2013). 3. 1/16 texts explicitly draws the distinction (Leary, 2012). 7/16 almost draw it (e.g., ‘law’/‘theory’ distinction). 8/16 fail to draw the distinction. 4. No text systematically discusses different research methods in relation to the distinction.

20. 5. At best, textbook treatments of theory construction focus on hypothetico-deductive theory testing for empirical adequacy. 6. Methods specifically tailored to theory generation (e.g., exploratory factor analysis), theory development (e.g., analogical modeling), and theory appraisal (e.g., inference to the best explanation) all deserve a proper place in research methods textbooks (Haig, 2005). 7. Methods textbooks in psychology emphasize data analysis at the expense of theory construction.

21. The Neglect of Philosophy of Science The philosophy of science is an important part of scientific methodology. It is a major resource for leaning about the conduct of scientific research. Yet, the 16 texts make little or no use of it (3/16 mention it; none use it). Popper (1959) and Kuhn (1962), are sometimes mentioned; Lakatos (1970) and Laudan (1977) are occasionally mentioned. None of them are used. The last 25 years of developments in the philosophy of science have been completely ignored (see Blachowicz, 2009).

22. As a consequence of this general neglect, these texts have failed to capture the profound changes in our understanding of science that philosophy of science has brought about in the last 50 years (e.g., Proctor & Capaldi, 2001). Early editions of the McGuigan and Leary texts stressed the importance of philosophy of science in understanding research methods. “An understanding of the philosophy of science is important to an understanding of what science is, how the scientific method is used, and particularly of where experimentation fits into the more general framework of scientific methodology”. (McGuigan, 1960, p. 111)

23. Interestingly, both McGuigan and Leary dropped reference to philosophy of science in later editions (publishers’ pressure to conform?). In his section on philosophy of science, Leary (2001) claimed: 1. Many philosophers of science are behavioral scientists. 2. Many researchers take courses in the philosophy of science. I think both claims are false. It would be good for both disciplines if they were true.

24. Concluding Thoughts 1. Textbooks should present inductive, hypothetico-deductive, and Bayesian methods, as well as inference to the best explanation, as major local theories of scientific method with domain specific applications. 2. Knowledge of these scientific methods would help learners better understand the process of phenomena detection, the complexities of theory testing, and the nature of explanatory reasoning in science. 3. The dominant “twofer” model (Aiken, West, & Millsap, 2008) (where researchers with expertise in a substantive area of psychology, teach statistics and research methods courses) extends to writing research methods textbooks.

25. This two-tiered twofer model poses serious problems for psychology because it contributes to a decrease in the quality of instruction about scientific method generally (and research methods more specifically). 4. The content of these four major theories of method cannot be taught without making substantial use of the relevant philosophy of science literature(this use of philosophy of science should extend to instruction about methodological processes more generally). 5. Textbooks on research methods should be written by professional research methodologists, thus ensuring that they are appropriately methodologically informed. 6. These texts should deal with the full range of behavioral research methods and strategies.

26. 7. Psychology needs just a few up-to-date general research methods textbook (say, 5, not 50) written by professional methodologists. 8. These texts would likely vary depending on the methodological predilections of the authors. Example: Rosnow and Rosenthal (2013), Beginning Behavioral Research: A Conceptual Primer (emphasis on meta-analysis, effect sizes, and the binomial effect-size display, reflecting Rosenthal’s own methodological work). 9. This variation in textbook content would be appropriate for an education in behavioral research methods, where it is genuinely controversial as to what we should learn.

27. References Aiken, L. S., West, S. G., & Millsap, R. E. (2008). Doctoral training in statistics, measurement, and methodology in psychology. Replication and extension of Aiken, West, Sechrest, and Reno’s (1990) survey of PhD programs in North America. American Psychologist, 63, 32-50. Blachowicz, J. (2009). How science textbooks treat scientific method: A philosopher’s perspective. British Journal for the Philosophy of Science, 60, 303-344. Dewey, J. (1910). How we think. Boston: Heath. Haig, B. D. (2005). An abductive theory of scientific method. Psychological Methods, 10, 371-388. Haig, B. D. (2013). Detecting psychological phenomena: Taking bottom-up research seriously. American Journal of Psychology, 126, 135-153.

28. Haig, B. D. , & Ellis, C. (2013). How research methods textbooks mislead psychologists about scientific method (in preparation). Kuhn, T. S. (1962/1970). The structure of scientific revolutions (2nd ed.) Chicago: University of Chicago Press. Lakatos, I. (1970). Falsification and the methodology of scientific research programmes. In I. Lakatos & A. Musgrave (Eds.), Criticism and the growth of knowledge (pp. 91-196). Cambridge: Cambridge University Press. Lakin, J. L., Giesler, R. B., Moros, K. A., & Vosmik, J. R. (2007). HOMER as an acronym for the scientific method. Teaching of Psychology, 34, 94-96. Laudan, L. (1977). Progress and its problems: Towards a theory of scientific growth. Berkeley: University of California Press.

29. Nickles, T. (1987). Methodology, heuristics, and rationality. In J. C. Pitt & M. Pera (Eds.), Rational changes in science (pp. 103-132). Dordrecht: Reidel. Popper, K. R. (1959). The logic of scientific discovery. London: Hutchinson. Proctor, R. W., & Capaldi, E. J. (2001). Improving the science education of psychology students: Better teaching of methodology. Teaching of Psychology, 28, 173-181. Rudolph, J. L. (2005). Epistemology for the masses: The origins of “the scientific method” in American schools. History of Education Quarterly, 45, 341-376.