Threats to External Validity in Research Studies

1. How does this relate to external validity of a research study? What is the biggest threat to external validity, in terms of sampling?

3. Now let us look at our class research study: Parenting and Political Tolerance. Based on the handout: • Does the study attempt to make a frequency claim, an association claim, or a causal claim? • How does one go from an association claim to a causal claim? Hint: internal validity! • Based on your answer, how important is it to have a representative, non biased sample? • What are the hypotheses of the research study? What is a “null hypotheses” of the study? There will be various answers • Are the variables manipulated or measured? • Identify which variables are dependent, and which ones are independent. What do you base this on? Might there be a mediating variable? • Draw a scatterplot depicting some results that would be consistent with our research hypothesis. Now draw a scatter plot that would be consistent with our null hypotheses.

4. Construct validity for our study • What is construct validity? • How were our variables operationalized? • Did we do it, or did someone else do it? • What does it mean if a measure has been previous “validated” • List the three variables in terms of measurement scales • Under each variable list the sub scale measurements

7. https://www.researchgate.net/publication/283946872_Reliability_and_Validity_of_the_Multidimensional_Locus_of_Control_IPC_Scale_in_a_Sample_of_3668_Greek_Educatorshttps://www.researchgate.net/publication/283946872_Reliability_and_Validity_of_the_Multidimensional_Locus_of_Control_IPC_Scale_in_a_Sample_of_3668_Greek_Educators Convergent and Discriminant Validity of the Locus of Control Construct. Borich, Gary D.; Paver, Sydney W. Eighty undergraduates were administered four self-report locus of control inventories, in order to evaluate the convergent and discriminant validity of four categories common to these inventories: chance, fate, personal control, and powerful others. The four inventories were: (1) Internal, Powerful Others and Chance scales; (2) James Internal External Locus of Control Scale; (3) Nowicki Strickland Locus of Control Scale; and (4) Rotter Internal External Locus of Control Scale. Pearson correlations were computed between methods (inventories), and traits (categories), to construct a multitrait-multimethod matrix, following the Campbell and Fiske model. Validity was evaluated by observing intercorrelations among cells in the matrix. Convergent validity was consistently demonstrated for only one category--fate; discriminant validity was not evident for any comparison category across the four instruments. Rates of validity over all comparisons were 66.6% for convergent and 12.5% for discriminant. One explanation for inconsistent findings across locus of control studies is that the multitrait-multimethod matrix accounts for more variability than the categories measured. (The appendix lists items in each instrument which measure chance, fate, personal control, and powerful others.) (CP)

11. https://www.sciencedirect.com/science/article/pii/S0191886914006989https://www.sciencedirect.com/science/article/pii/S0191886914006989

14. Study Questions: week four • A maximum of ten extra credit points can be achieved by turning in an analysis of the debate: Is there a free speech crisis on campus? (see video link under week four) • In your own words, summarize the main arguments by each of the four panelists. What empirical claims were made? What normative claims were made? Be sure to identify hidden implicit claims (unstated). • Which arguments were most convincing to you and why? • What is a biased sample? How might this effect the external validity of a study? • How could you obtain a biased sample? How do researchers try to avoid biased samples? What is random sampling? • What types of claims would be most effected by sampling bias? Why might our study (parenting and political tolerance) is less concerned about this? • Be able to identify a claim that has a problem with external validity, due to sampling bias. • Review construct validity. Why is this important in many psychological studies? • How might you measure “happiness”, as a social psychologist? Why are definitions important in terms of construct validity? • Understand the following forms of construct validity in terms of definitions and be able to give examples: face validity, criterion oriented validity, convergent validity, and discriminant validity. I will ask questions similar to the ones that I asked on Friday.

15. If I give you the scenario of “Tom is shy, reserved and keeps to himself”, and I ask you if he is enrolled in a math PhD program or business school: • How would most people answer the question? Now describe how system one makes an incorrect heuristic judgment. In your description, use the following terms: base rates, priors, stereotypes, predicting by representativeness • Now answer the question using system two and Bayesian statistics/thinking. • Be familiar with the description of Linda on p. 156 in TFS (Kahneman) • What do most people guess: Is Linda more likely to be a bank teller or a bank teller and active in the feminist movement? • Which answer is correct? • Why do people generally answer this question incorrectly? What is this called? Be able to draw a Venn diagram that demonstrates this. • Why do we overemphasize the “active in the feminist movement” and discount the “bank teller” (hint…stereotypes and predicting by representativeness). What does Kahneman mean by “plausible” versus “probable”? • Are larger set valued more in a joint evaluation versus a single evaluation? Why? What is system one doing here and what is it incapable of doing that creates the flawed thinking that “less is more”? Be sure to understand this concept with examples from the book. • How can we make system two less lazy, and what might be the best way to “bust” stereotypes?

16. Review the concepts of “regression to the mean/mediocrity”: • Why did Sir Francis Galton observe that offspring didn’t resemble their parents, and that the more extreme parents had offspring that regressed more toward “mediocrity”? • Be able to explain the following claim: “highly intelligent women tend to marry men who are less intelligent than they are”. In your explanation, make a scatter plot, and refer to correlation values of r. Why is this difficult for most people to comprehend? (hint…causation and r=1.0). • If I sold you a B.S. sugar supplement meant to treat severe depression and anxiety, why might it work, other than having a placebo effect? • Taming System One predictions: “Julie is currently a senior in a state university. She reads fluently when she was four years old. What is her GPA? • How does system one answer this question? What is a typical prediction? Does it substitute a complex question for a heuristic question? What does it assume? Draw a scatterplot of this! • What should you do instead? (hint, Mike says: “decorrelate, and anchor on a base rate”) • Be able to explain the differences between the two predictions above, and why the latter is more accurate.

17. In regards to our Parenting and Political Tolerance study, answer the following questions: • Does the study attempt to make a frequency claim, an association claim, or a causal claim? • How does one go from an association claim to a causal claim? Hint: internal validity! • Based on your answer, how important is it to have a representative, non biased sample? • What are the hypotheses of the research study? What is a “null hypotheses” of the study? There will be various answers • Are the variables manipulated or measured? • Identify which variables are dependent, and which ones are independent. What do you base this on? Might there be a mediating variable? • Draw a scatterplot depicting some results that would be consistent with our research hypothesis. Now draw a scatter plot that would be consistent with our null hypotheses. • How were our variables operationalized? • Did we do it, or did someone else do it? • What does it mean if a measure has been previous “validated” • List the three variables in terms of measurement scales • Under each variable list the sub scale measurements

18. In your own words, interpret the following statement: “intuitions come first, strategic reasoning second”. In your interpretation, try to either create a concept map or a detailed description that summarizes Part one of Haidt’s book. • What is confirmation bias, and be able to identify this bias if given an example. • What is cognitive dissonance, and be able to identify this if given and example. • Please review the ethical theories of deontology and consequentialism. I may ask questions like the ones that I asked previously on the last quiz. • Be able to succinctly answer the following questions: Where do you get your moral foundations from? Where do humans get their moral foundations from? • Why might a monkey be afraid of a plastic snake, even though it has never seen a snake before in its life? • What is bio-cultural evolution? Why does Haidtuse taste buds and different cuisines as an analogy to human morality? How are taste buds biological “social receptors”? Why might the students in our class have common moral foundations, but a different emphasis on which foundation is more important? Answer this question from an evolutionary perspective, and a bio-cultural perspective. • Be able to identify the five moral foundations in the Moral Foundations Theory (MFT). For each foundation be able to identify examples (that may be different than your table) of the following: adaptive challenge, original triggers, and current triggers. Pay special attention to evolutionary explanations for why we might have these moral foundations, like the ones we covered on Friday.