Threats to the Credibility of Research

1. Threats to the Credibilityof Research Farzin Madjidi, Ed.D. Pepperdine University G.S.E.P.

2. Threats to Credibility • External Validity • Internal Validity • Sound Practices • Note: These are very different from validity of a test item or an instrument

3. External Validity • The extent to which the results can be generalized to other subjects or setting • External validity can only be measured on a case-by-case basis • Note: In experimental research, the primary purpose is to control internal validity (minimize external influences). It is not unusual to have experimental studies strong in internal validity and weak in external validity

4. Threats to External Validity • Population Validity • Is your subject-sample-population consistent? • Sample size • Setting Effects • Data are not corroborated between researcher and participants • Theoretical Effect • Findings are not contrasted with the results of other research • Type of Logic Used • Deductive Logic • Inductive Logic

5. Threats to External ValidityType of Logic Used Deductive Logic: • Top-down in nature • A form of inference; observe a small group and draw inferences to the larger group • If conclusions follow the premises, results are valid • If one or more of the premises are untrue, or if the argument form is invalid, the results are invalid • Limitations: • Restriction of assumptions • Even all the assumptions don’t occur simultaneously, the theory may hold

6. Threats to External ValidityType of Logic Used Inductive Logic: • Uses the knowledge of the past to explore the state of the future • Bottom-up in nature • Used mostly in Qualitative designs • Limitation • Fallacy of Composition; what is true for the part may not be true with the whole

7. Internal Validity • The degree to which the independent variable and not the extraneous or confounding variables produce the observed results • A study is considered “Strong” if most plausible extraneous and confounding variables are controlled

8. Threats to Internal Validity History (impact of time) • The amount of time that elapses between the intervention and the measurement of the outcome • Impact of events that may occur during the experiment (layoffs during your assignment) • Time of the day when the intervention/experiment is conducted (morning may yield different results than afternoons or evenings) • Absence of baseline data

9. Threats to Internal Validity Selection • Random Selection/Assignment • One location receives an intervention and another location doesn’t. Subjects within assigned groups are not randomly selected • Purposeful sampling rationale and process is omitted • Use of Volunteer • Volunteers may respond differently than non-volunteers

10. Threats to Internal Validity Maturation (changes in subjects over time) • Physical changes • Social changes • Mental changes • Fatigue • Boredom • Hunger • Other inappropriate behavior

11. Threats to Internal Validity Pre-testing (measuring the DV before intervention) • Subjects may act differently (sensitized) • Pre-tests may lead to improved scores on the post-test Instrumentation • Changing observer/interviewer • Scores on some standardized tests don’t change by very much!

12. Threats to Internal Validity Statistical Regression • Tendency of extreme scores to get closer to the mean on subsequent measurements • This is a result of measurement error and statistical probability Diffusion of Treatment • If the members of “experiment” group come in contact with the “control” group members, or know about the experiment, the results may get effected

13. Threats to Internal Validity Subject Effects (impact of experimenter’s characteristics) • Subjects may not behave naturally and honestly (reactivity) • Subject may provide demand characteristics • Socially desirable behavior • Increased positive behavior (Hawthorne effect) • Compensatory Rivalry - John Henry effect (since you didn’t pick me for the experiment group, I will show you!) • Attrition (loss of subjects) • Resentful Demoralization (the reverse of the above)

14. Threats to Internal Validity Experimental Effects • When the characteristics of the experimenter effects the outcome • Degree of contact with the experimenter • Age, sex, race, hostility, authoritarianism, etc. • Expectations of the Researcher Alternative Explanations (occurs when there is a lack of multiple perspectives) • Key is: Minimizing the reasonable and Plausible threats that are likely to affect the results

15. Sound Practices • Quantitative Analyses • Consistency of logic (Problem/purpose/RQs) • Proper inference (AU/Sample/Population) • Appropriate statistical analysis • Tests of significance before inference • Design methods, sampling, and data collection methods clearly stated • All assumptions are clearly stated • Clear statement of the researcher’s background, interests and bias

16. Sound Practices • Qualitative Studies • Prolonged engagement and persistent observation • Triangulation (use of multiple sources) • External check of research process through peer review and external checks • Negative case study (revising hypothesis to eliminate outliers until all cases fit • Clear statement of the researcher’s background, interests and bias

17. Sound Practices • Qualitative Studies (cont’d) • The conceptual and theoretical framework for the study are clear • The method of selecting participants are clear • A clear description of researcher involvement in the setting are included • Detailed and thorough fieldnotes with complete descriptions are maintained • Descriptions and interpretations are separated

18. Sound Practices • Qualitative Studies (cont’d) • Member’s check - to check the credibility of interpretations and findings with subjects • Providing rich, thick descriptions • External audits - allowing an external consultant to audit both the process and the product of the account to assess their accuracy • Addressing the credibility of research