1 / 23

Maximizing Construct and External Validity in Experiments

Learn about the importance, threats, and assessment of construct validity and external validity in experimental research, including key factors like definitions, threats, and assessment methods.

sdarrow
Download Presentation

Maximizing Construct and External Validity in Experiments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Construct and External Validity in Experimental Research ♣ Chapter 8 Back to Brief Contents • Construct Validity  • External Validity  • Cautions in Evaluating the External Validity of Experiments  • Relationship between Internal and External Validity 

  2. 8.1 Construct Validity-1 Back to Chapter Contents • Definition—extent to which we can infer higher-order constructs for our operations Fig 8.1 • Constructs are used for • Research participants • Independent variable • Dependent variable • Experimental setting

  3. 8.1 Construct Validity-2 Back to Chapter Contents • Assessing Construct Validity  • Threats to Construct Validity  • Reactivity to the Experimental Situation  • Experimenter Effect 

  4. Assessing Construct Validity-1 Back to Chapter Contents • Have a clear definition of the construct of interest • Problem is identifying prototypical features of the constructs • Affects ability to identify the concrete operations used to represent the construct

  5. Assessing Construct Validity -2(end) Back to Chapter Contents • Assess the match between the constructs and the operations used to represent them • Ways to assess • Content validity • Criterion-related validity • Predictive • concurrent • Any other source of evidence

  6. Threats to Construct Validity Back to Chapter Contents • Inadequate explanation of the construct Tab 8.1 • Construct confounding • Mono-operation bias • Mono-method bias • Confounding constructs with level of constructs • Treatment-sensitive factorial structure • Reactive self-report changes • Reactivity to the experimental situation • Experimenter effects • Novelty and disruption effects • Compensatory equalization • Compensatory rivalry • Treatment diffusion

  7. Threats to Construct Validity Back to Chapter Contents • Inadequate explanation of the construct If a construct is not adequately explained and analyzed, it can lead to a set of operations that do not represent the construct adequately • Construct confounding the operations used in a study represent more than one construct • Mono-operation bias a study uses only one operationalization of a construct This typically results in an underrepresentation of the construct and lowers construct validity

  8. Threats to Construct Validity Back to Chapter Contents • Mono-method bias a study uses only one method (e.g., physiological recording) to operationalize a construct The method used may influence the results • Confounding constructs with level of constructs a study investigates only a few levels of a construct (e.g., three doses of a drug), but makes inferences about the overall construct (e.g., the overall effect of the dose) • Treatment-sensitive factorial structure an instrumentation change that occurs because of the experimental treatment

  9. Threats to Construct Validity Back to Chapter Contents • Reactive self-report changes changes that a research participant may make on self-report measures as a result of a motivational shift after being included in the experimental study • Reactivity to the experimental situation research participants’ perceptions and motives can affect the responses they make to the dependent variable and these responses can be interpreted as part of the treatment construct being tested

  10. Threats to Construct Validity Back to Chapter Contents • Experimenter effects the experimenter’s attributes and expectancies can influence the responses made by the research participants and these responses can be interpreted as part of the treatment construct being tested • Novelty and disruption effects Research participants usually respond better to a new and novel situation and poorly to one that disrupts their routine These effects are part of the overall treatment effect

  11. Threats to Construct Validity Back to Chapter Contents • Compensatory equalization individuals try to provide the same benefits or services to the control group that are received by the experimental group • Compensatory rivalry individuals resent being assigned to the control group and respond more negatively than would be expected, because of the resentment they feel • Treatment diffusion individuals in one treatment group receive some or all of another group’s treatment

  12. Reactivity to the Experimental Situation 1/3 Back to Chapter Contents • Refers to research participants’motives and perceptions influencing the response to the DV • Participant Effect • Demand characteristics Any of the cues available in an exp, such as instructions, experimenter, rumors, experimental settings • Primary motive —positive self-presentation

  13. Reactivity to the Experimental Situation 2/3 Back to Chapter Contents • Conditions producing a positive self-presentation motive • Tedeschi, Schlenker, & Bonoma (1971) P believe that others view their behavior as indicative of their true intentions, beliefs, or feelings Exp. constructed  P believe that others think their behavior is externally determined

  14. Reactivity to the Experimental Situation 3/3 Back to Chapter Contents • Implication for research • Intertreatment interaction Perception by P in different treatment groups that they can fulfill the positive self-presentation motive by responding in different ways • Intratreatment interaction Perception by P in the same treatment group that they can fulfill the positive self-presentation motive by responding in different ways

  15. Experimenter Effect1/3 Back to Chapter Contents • Experimenter Effect Fig 8.2 • Experimenter has motive of supporting the study hypothesis • Can unintentionally lead to recording errors

  16. Experimenter Effect2/3 Back to Chapter Contents • Ways experimenter may bias the study • Experimenter attributes • Biosocial attributes (e.g.) age, sex, race • Psychosocial attributes psychometrically determined characteristics (e.g.) anxiety level, hostility, authoritarianism, intelligence, dominance, warmth • Situational factors (e.g.) prior contact of P, naive or experienced • Experimenter expectancies (cont.)

  17. Experimenter Effect3/3 Back to Chapter Contents • Ways experimenter may bias the study • Experimenter attributes • Experimenter expectancies • Effect on experimenter (e.g.) recording errors (biased, but low error rate) • Effect on research participant Tab 8.2 • Mediation of expectancies • Handling in animal research • Nonverbal communication in human studies • Magnitude of expectancies—can exist in animal and human research and can be greater than the IV Fig 8.3

  18. 8.2 External Validity 1/4 Back to Chapter Contents • Definition—generalizing across people, settings, treatment variations, outcomes and times • Represents a test of interactions • Threats to external validity • Population validity • Ecological validity • Temporal validity • Treatment variation validity • Outcome validity

  19. 8.2 External Validity 2/4 Back to Chapter Contents • Population Validity The extent to which the results of a study can be generalized to the largerpopulation • Two-step inferential processFig 8.4Tab 8.3 Sample  Experimentally accessible population  Target population • College students as P less emotionally and impulsively in laboratory studies • Selection by Treatment Interaction if interaction exists  cannot generalize to the target population

  20. 8.2 External Validity 3/4 Back to Chapter Contents • Ecological Validity The extent to which the results of a study can be generalized across settings or environmentalconditions • Temporal Validity The extent to which the results of an experiment can be generalized across time • Seasonal variation A variation that occurs at regular time intervals • Cyclical variation A regular variation that occurs within people and other organisms (ex) circadian rhythm, MC

  21. 8.2 External Validity 4/4 Back to Chapter Contents • Treatment Variation Validity (IV) The generalizability of results across variation of the treatment • Outcome Validity (DV) The generalizability of results across different but related dependent variables

  22. 8.3 Cautions in Evaluating the External Validity of Experiments Back to Chapter Contents • Some studies conducted to increase knowledge and not to generalize to a real-life situation • Some studies assess a theoretical process • Moving out of the laboratory does not insure generalization

  23. 8.4 Relationship between Internal and External Validity Back to Chapter Contents • Relationship between internal and external validity—often an inverse relationship

More Related