Data Collection: Experimentation, Survey and Field Research

1. Data Collection: Experimentation, Survey and Field Research PO 390W: Research Design for Political Science

2. The Experiment • The experiment is the most systematic and fruitful form of data collection in the social sciences • Allows for maximum control over extraneous variables through the employment of control and stimulus groups • All of the remaining forms of data collection seek to emulate the experimental method • Throughout, you should be looking to determine how the experiment relates to your likely choice of data collection and analysis • We will see, however, that replicating experimental conditions in the social sciences is often quite difficult

3. Basic Experimental Requirements • Random Assignment • Controlling for extraneous variables by allowing each subject to have an equal chance of being in either group (absence of selection bias) • Independent Variable Manipulation • Systematic inclusion/exclusion of key independent variable to various groups • Consequent Dependent Variable Measurement • Ascertaining the “reading” of the DV given the presence or absence of the IV • Comparison/Control Group • Omitting the key IV for the purposes of comparing eventual results with stimulus group • Constancy of Conditions Across Groups (aside from experimental manipulation) • Maintaining strict exclusion of extraneous variable influence

4. Problems in Conducting Social Scientific Experiments • Much of social science necessitates field experimentation (natural setting, but incomplete control) • Must try to minimize disruption of normal behavior • Immense potential ethical problems with trying to ensure control in field experiments

5. Internal and External Validity • For the experiment (as for all collection techniques), scientists must do their best to maximize internal and external validity • Internal validity: When extraneous variables can be completely ruled out as the cause of a particular experimental outcome (importance of internal control) • External Validity: When outcomes are generalizable to contexts outside of the particular experiment (importance of sound sampling and capacity to replicate) • Often, the analytical methods chosen by social scientists involves an implicit tradeoff between these two (e.g., quantitative vs. qualitative); however, experiments and other collection methods should seek to accomplish both (How? Look at the Mythbusters example)

6. Threats to Internal Validity in Experiments • History: Environmental changes during experiment that affect outcome • Maturation: Subject changes during experiment that effect outcome • Testing Effects: Changes in what is being measures that are brought about by reactions to measurement process • Instrumentation: Unwanted changes in characteristics of measuring instrument/ procedure (“learning”) • “Regression to the Mean”: Normalization of extreme scorers (“learning”) • *Selection: Systematic differences in composition of control and experimental groups • Differential Attrition: Loss of subjects in experiment, resulting in homogeneity of remaining subjects (obscuring difference between control and stimulus groups)

7. Survey Research • Survey research is perhaps the most well-developed data collection method in certain areas of political inquiry (especially public opinion and election studies) • Question: How can a pollster administer surveys to 1500 Americans, and yet get an accurate reading on the opinions of all Americans? • The answer lies in the methods employed by survey researchers, all of which seek (in some way) to emulate the experimental method

8. The Basics of Surveying • Large number of respondents chosen through probability sampling • The more respondents, the better; but: • Should fulfill the requirements of the central limit theorem (asymptotic properties) • Should be representative of all sectors of the group for which inferences are to be drawn • Maximum external validity • Systematic questionnaire/interview procedures • Questions must seek to elicit intended responses without “leading” or otherwise biasing the response • Attractiveness of various levels of structure (sensitivity of issues; generality of desired response) • Numerical coding and quantitative analysis of responses • Ultimate goal is to collect data and reach probabilistic conclusions (i.e., percentiles) • Secondary analysis: Aggregation of surveys; analyses of extant surveys for other research questions

9. The Limitations of the Survey • Ethical considerations • Can’t obtain information about respondents from other sources • Difficult to establish causal relationships • Variable measurement at single point • Difficulty in effecting significant control • Limited adaptability (standardized nature) • “Reactivity” (e.g., socially acceptable answers) and self-censoring in responses

10. Field Research • Field research involves the effort to gain understanding by observing naturally-occurring events • Importance of “detached observation” • Getting an “insider’s view of reality” • Should strive to maintain unobtrusiveness • Used to conduct case studies of communities, groups, occupations, or everyday life (data collection for hypothesis testing) • Quite useful when necessary to preserve “whole” events, to immerse oneself in a situation, or when subjects are unable/unwilling to participate in surveys

11. The Process of Field Research • Selecting Topic/Setting • Topic should be doable/interesting • Setting should be close at hand where researcher can readily fit in • Access • Importance of “Gatekeepers” in group settings • Sometimes need to become accepted member of group or develop “key informants” • Presentation and Establishment of Roles • Covert research vs. group membership (peripheral, active, or complete) • Methodological Empathy • Note Taking • “Field jottings”; daily field notes; organizing information • Analysis • “Grounded Theory” development; coding in quantitative situations • Reporting • Identifying larger meaning of findings

12. The Limitations of Field Research • While field research can be an extremely useful tool for gaining first-hand cultural knowledge, it entails some difficulties • Costly (time and money) • Situation of interest may be fleeting • Requires excellent observational/interpretive skills • Risk of “going native” in participant observation • Extreme limitations on external validity (esp. given that subjects often selected on basis of availability or convenience)