Experimental Design, Statistical Analysis

1. Experimental Design, Statistical Analysis CSCI 4800/6800 University of Georgia March/April 2006 Eileen Kraemer

2. Terminology • empirical study • based on observations and measurements • probabilistic • based on probabilities, inferences • causal • based on cause-effect relationships

3. Types of research questions • Descriptive. • designed primarily to describe what is going on or what exists. • For example, a study in which you observe and note the current practice of users in performing a task of interest • Relational. • designed to look at the relationships between two or more variables. • For example, a study in which you look at the relationship between user’s preferred background color and font type • Causal • a study is designed to determine whether one or more variables (e.g., a program or treatment variable) causes or affects one or more outcome variables. • For example, a study of the effect of font size on time-to-complete or error rates in a task of interest

4. Time in research • cross-sectional versus longitudinal studies. • A cross-sectional study is one that takes place at a single point in time. • A longitudinal study is one that takes place over time -- we have at least two (and often more) waves of measurement in a longitudinal design. • repeated measures - two or a few waves of measurement • time series - many waves of measurement over time • Analysis considerations: Time series analysis requires that you have at least twenty or so observations. Repeated measures analyses (like repeated measures ANOVA) aren't often used with as many as twenty waves of measurement.

5. Relationships between variables • correlational v. causal • positive • negative (inverse)

6. Variables • variable • any entity that can take on different values. • attribute • a specific value of a variable. • independent variable • the thing you change • dependent variable • the thing you expect to change as a result of your manipulation of the independent variable – the thing you measure • Example: I perform an experiment in which I give apply fertilizer at concentrations of 5%, 10%, and 20% to some number of otherwise identical plants. I measure the growth of the plants. The concentration of fertilizer is the independent variable, the height of the plant is the dependent variable.

7. Well-chosen variables/attributes • exhaustive - should include all possible responses (all possible values of plant height, for example) • mutually exclusive - no response should be able to have two attributes simultaneously (2-4 inches and 4-6 inches shouldn’t be two possible attributes – exactly 4 in. would fall into two categories)

8. Hypotheses • hypothesis • a specific statement of prediction • For the hypothetical-deductive model : two hypotheses • one that describes your prediction (alternative hypothesis) (H1) • one that describes all the other possible outcomes (null hypothesis) (H0) • One-tailed v. two-tailed • One-tailed – the prediction specifies a direction • H1: increase fertilizer application will increase the height of the plant (H1) • H0: increased fertilizer application will not increase the height of the plant • Two-tailed – the prediction does not specify a direction • H1: increased fertilizer application will affect the height of the plant • H0: increased fertilizer application will not affect the height of the plant • In either case, the goal of testing and analysis is to accept one hypothesis and reject the other

9. Sampling • the process of selecting units from a population of interest in a way that permits us to study those samples and then generalize our results back to the population • external validity • the degree to which study conclusions would hold for other experimenters in other similar studies • the approximate truth of the inferences and conclusions that result from the study

10. Sampling Model for generalization • identify the population you wish to generalize to • draw a fair sample of that population • conduct research with sample • generalize back to population from which sample is drawn

11. Sampling terminology • theoretical population • group you wish to generalize to • accessible population • subset of that population that is accessible to the experimenter • sampling frame • list of the accessible population you’ll draw your sample from • sample • group selected to be in your study

12. Selection, assignment • selection • how you draw your sample from a population • assignment • how you assign members of the sample to groups in your study • Goal: avoid systematic error, bias

13. Statistical sampling terminology, continued • response • specific measurement value that a sampling unit provides • statistic • calculated across the response from the sample • parameter • calculated across the population • a statistic provides an estimate of a parameter

14. Terminology, continued • sampling distribution - the distribution of an infinite number of samples of the same size as the sample in our study • standard deviation • the spread of scores around the average in a single sample • standard error, sampling error • the spread of averages around the average of averages ina sample distribution • indicates the precision of our statistical estimate • calculated based on standard deviation of sample • larger sample size -> smaller standard error

15. Variance • variance – a measure of how spread out a distribution is. The average squared deviation of each value from its mean. Example: values = {1, 2, 3} • 2 =((1-2)2 + (2-2)2 + (3-2)2)/3 = 0.667 • for a population: • 2 = Σ(X – μ)2 / N • μ = mean, N = number of samples • for a sample: • s2 = Σ (X – M)2/(N-1) • M = mean of the sample

16. Standard deviation • for a population: •  = sqrt(2) • for a sample: • s = sqrt(s2)

17. Normal distribution: 68-95-99 rule • normal dist = bell-shaped curve • 68% of cases fall w/in one S.D. • 95% of cases fall w/in two S.D. • 99% fall w/in three S.D.

18. Basic sampling concept: • If we had a sampling distribution, we would be able to predict the 68, 95 and 99% confidence intervals for where the population parameter should be! • We don't actually have the sampling distribution • We do have the distribution for the sample itself. • From that distribution we can estimate the standard error (the sampling error) because it is based on the standard deviation and we have that. • We still don't actually know the population parameter value, but we can use our best estimate for that -- the sample statistic. • Now, we can use • the mean of our sample as the mean of the sampling distribution • standard deviation and sample size to estimate the standard error • SE = s / sqrt(N) • to estimate confidence intervals for the population parameter.

19. An example: • We draw a 100 member sample from a population • M = 3.75 • s = 0.25 • SE = 0.25 / 10 = 0.025 • p(3.725 < μ < 3.775) = 0.68 • p(3.700 < μ < 3.800) = 0.95 • p(3.675 < μ < 3.825) = 0.99 • these are known as confidence intervals