Download
1 / 43
430 likes | 554 Views
INF 397C Introduction to Research in Library and Information Science Fall, 2003 Day 5. The Scientific Method. More than anything else, scientists are skeptical.
E N D
INF 397CIntroduction to Research in Library and Information ScienceFall, 2003Day 5
More than anything else, scientists are skeptical. • P. 28: Scientific skepticism is a gullible public’s defense against charlatans and others who would sell them ineffective medicines and cures, impossible schemes to get rich, and supernatural explanations for natural phenomena.”
Research Methods S, Z, & Z, Chapters 1, 2, 3, 7, 8 Researchers are . . . • like detectives – gather evidence, develop a theory. • Like judges – decide if evidence meets scientific standards. • Like juries – decide if evidence is “beyond a reasonable doubt.”
Science . . . • . . . Is a cumulative affair. Current research builds on previous research. • Scientific Method: • Empirical (acquires new knowledge via direct observation and experimentation) • Systematic, controlled observations. • Unbiased, objective. • Operational definitions. • Valid, reliable, testable, critical, skeptical.
CONTROL • . . . Is the essential ingredient of science, distinguishing it from nonscientific procedures. • The scientist, the experimenter, manipulates the Independent Variable (IV – “treatment – at least two levels – “experimental and control conditions”) and controls other variables.
More control • After manipulating the IV (because the experimenter is independent – he/she decides what to do) . . . • He/she measures the effect on the Dependent Variable (what is measured – it depends on the IV).
Key Distinction • IV vs. Individual Differences variable • The scientist MANIPULATES an IV, but SELECTS an Individual Differences variable (or “subject” variable). • Can’t manipulate a subject variable. • “Select a sample. Have half of ‘em get a divorce.”
Operational Definitions • Explains a concept solely in terms of the operations used to produce and measure it. • Bad: “Smart people.” • Good: “People with an IQ over 120.” • Bad: “People with long index fingers.” • Good: “People with index fingers at least 7.2 cm.” • Bad: Ugly guys. • Good: “Guys rated as ‘ugly’ by at least 50% of the respondents.”
Validity and Reliability • Validity: the “truthfulness” of a measure. Are you really measuring what you claim to measure? “The validity of a measure is supported to the extent that people do as well on it as they do on independent measures that are presumed to measure the same concept.” • Reliability: a measure’s consistency. • A measure can be reliable without being valid, but not vice versa.
Theory and Hypothesis • Theory: a logically organized set of propositions (claims, statements, assertions) that serves to define events (concepts), describe relationships among these events, and explain their occurrence. • Theories organize our knowledge and guide our research • Hypothesis: A tentative explanation. • A scientific hypothesis is TESTABLE.
Goals of Scientific Method • Description • Nomothetic approach – establish broad generalizations and general laws that apply to a diverse population • Versus idiographic approach – interested in the individual, their uniqueness (e.g., case studies) • Prediction • Correlational study – when scores on one variable can be used to predict scores on a second variable. (Doesn’t necessarily tell you “why.”) • Understanding – con’t. on next page • Creating change • Applied research
Understanding • Three important conditions for making a causal inference: • Covariation of events. (IV changes, and the DV changes.) • A time-order relationship. (First the scientist changes the IV – then there’s a change in the DV.) • The elimination of plausible alternative causes.
Confounding • When two potentially effective IVs are allowed to covary simultaneously. • Poor control!
Intervening Variables • Link the IV and the DV, and are used to explain why they are connected. • Here’s an interesting question: WHY did the authors put this HERE in the chapter? • Because intervening variables are important in theories.
A bit more about theories • Good theories provide “precision of prediction” • The “rule of parsimony” is followed • The simplest alternative explanations are accepted • A good scientific theory passes the most rigorous tests • Testing will be more informative when you try to DISPROVE (falsify) a theory
Populations and Samples • Population: the set of all cases of interest • Sample: Subset of all the population that we choose to study.
Ch. 3 -- Ethics • Read the chapter. • Understand informed consent, p. 57 – a person’s expressed willingness to participate in a research project, based on a clear understanding of the nature of the research, the consequences of declining, and other factors that might influence the decision. • Odd quote, p. 69 – Debriefing should be informal and indirect. • Know that UT has an IRB: http://www.utexas.edu/research/rsc/humanresearch/
Ch. 7 – Independent Groups Design • Description and Prediction are crucial to the scientific study of behavior, but they’re not sufficient for understanding the causes. We need to know WHY. • Best way to answer this question is with the experimental method. • “The special strength of the experimental method is that it is especially effective for establishing cause-and-effect relationships.”
Good Paragraph • P. 196, para. 2 – Discusses how experimental methods and descriptive methods aren’t all THAT different – well, they’re different, but related. And often used together.
Good page – P. 197 • Why we conduct experiments • If results of an experiment (a well-run experiment!) are consistent with theory, we say we’ve supported the theory. (NOT that it is “right.”) • Otherwise, we modify the theory. • Testing hypotheses and revising theories based on the outcomes of experiments – the long process of science.
Logic of Experimental Research • Researchers manipulate an independent variable in an experiment to observe the effect on behavior, as assessed by the dependent variable.
Independent Groups Design • Each group represents a different condition as defined by the independent variable.
Random . . . • Random Selection vs. Random Assignment • Random Selection = every member of the population has an equal chance of being selected for the sample. • Random Assignment = every member of the sample has an equal chance of being placed in the experimental group or the control group. • Random assignment allows for individual differences among test participants to be averaged out.
Let’s step back a minute • An experiment is personkind’s way of asking nature a question. • I want to know if one variable (factor, event, thing) has an effect on another variable – does the IV influence the DV? • I manipulate some variables (IVs), control other variables, and count on random selection to wash out the effects of all the rest of the variables.
Block Randomization • Another way to wash-out error variance. • Assign subjects to blocks of subjects, and have whole blocks see certain conditions. • (Very squirrelly description in the book.)
Challenges to Internal Validity • Testing intact groups. (Why is the group a group? Might be some systematic differences.) • Extraneous variables. (Balance ‘em.) (E.g., experimenter). • Subject loss • Mechanical loss, OK. • Select loss, not OK. • Demand characteristics (cues and other info participants pick up on) – use a placebo, and double-blind procedure • Experimenter effects – use double-blind procedure
Role of Data Analysis in Exps. • Primary goal of data analysis is to determine if our observations support a claim about behavior. Is that difference really different? • We want to draw conclusions about populations, not just the sample. • Two ways – stat and replication.
Two methods of making inferences • Null hypothesis testing • Assume IV has no effect on DV; differences we obtain are just by chance (error variance) • If the difference is unlikely enough to happen by chance (and “enough” tends to be p < .05), then we say there’s a true difference. • Confidence intervals • We compute a confidence interval for the “true” population mean, from sample data. (95% level, usually.) • If two groups’ confidence intervals don’t overlap, we say (we INFER) there’s a true difference.
What data can’t tell us • Proper use of inferential statistics is NOT the whole answer. • Scientist could have done a trivial experiment. • Also, study could have been confounded. • Also, could by chance find this difference. (Type I and Type II errors – hit this for real in week 5.)
This is HUGE. • When we get a NONsignificant difference, or when the confidence intervals DO overlap, we do NOT say that we ACCEPT the null hypothesis. • Hinton, p. 37 – “On this evidence I accept the null hypothesis and say that we have not found evidence to support Peter’s view of hothousing.” • We just cannot reject it at this time. • We have insufficient evidence to infer an effect of the IV on the DV.
Notice • Many things influence how easy or hard it is to discover a difference. • How big the real difference is. • How much variability there is in the population distribution(s). • How much error variance there is. • Let’s talk about variance.
Sources of variance • Systematic vs. Error • Real differences • Error variance • What would happen to the standard deviation if our measurement apparatus was a little inconsistent? • There are OTHER sources of error variance, and the whole point of experimental design is to try to minimize ‘em. Get this: The more error variance, the harder for real differences to “shine through.”
One way to reduce the error variance • Matched groups design • If there’s some variable that you think MIGHT cause some variance, • Pre-test subjects on some matching test that equates the groups on a dimension that is relevant to the outcome of the experiment. (Must have a good matching test.) • Then assign matched groups. This way the groups will be similar on this one important variable. • STILL use random assignment WITHIN the groups. • Good when there are a small number of possible test subjects.
Another design • Natural Groups design • Based on subject (or individual differences) variables. • Selected, not manipulated. • Remember: This will give us description, and prediction, but not understanding (cause and effect).
We’ve been talking about . . . • Making two groups comparable, so that the ONLY systematic difference is the IV. • CONTROL some variables. • Match on some. • Use random selection to wash out the effects of the others. • What would be the best possible match for one subject, or one group of subjects?
Themselves! • When each test subject is his/her own control, then that’s called a • Repeated measures design, or a • Within-subjects design. (And the random groups design is called a “between subjects” design.)
Repeated Measures • If each subject serves as his/her own control, then we don’t have to worry about individual differences, across experimental and control conditions. • EXCEPT for newly introduced sources of variance – order effects: • Practice effects • Fatigue effects
Counterbalancing • ABBA • Used to overcome order effects. • Assumes practice/fatigue effects are linear. • Some incomplete counterbalancing ideas are offered in the text.
Which method when? • Some questions DO lend themselves to repeated measures (within-subjects) design • Can people read faster in condition A or condition B? • Is memorability improved if words are grouped in this way or that? • Some questions do NOT lend themselves to repeated measures design • Do these instructions help people solve a particular puzzle? • Does this drug reduce cholesterol?
Hinton typo • P. 62, para. 1: “. . . population standard deviation, µ, divided by . . . .”
Midterm • Emphasize • How to lie with statistics – concepts • To know a fly – concepts • SZ&Z – Ch. 1, 2, 7, 8 • Hinton – Ch. 1, 2, 3, 4, 5 • De-emphasize • SZ&Z – Ch. 3 • Other readings • Totally ignore for now • SZ&Z – Ch. 14 • Hinton – Ch. 6, 7, 8
