Reliability Introduction to Communication Research School of Communication Studies

1. Reliability Introduction to Communication Research School of Communication Studies James Madison University Dr. Michael Smilowitz

2. Reliability and Validity Why bother? Remember the principle of isomorphism? “To what extent do the data yielded by the measurement schemes accurately represent the nature and structure of the phenomenon being measured” (Smith, 1988, p. 47)

3. Reliability and Validity Important also to Replicability If a measuring device is not consistent, subject to error, or does not truly measure the constructs under investigation, then it is not likely that subsequent research will yield similar results.

4. Reliability and Validity The Basics Reliability: An evaluation of the internal consistency of the measuring devices. Does the instrument measure the same way every time? Validity: An evaluation of the measures relationship to external characteristics. Does the measuring device really measures what it claims to measure?

5. Reliability What’s the point? Repeated measurements by the same device should yield (roughly) the same set of responses. Why “roughly” ? Because error is expected when measuring human behavior.

6. Reliability Sources of error: 1. Chance. 2. Subject fatigue. 3. Subject carelessness. 4. Fluctuations in memory. 5. Subject familiarity with the measuring device.

7. Reliability Sources of error: 6. Measurement conditions (room too hot, too close to lunch, noisy). 7. Confusing or vague language in the test. 8. Items not relevant to subject’s experience.

8. Reliability Assessing Reliability Various methods can be used, but all depend on statistically analyzing the results to determine the reliability coefficient. A correlation statistic which measures the amount of association or coincidence of things.

9. Reliability Reliability coefficients -- as correlation statistics -- range in value between 0 (no correlation) and 1 (perfect correlation). Why bother? Looking at the reliability coefficient value tells you something about the usefulness of the measuring scheme used by the researchers. There are certain values, set by convention, that are used to evaluate the measuring schemes reliability.

10. Reliability A really good measure has a coefficient of .9 or higher. A good measure is between .8 and .89. A fairmeasure is between .7 and .79. A flop is less than .7, for communication journals. Some journals sometimes accept values above .6.

11. Reliability Test-retest: (sometimes called the matching-pairs procedure) Involves giving the measure twice to the same group of people. The reliability coefficient reports the extent to which the two sets of scores are consistent.

12. Reliability Test-retest method: This method is relatively straight forward in its logic and is relatively easy to employ. Disadvantages: 1. Time-consuming: two administrations are necessary. 2. Likely to “sensitize” people to the second administration, prompting them to try and remember their first response and respond in the same way again. Leads to an “inflated” estimate of reliability. 3. People’s views may change between tests. To overcome the “recall” problem from above, the second test is administered after some period of time. In the interim, the subjects may have changed. Leads to an “underestimate” of reliability. The amount of ideal time for overcoming problems 2 and 3 is a judgment call.

13. Reliability Alternate forms: Involves administration of two parallel versions of the same instrument to the same group of subjects. The researcher produces a large number of items to measure the variable (s) of interest. Then items are selected to produce two nearly identical instruments. The advantage is in overcoming the “time” problem of the test-retest. The disadvantages: 1. Very difficult to produce two truly parallel versions of the same instrument. 2. Subjects may become aware of the similarity, causing over-estimation of the instrument’s reliability.

14. Reliability Split half approach (an internal consistency method): The instrument is administered to the same group of subjects at one time. After administration, the research divides the test into two parts, and then checks the consistency between the two scores from each part. One technique is to compare first half with second half. (Subjects may experience fatigue during second half, so not usually a good idea. Odd-even a better technique.) Cronbach’s Alpha - most common technique...thanks to the computer. 1. The computerized analysis randomly selects multiple pairs of subsets from an instrument; 2. correlates each pair of scores; 3. and then uses the composite correlation between all the paired subsets as an index of the total instruments internal consistency.

15. Reliability Item-to-total (another internal method): This procedure computes the reliability of any one particular item against the entire test. There should be a high correlation between any one item and the score on the entire test. Most appropriate when scoring “correct” or “incorrect” answers. The K-R 20 is computerized statistical routine for doing this type of reliability (Kuder-Richardson formula 20)

16. Reliability Intercoder Reliability: Often seen in communication research. Used to determine the consistency among multiple observers who are categorizing the same phenomenon. The Geutzkow Estimate is often used for assessing reliability.