Reliability and Validity checks

1. Reliability and Validity checks S-005

2. Checking on reliability of the data we collect • Compare over time (test-retest) • Item analysis • Internal consistency • Inter-rater agreement

3. Compare over timeTest-Retest reliability • One sample at two (or more) times • Very convincing in theory • Often hard to do in practice • Time interval? Memory effects? Special sample? • Correlation of time-1 answers with time-2 answers • Other approaches are often approximations of this idea

4. Split-half reliability • Easier than test-retest checks • Requires only one time point • Works when there is a scale or set of questions on a single topic • Divide the items into two sets (two halves) • Correlate the scores on the two halves • Often adjusted by the Spearman-Brown correction • Gives us an estimate of the test-retest reliability

5. Item-analysis approachesWhen there is a set of questions about a single topic • Examine the answers to each item • How many answered correctly? Percent correct? Item difficulty. • Or, if there is no “correct” answer, look at how the answers were distributed • Agree / neutral / disagree • Examine the “wrong” answers that are chosen • Find items that are too hard or too easy • Or those that have little variability (too boring? too trivial?) • Do you really need these? • Sometimes these are very important • Test publishers tend to delete the “easy” and “hard” items • Correlate the “item responses” with the “total responses” • High correlations indicate consistency • Low correlations indicate “different” or “weak” items • Negative correlations indicate “something interesting” • Confusing wording? The item doesn’t belong?

6. Internal consistency reliabilityWhen there is a “scale” or set of questions on a single topic • Cronbach’s coefficient alpha • a measure of “internal consistency” • Look at all of the items • Check the “average correlation” • Then adjust for the number of items • Find items that do not correlate with others • Check the item-total correlations • If low, delete these or move them elsewhere • Assess the overall internal consistency

7. Internal consistency reliabilityComparing answers from different sources • Compare similar questions that appear in different parts of the questionnaire • Compare answers from different places during an interview • Compare interview responses with questionnaire responses • Compare questionnaires with actual observations

8. Inter-rater agreement • Useful in checking on coding open-ended answers, observations, etc. • Try this on a sample or pilot study • Check the overall percent agreement • Sometimes we adjust for “chance agreement” -- Cohen’s Kappa • A very important step in lots of studies • If agreement is high, then okay to rely on one primary coder or rater • If not high, then perhaps we need more than one rater • Or perhaps we need to revise or clarify the coding rules • Then check on things again • There are often several iterations here • Keep going until the agreement is acceptable

9. Check out some examples • Bayley Scales of Infant Development • Inter-rater agreement example • Internal consistency example • Then try some clicker questions!

10. Observing students and teachers in classrooms.What type of reliability check is most important? • Inter-observer agreement (have more than one observer) • Time 1 - Time 2 (Observe at two or more times) • Consistency within the classroom sessions • Other

11. Coding transcripts from individual interviewsWhat type of reliability check is most helpful? • Have multiple transcribers • Inter-rater agreement • Internal consistency checks • Other

12. Using answers from questionnaires.What type of reliability check is most important? • Inter-rater agreement • Internal consistency checks • Item-analysis checks • Other

13. Using a mix of open-ended and closed-ended questions on a questionnaire.Why is this a good idea? • Internal consistency checks • Makes replying less boring • Terry has said this about 50 times, so it must be a good idea • Other • All of the above