Discussion Overview: Measurement

1. Discussion Overview: Measurement • I) Reliability of Measures • II) Construct Validity • III) Measurement scales

2. I) Reliability of Measures • Reliability • The consistency or stability of a measure • Assessing a restaurant’s food • Three important variables • How many testers? (Observers) • Interrater reliability • How many different entrees? (Observations) • Internal consistency • How many times? (Occasions) • Test-retest

3. The degree to which independent raters agree on an observation Have two (or more) judges rate the same people Trained and independent raters, using a coding scheme Interrater Reliability

4. Interrater Reliability

5. Interrater Reliability

6. Internal Consistency • Internal consistency – the degree to which all specific items of a measure behave the same way • Measure the same people with multiple items • Different questions in a survey • Different behaviors in observation

7. Extraversion • I am outgoing. ____ • I am friendly. ____ • I am talkative. ____ • I am gregarious.____

8. Internal consistency • Split-half reliability – correlation of scores on one half of the test with scores on the other half • Cronbach’s alpha – the average of all possible correlations between items

9. ‘One of these things just doesn’t belong’ • One of these things is not like the others, One of these things just doesn't belong

10. Test-Retest Reliability • The degree to which a measure correlates positively with itself over time • Consistency of the measure over time • Measure the same people at two (or more) points in time • Desirable for stable traits, but not for transient states

11. The “More is Better Rule” • Reliability is likely to increase as we increase the number of… • Observers (or raters) • Observations (or items) • Occasions • Measurement error will average out

12. II) Construct Validity • How well an operational definition represents the construct of interest • The degree to which the construct can be inferred from the operational definition of that construct

13. Indicators of Construct Validity • Face validity • Criterion validity • Predictive validity • Concurrent validity • Convergent validity • Discriminant validity

14. Face Validity • Face validity – Does the measure appear to measure the construct of interest? • Does the measure “on the face of it” look like what it’s supposed to measure? • Not necessary or sufficient for a good measure

15. Predictive Validity • Predictive validity – Is the measure associated with variables it should theoretically predict? • LSAT – Law school performance • Self-esteem – Depression • Shyness – Social anxiety

16. Concurrent Validity • Concurrent validity – Does the measure differ between groups it ought to differ between? • Also called “known groups validity” • E.g., clinically depressed versus non-depressed groups

17. Convergent Validity • Convergent validity – Is the measure associated with other established measures of the same construct? • Self-report - Observations • Physiological measure - Self-report • Self-report 1 – Self-report 2

18. Discriminant Validity • Discriminant validity – Is the measure NOT associated with measures of other constructs? • Self-esteem scores not associated with locus of control scores • Problem solving knowledge not associated with factual knowledge

19. Measurement Reliability & Validity • Reliability: Is the measure consistent? • Validity: Does the measure adequately reflect the construct of interest? Reliable and Valid Reliable, not Valid Not Reliable, not Valid

20. Relationship between Reliability and Validity • Can be reliable but not valid • To be valid it must be reliable • But reliability is not sole condition for validity • Both reliability and validity are necessary for accurate measurement in a research study.

21. Measurement Scales • Nominal scales • Ordinal scales • Interval scales • Ratio scales

22. Nominal Scales • AKA Categorical scales • No numerical/quantitative properties. Categories or group simply differ from one another • Examples: • Men or women • Right or left handed • Catholic, Protestant, Jewish, Hindu, Buddhist… • Numbers on basketball jerseys • Zip codes

23. Ordinal Scales • Allow us to rank order the levels of the variables being studied • Examples • Social class • lower class, working class, middle class, and upper class • College football standings • Letterman’s Top Ten

24. Top Ten Questions to ask Yourself Before Eating Spinach? • 10. Was my spinach properly sprayed with Lysol? • 9. Isn't it still safer than eating a New York City hot dog? • 8. So all those years my mom made me eat spinach, she was trying to kill me? • 7. Is this the right side dish for my Mad Cow burger? • 6. Are my papers in order? • 5. If I get sick, will my wife TiVo Ventriloquist Week on the Late Show? • 4. Should I also avoid kale? • 3. If I'm going to eat something deadly, shouldn't it be delicious Pop-Tarts? • 2. What would Popeye do? • 1. Do I really want my obituary to read: "Man Dies A La Florentine?"

25. Interval Scales • The difference between the numbers on the scale is meaningful • Scores separated by equal intervals • Examples • Temperature (Fahrenheit or Celsius) • Scores on personality measure

26. Ratio Scales • Scores separated by equal intervals and there is an absolute zero • Examples • Length • Weight • Time • Number of responses

27. Scales of Measurement

28. Concept Check • Which scale of measurement best describes the following: • Telephone numbers • Distances from Budapest to cities in the US • Scores on an extraversion personality assessment • Ranking of basketball teams in the Big Ten