Lecture Overview: Measurement

1. Lecture Overview: Measurement • 1) Reliability of Measures • 2) Construct Validity • 3) Measurement scales

2. 1) Reliability of Measures • Reliability • The consistency or stability of a measure • Assessing a restaurant’s food • Three important variables • How many testers? (Observers) • Interrater reliability • How many different entrees? (Observations) • Internal consistency • How many times? (Occasions) • Test-retest

3. The degree to which independent raters agree on an observation Have two (or more) judges rate the same people Trained and independent raters, using a coding scheme Interrater Reliability

4. Interrater Reliability

5. Interrater Reliability

6. Internal Consistency • Internal consistency – the degree to which all specific items of a measure behave the same way • Measure the same people with multiple items • Different questions in a survey • Different behaviors in observation

7. Extraversion • I am outgoing. ____ • I am friendly. ____ • I am talkative. ____ • I am gregarious.____

8. Internal consistency • Split-half reliability – correlation of scores on one half of the test with scores on the other half • Cronbach’s alpha – the average of all possible correlations between items

9. ‘One of these things just doesn’t belong’ • One of these things is not like the others, One of these things just doesn't belong

10. Test-Retest Reliability • The degree to which a measure correlates positively with itself over time • Consistency of the measure over time • Measure the same people at two (or more) points in time • Desirable for stable traits, but not for transient states

11. The “More is Better Rule” • Reliability is likely to increase as we increase the number of… • Observers (or raters) • Observations (or items) • Occasions • Measurement error will average out

12. 2) Construct Validity • How well an operational definition represents the construct of interest • The degree to which the construct can be inferred from the operational definition of that construct

13. Indicators of Construct Validity • Face validity • Criterion validity • Predictive validity • Concurrent validity • Convergent validity • Discriminant validity

14. Face Validity • Face validity – Does the measure appear to measure the construct of interest? • Does the measure “on the face of it” look like what it’s supposed to measure? • Not necessary or sufficient for a good measure

15. Predictive Validity • Predictive validity – Is the measure associated with variables it should theoretically predict? • LSAT – Law school performance • Self-esteem – Depression • Shyness – Social anxiety

16. Concurrent Validity • Concurrent validity – Does the measure differ between groups it ought to differ between? • Also called “known groups validity” • E.g., clinically depressed versus non-depressed groups

17. Convergent Validity • Convergent validity – Is the measure associated with other established measures of the same construct? • Self-report - Observations • Physiological measure - Self-report • Self-report 1 – Self-report 2

18. Discriminant Validity • Discriminant validity – Is the measure NOT associated with measures of other constructs? • Self-esteem scores not associated with locus of control scores • Problem solving knowledge not associated with factual knowledge

19. Reliable and Valid Reliable, not Valid Not Reliable, not Valid Measurement Reliability & Validity • Reliability: Is the measure consistent? • Validity: Does the measure adequately reflect the construct of interest?

20. Relationship between Reliability and Validity • Can be reliable but not valid • To be valid it must be reliable • But reliability is not sole condition for validity • Both reliability and validity are necessary for accurate measurement in a research study.

21. Measurement Scales • Nominal scales • Ordinal scales • Interval scales • Ratio scales

22. Nominal Scales • AKA Categorical scales • No numerical/quantitative properties. Categories or group simply differ from one another • Examples: • Men or women • Right or left handed • Catholic, Protestant, Jewish, Hindu, Buddhist… • Numbers on basketball jerseys • Zip codes

23. Ordinal Scales • Allow us to rank order the levels of the variables being studied • Examples • Social class • lower class, working class, middle class, and upper class • College football standings • Letterman’s Top Ten

24. Top Ten Bush Goals For His Second Term 10. Fewer idiotic remarks; more hilarious pratfalls. 9. Add mother Barbara to Mount Rushmore. 8. Combine Nebraska and Kansas into new state: Nebransas. 7. Spice up boring state dinners with tasty fish sticks! 6. Improve communication skills from poor to fair. 5. Catch up on his "Smokey And The Bandit" collection. 4. Get Ray Stevens to write some funny lyrics for "Hail To The Chief" 3. Ride every roller coaster in the country. 2. Install remote-activated button in Oval Office so he can blow stuff up right from his desk! 1. Begin vote-rigging process for Jeb's White House run in 2008.

25. Interval Scales • The difference between the numbers on the scale is meaningful • Scores separated by equal intervals • Examples • Temperature (Fahrenheit or Celsius) • Scores on personality measure

26. Ratio Scales • Scores separated by equal intervals and there is an absolute zero • Examples • Length • Weight • Time • Number of responses

27. Scales of Measurement

28. Concept Check • Which scale of measurement best describes the following: • Telephone numbers • Distances from Budapest to cities in the US • Scores on an extraversion personality assessment • Ranking of basketball teams in the Big Ten