Test Construction and Measurement

1. Test Construction and Measurement

2. An Experiment • Researcher gave students the Diagnostic Inventory Blank • Hobbies, reading interests, secret hopes and ambitions • Then gave students typed descriptions of their personalities • Asked students to rate how well personality sketch described them

3. Sample Personality Description • You have a need for other people to like and admire you • You have a tendency to be critical of yourself • While you have some personality weaknesses, you are generally able to compensate for them

4. Disciplined and self-controlled outside, you tend to be worrisome and insecure inside • You pride yourself as an independent thinker and do not accept others’ statements without proof • At times you are extraverted and sociable, while at other times you are introverted and reserved

5. Result • Almost all students were very impressed with how well DIB described them • Rated DIB as very accurate personality test

6. Problem • Every student was given exactly the same personality description

7. The Lesson • Beware of the Barnum effect • Tendency of people to see vague, universal statements as descriptive of themselves

8. Major Point • Real psychological measurement is a complicated and difficult process

9. A Preview • Correlation • Steps in constructing a psychological test • Reliability and validity • Factor analysis

10. Correlational Research • Focuses on relationships among variables • Changes in one variable are associated with changes in another variable

11. Correlation Coefficient • Number which expresses the direction and strength of the relationship between 2 variables • Ranges from -1 to 1 • Index of the degree to which scores on one measure can be used to predict scores on a 2nd measure

12. Direction • Indicated by + or - sign (slope) • Positive correlation • as one variable goes up, so does the other • Negative correlation • As one variable goes up, the other goes down

13. Strength • Indicated by absolute value • perfect positive relationship = 1 • perfect negative relationship = -1 • no relationship = 0

14. Percent of Variance • Percent of variance in “measure A” that can be accounted for “measure B” • square correlation coefficient and multiply by 100 • Correlation of .50 means we can account for 25% of variance

15. Causality • Correlation just tells you that 2 variables are related • Can’t make causal interpretations

16. Fact: Time spent on the internet is positively correlated with depression • Possible interpretations • Spending lots of time on internet causes depression • Being depressed causes you to spend lots of time on internet • Some third variable, such as living by one’s self, causes both

17. Steps in Psychological Test Construction

18. Major Point • It is difficult, but not impossible, to construct a meaningful psychological test

19. Steps in Test Construction 1. Decide what to measure • Identify construct • Idea that helps us makes sense of world around us • Not directly observable • Examples: intelligence, extraversion, racism, pessimism, creativity

20. Steps (continued) 2. Develop a set of items/questions • Search literature • Get experts or lay people to tell us what construct means to them

21. Steps (continued) 3. Get sample of people to answer items • From population you want to use test for

22. Steps (continued) 4. Evaluate each item • Correlate each item with mean of whole set • Correlate each item with item directly assessing self-reported racism • drop bad items

23. Steps (continued) 5. Select a set of items for further study • Want normal distribution • Drop high YES and high NO items

24. Steps (continued) 6. Assess reliability of entire test • Consistency of measurement • 3 major types

25. Reliability 1. Inter-rater: • Extent to which different people scoring same test get same result • Correlate set of tests scored by one rater with same set of tests scored by different rater

26. Reliability 2. Test-retest: • Extent to which people get same results if take test again • Subjects take test twice. Correlate set of time 1 scores with time 2 scores

27. Reliability 3. Internal consistency: • Split-half: correlation between one half of test and other half • Coefficient alpha: average of all possible split-half reliabilities

28. How high should reliability correlations be? • Expect r = .80 or better

29. Factors that influence reliability • Clarity of items • Motivation of test taker • Number of items

30. Steps (continued) 7. Assess validity of entire test • Extent to which test measures what it is supposed to measure • Face validity not sufficient • Do series of validity studies

31. Ways to Measure Validity 1. Criterion validity • Correlation between test and concrete, directly observable criterion • Example: correlate self-report of weight with actual weight on scale

32. Ways to Measure Validity (continued) 2. Content validity • Adequate coverage of target domain • Example: test of chapters 1-4 which only covers chapter 2 and 3 lacks content validity

33. Ways to Measure Validity (continued) 3. Convergent validity • Agreement among alternative measures of same construct • Example: correlation between ACT and SAT 4. Discriminant validity • Lack of correlation between tests that are intended to measure different constructs • Example: expect low correlation between ACT and test of aggression

34. Threats to Validity Response tendency • Assign numbers to items for reasons that have little to do with the construct the item is intended to measure

35. Response Tendencies • Extremity tendency • Use end of scales • Acquiescence tendency • Agree with questions • Social desirability • Answer in a way that makes you look good

36. Factor Analysis • Statistical technique that examines pattern of correlations among multiple tests or items • Tests or items that correlate strongly with one another are considered to represent a common, underlying factor

37. Interpreting Factor Analysis • Each item has a factor loading: correlation between item and factor • Marker variable • item that has high factor loading (correlation) with given factor • closely related to meaning of factor • Blend • item that loads moderately high on more than one factor • not a pure measure of factor, related to two or more factors