Validity

1. Validity

2. Validity - definition • Intuitive: validity is the extent to which a test measures what it purports to measure. • More precisely: validity refer to the interpretation of a test score for a particular purpose or use.

3. Validity - examples • Which test should we use to diagnose whether a patient is psychotic? • A test was constructed to select people for manager positions. How to evaluate it? • How to judge whether a test measures tendency toward agressive behavior? • How evaluate whether a test is a good measure of school achievements?

4. Validity • Thus, a test might be valid for a particular aim or use. • Validity is a matter of degree: it is not all-or-none affair. • Some test are quite good for some purposes (e.g. selection), but worse for other (e.g. diagnosis).

5. Validity • Validity tells us to what extent a test will differentiate people or predict particular behavior. • Validity will also verify the definition of the construct underlying the test. • Construct might be any psychological characteristic; e.g.: depression, math-anxiety, positive affect, PTSD etc.

6. Construct underrepresntation • Validity: do the construct theory (definition) and the test overlap? • The part of the construct not covered by the test we call cosntruct underrepresentation.

7. Construct underrepresntation Test Valid Measurement Construct

8. Construct irrelevant variance Valid Measurement Construct Irrelevant Variance Construct Test

9. Example • Construct: Depression – negativethoughts, feelings and lowmotivation. 4 Items: • I oftenfeelupset. • I worry a lot. • I oftenthinkthat I amworthless. • I’musuallytensed and anxious.

10. Example… • The items don’t cover all aspects of the definition, e.g. motivation. • In practice: validity means that we conducted some empirical studies showing how the test works.

11. Types of Validity • Content validity • Criterion-related validity • Construct validity

12. Content Validity • Does the test content well represents the construct. • Difficult to cover all possible content of a construct – e.g. we can’t ask about all possible situations. • We choose a sample of items that represents e.g. extraversion.

13. Example – school test on math • Teacher taught students arithmetic, algebra and geometry. • For the final test he should include equal number of tasks reffering to the branches of mathematics he taught.

14. Content Validity • Definition of the construct determines the sample of items we should choose. • The definition should be precise. • Usually experts judge the content validity.

15. Content Validity - problems • When we have precise definition it’s easy (e.g. example on depression) • Usually psychological constructs are very broad, e.g.: • Intelligence – ability which enhance the adaptation to the environment. • School Achievement - reflect basic knowledge and understanding gained in school. Mastery of concepts, principles and knowledge.

16. Example • Construct: athletic ability • What to include to the test? • Speed, power, agility, balance, coordination • What else? • Cardiovascular endurance? Flexibility?

17. Face validity • It’s not really validity. • Face validity refers to what a test look like to measure. • A researcher might think that his test measures anxiety, but he need to support it with empirical data. • E.g. what people think that the test measures.

18. Criterion-related validity • Relationship between test score and some other criterion that is taken as an important indicator of the construct. • Two contexts for criterion-validity: • Concurrent validity • Predictive validity

19. Concurrent validity • Checking on agreement between test performance and current status on some other variable, e.g.: • comparison of IQ with students grades • Comparing scores on test measuring schizophrenia with psychiatric diagnosis • Correlating Job Performance Test with work effectiveness

20. Predictive validity • The score on test predicts status on some crietrion that will be attained in the future. • Time is important, e.g.: • From IQ test taken in high school predicting GPA at the end of freshman year in college. • Basing on personality test predicting people success at workplace.

21. Applications of crietrion-validity • External, realistic criterion defining the construct of interest • Comparison with other test. • Statistics: usually correlation or regression

22. External, realistic criterion • The external criterion is what we really like to have information about. • Getting information on the criterion may be time-consuming. • Test is easy in administration. • E.g., to screen job applicants.

23. Example 1 • Test measuring anxiety as a trait – tendency toward anxious reactions. • Research in Polish army: The test score was compared with soldiers misbehaviour (e.g. their punishments etc.) during the service • Type of validity?

24. Example 2 • Test measuring anxiety as a state – temporary experience of anxiety. • Research in Polish army: The test was adiminister at the beigining of soldiers service and compared with the lenght of their duty (whether they were dismissed). • Type of validity?

25. Correlation with other tests • Other tests which is valid might be a criterion. • Why to create a new test if there’s already one? E.g.: • New test – shorter • Some other version, like computer • Use new knowledge on the construct

26. Criterion contamination • When performance on the test influences status on the criterion. • E.g. establishing the validity of Depression Scale. • We refer to clinicians ratings on a group of patients about severity of depression • The ratings based among others on Depression Scale.

27. Construct validity • A test attempts to measure some construct. • The construct has a definition which implies some predictions. • Evidence for the construct interpretation of a test may be obtained from variety of soucres.

28. Construct validity • Experimental demonstration that a test is measuring the construct it claims to be measuring. • The possible experiments are derived from the construct definition and theory. • The theoretical predictions are tested empirically.

29. Types of construct validity • Contrasted groups • The Multitrait-Multimethod Matrix • Internal structure • Effect of Experimental Variables • Response process

30. Contrasted groups • To demonstrate that the test differentaites one group from another. • If there are significant differences – the test is valid. • The group differences are deducted from the construct theory.

31. Example 1 • We have test diagnosing psychosis. • We compare two groups: • Patients diagnosed by three independent clinicians as schizophrenic. • Healthy individuals without psychological problems, known to be functioning normally in familly and work. • Patients should have higher result.

32. Example 2 • Test measuring sensation seeking. • We assume that people choose works and activities according to their temperament. • We compare two groups: • Mountaineers • Librarians

33. Example 3 • Test of Emotional Intelligence. • We assume that EI increases with age. • We compare: • Students in high school • Students from University • Young adults after college • Other name: developmental changes.

34. The Multitrait-Multimethod Matrix • Comparing results of the test with several other tests and using several different methods. • Different methods: • Self –report • Projective technics • Obseravtion • Peer ratings

35. The Multitrait-Multimethod Matrix • Various tests: • Measuring similar construct as the test we try to valid • Measuring other constructs • Demonstrate that correlations between similar constructs are higher, regardless of the method, than between different constructs.

36. Convergent vs discriminant • Correlations between similar constructs – convergent validity. • E.g. Depression Scale correlates with depressive mood. • No relationship with othe constructs – discriminant validity. • Depression Scale show no correlation with perceived stress.

37. Example

38. Internal structure • Evaluating internal structure. • Can we obtain the number of factors that is assumed in the theory.

39. Example • Construct: Depression – negative thoughts, feelings and low motivation. • In the factor analysis we should obtain 3 factors.

40. Effect of experimental variables • We make predictions how the test score will change under given conditions. • It’s usually on the same sample of people.

41. Example 1 • Test measuring anxiety as a state. • We measure anxiety before and after exam on the same group • Expected changes: before higher than after. • What about test measuring traits?

42. Example 2 • Test of neuroticism • Natrual experiment • We measure the level of neuroticism: • before psychotherapy (1) • Immediately after the treatment (2) • A year after finishing therapy (3)

43. Expectation

44. Example 3 • Test of work satisfaction • Comparing: • Score on the test before raise of the salary • Score after the raise

45. Problems • When there is no effect: • We can’t determine whether it was the experimental manipulation or the test is bad. • Many studies verifying validity and try to report new findings at the same time.

46. Response process • Study of how examinees go about responding to a test. • E.g. when studying cognitive performance, e.g. whether subjects learn fast (increasing accuracy). • Eye movements tracking when responding to projective test (e.g. with amibigous stimulus).

47. Validity - summary • Validity – designing research that may support the theory of the construct. • It depends on the researchers creativity – we can think of any empirical way which help to verify the construct.

48. Reliability and validity relations • A test which is not reliable can’t be valid. • A test with high reliability can have low validity.

49. Assess vailidity • Test measuring criminal tendencies. • Test measuring positive mood. • Agressive behavior as a: • Trait • State • Authoritarianism