Chapter 6 Validity §1 Basic Concepts of Validity

1. Chapter 6 Validity§1Basic Concepts of Validity • What is the Validity? • Interpretation • The validity of a test concerns what the test measure and how well it does so. • —Anne Anastasi

2. It tell us what can be inferred from test scores —Anne Anastasi

3. Figure6.1One Funny Picture

4. Validity can be defined as the agreement between a test score or measure and the quality it is believed to measure. — Robert M. Kaplan Dennis P. Saccuzzo Does the test measure what it is supposed to measure?

5. Validity is the evidence for inferences made about a test score. —AERA, APA, NCME STANDARS FOR EDUCATIONAL AND PSYCHOLOGICAL TESTING

6. Validity effected by random and systematic errors. Random errors and systematic errors both reduce the accuracy of the test.

7. Mathematic Definition of Validity Validity coefficientis the ratio of The variance concerned to the trait measured to observed score variance. (6.1)

8. Comparing Validity with Reliability The reliability of test is low, usually, the validity is low too; The reliability of test is high, the validity isn’t necessarily high. Figure 6.2 Components of the Variance of Observed Scores Reliability is a necessary premise for validity and validity represents the ultimate purpose of the test.

9. Three Types of Validity Criterion-Related Validity Content-Related validity Construct –Related Validity Note: The most recent standards emphasize that validity is a unitary concept. The use of categories does not imply that there are distinct forms of validity

10. Effect Factors for Validity • Test Itself • Test Administration and Scoring • Examinees • The Criterion Chosen for Criterion Validity

11. Effect from test itself The statement of the items is clear or not The items represent the trait measured or not The length of the test is adequate or not The test difficulty is proper or not. …

12. Test administration and scoring Whether the sample is representative, heterogeneous. Whether the testing conditions are appropriate and unexpected disturbances occur. Whether the tester administers the test according to the manual. Whether the test guides for examinees are clear. Whether the Scoring system is object and standard.

13. Examinees Interests and Motivation on the Test Emotional State and Attitude During the Testing State of Physical Health Experiences on Test

14. The criterion chosen for criterion validity

15. §2 Content Validity and Construct Validity • Content Validity Interpretation Content validity involves the careful definition of the domain of behaviors to be measured by the test and the logical design of items to cover all the important areas of the domain.

16. The purpose of a content validity is to assess whether the items adequately represents a performance domain or construct of specific interest It is established through a rational analysis of the content of a test.

17. Steps for Content Validation Using Experts Judgment • Defining the performance domain of interest • Selection a panel of qualified experts in the content domain • Providing a structured framework for the process of matching items to the performance domain • Collecting and summarizing the data from the matching process

18. Application Content validity is most often employed with achievement test, so the performance domain is often defined by a list of instructional objectives. Content validity is also applicable to certain occupational test designed for employee selection and classification.

19. Table6.1 Table of Instructional objectives

20. Distinction form Face Validity The face validity refers to what it appears superficially to measure, not to what the test actually measures.

21. Construct Validity Interpretation The construct validity of a test is the extent to which the test may be said to measure a theoretical construct or trait.

22. What is Construct? Each construct is developed to explain and organize observed response consistencies. It derives from established interrelationships among behavioral measures. Examples: scholastic aptitude, intelligence, verbal fluency, anxiety, depression, self-esteem, etc..

23. Construct validation has focused attention on the role of psychological theory in test construction and on the need for formulate hypotheses that can be proved or disproved in validation process. Anne Anastasi

24. Procedures for Construct Validation Correlations between a measure of the construct and designated Internal Consistency Differentiation between Groups Development Changes Factor Analysis Multitrait –multimethod matrix

26. Example How to Search the Evidences for a Supposed Intelligence Test? • State the theory hypotheses of test: 1. Intelligence grows with the age growing 2. IQ is relatively stable 3. Intelligence is substantially related to school achievement 4.Intelligence is affected by inheritance • Administer the test to population and analyze the data. Judge: whether the test scores increase with the ages increasing; whether IQ and school achievements is correlated; IQs keep stably cross a time interval; whether the correlation between MZ is higher than the correlation between DZ.

27. §3 Criterion-Related Validity • Concepts 1.interpretation of Criterion-related Validity It is the degree on which the test scores can be related to a criterion. It indicate the effectiveness of a test in predicting an individual performance in specified activities.

28. Two Types Predictive Validity refers to the degree to which test scores predict criterion measurement that will be made at some point in the future. Concurrent Validity refers to the relationship between test scores and criterion measurements made at the time the test was given.

29. 2.What is criterion? The Criterion is some behavior that the test scores are used to predicted. For example, use the grade-point averages as the criterion of a school admissions test .

30. The problems About Criterion The reliability of criterion The validity of criterion Whether it can be measured Criterion contamination

31. Usually Used Criterion academic achievement ( for intelligence test) performance in specialized training (for special aptitude test) job performance contrasted group (for personality, domain-referenced test) psychiatric diagnosis ( for personality test ) ratings by schoolteachers, job supervisor previously available tests

32. Procedures of Criterion-Related Validation Validity Coefficient Discrimination Between Two Groups

33. Estimate Validity Coefficient • Pearson Product Moment Correlation Coefficient

34. Exercise 1 Suppose that 10 male applicants were examined one job interests test and the admitted as salesman by one company. The job interest test scores (X) and the sale amount for the first year (Y, unit is “ten thousands $”) of each applicant are listed in the following table. table 6.2 10 Applicants’ Test Scores and Sale Amount

35. Biserial Correlation Coefficient (for correlation between a continuous variable and a dichotomous variable) (6.2) , is the percentage of examinees who get point “1” on dichotomous variable , is equal to 1-p , is the mean of the test scores on the continuous variable of the examinees who get point “1” on dichotomous variable ,is the mean of the test scores on the continuous variable of the examinees who get point “0” on dichotomous variable , is the Y oirdinate of the standard normal curvve at the z-score associated with the p value. , is the standard deviation of test scores for all examinees on continusous variable

36. Research Case Use rb to estimate the validity of the fist application for WISC-R in Shanghai. Data concerned: the number of first level middle school students is 66 the number of second level middle school students is 286 the mean of IQs of the first level students is 114 the mean of IQs of the second level students is 96 the standard deviation of all students’ IQs is 14.53 if p=.1875, then Y is .2685

37. p=.1875, then Y is .2685

38. Exercise 2 The middle school students attended a math test. The mean scores of students who have been instructed with higher math program is 60.188, and their number is 382. The mean of the students who have accepted normal program is 47.429, and their number is 618. The standard deviation for all students is 11.910. Please estimate the validity coefficient of the math test.

39. 2. Discrimination Between Two Groups • Compare the means of two groups (t Test) Degree of freedom

40. Compute the overlap amount of the two groups Method 1 Compute the number of the examinees from one group (usually contrasted )whose test scores is higher than the mean of the other group; Compute the rate of the number of those test scores is higher than the mean for the other group; Then calculate the rate of the two numbers.

41. Method 2 Compute the overlap percentage of the score distribution for each group

42. §4 Application of Validity Coefficient • Predict the Criterion Score • Establish Regression Equation , is the predicted criterion score for a examinee , the test score of a examinee , is the regression coefficient, and , is the intercept, and

43. Example Figure 6.3 100 Examinees ’ Scores on Job Aptitude Test and Real Performance Scores

45. If one applicant get 6 on the test, then we can use the regression equation to predict his job performance in the future.

46. Exercise 3 Suppose a group of students from high school were examined a job interests test. Researcher obtained these statistics: The validity coefficient is 0.6. If John got 54 points on the job interest test, then what his criterion scores (job performance) would be?

47. 2. Estimate Error Standard Error of Estimate ( ) The error of estimate shows the margin of error to be expected in the individual’s predicted criterion score, as a result of the imperfect validity of the rest.

48. X1 X

49. Coefficient of Determination, indicating the proportion of the variance of criterion test scores which is related to the variance of the predictor test scores.

50. 3. Establish the approximate interval for an actual criterion Y