46-320-01 Tests and Measurements

1. 46-320-01Tests and Measurements Intersession 2006

2. More Correlation • Spearman’s rho: two sets of ranks • Biserial correlation: continuous and artificial dichotmous variable • Point biserial correlation: true dichotmous variable

3. Hypothesis Testing Review • Independent and Dependent variables • In Psychology we test hypotheses • Null Hypothesis (H0): a statement of relationship between the IV and DV, usually a statement of no difference or no relationship – we assume there is no relationship between IV and DV • Alternative/Research Hypothesis (Ha): states a relationship, or effect, of the IV on the DV

4. Hypothesis Examples • H0: Men and women do not differ in IQ (men = women) • Ha: Men and women do differ in IQ (men  women) • Any difference in value of the DV between the levels of the IV can be explained in 2 ways – the effect of the IV or sampling error

5. Hypothesis Testing with Correlations • Null Hypothesis: there is no significant relationship between X and Y • Alternative Hypothesis: there is a significant relationship between X and Y (r is significantly different from 0) • We can use Appendix 3 (p. 641) • df = N – 2 • robs = .832 • rcrit = .195 • Reject Ho

6. Regression • We know the degree to which 2 variables are related - correlation • How do we predict the score on Y if we know X? • Regression line • Principle of least squares

7. Equation Explained • Y’: predicted value of Y • b: regression coefficient = slope • Describes how much change is expected in Y with one unit increase in X • a: intercept = value of Y when X is 0

8. Line of Best Fit • Actual (Y) and predicted (Y’) scores are almost never the same • Residual • Deviations from Y’ at a minimum • Prediction • Interpreting plot

9. More Correlation • Standard error of estimate • Coefficient of determination • Coefficient of alienation • Shrinkage • Cross validation • Correlation does not equal causation! • Third variable

10. Multivariate Analysis • 3 or more variables • Many predictors, one outcome • Linear Regression: linear combination of variables • Weights • Raw regression coefficients • Standardized regression coefficients • Predictive power

11. More Multivariate • Discriminant Analysis • Prediction of nominal category • Multiple discriminant analysis • Factor Analysis • No criterion • Interrelation • Data reduction • Principal components • Factor loadings • Rotation

12. Reliability • Assess sources of error • Complex traits • Relatively free from error = reliable • Spearman, Thorndike 1904 • Coefficients • Kuder and Richardson 1934 • Cronbach 1972 on • IRT • True Score

13. Reliability • Error and True Score • X = T + E • Random Error produces a distribution • Mean is the estimated true score

14. Reliability • True score should not change with repeated administrations • Standard error of measurement • Larger = less reliable • Use to create confidence intervals

15. Reliability • Domain Sampling Model • Shorter test estimate, but sample = error • Reliability: Usually expressed as a correlation • Reliability: Sampling distribution, correlations b/w all scores, average correlation

16. Reliability • Reliability: • Percentage of observed variation attributable to variation in the true score • r = .30: 70% of variance in scores due to random factors

17. Sources of Error • Why are observed scores different from true scores? • Situational factors • Unrepresentative q’s • What else?

18. Test-Retest Reliability • Error of repeated administration • Correlation b/w 2 times • Consider: • Carryover effects • Time interval • Changing characteristics

19. Parallel Forms Reliability • 2 forms that measure the same thing • Correlation between two forms • Counterbalanced order • Consider time interval • Example: WRAT-3

20. Internal Consistency • Split-Half reliability • Divide and correlate (internal consistency) • Check method of dividing • Why use Spearman-Brown formula? • Each test ½ length – decreases reliability • Cronbach’s alpha – unequal variances

21. Internal Consistency • Intercorrelations among items within same test • Extent to which items measure same ability/trait • Low? Several characteristics? • Use KR20, coefficient alpha • Considers all ways of splitting data

22. Difference Scores • Same trait: reliability = 0 • Use z-score transformations • Generally low

23. Observer Differences • Estimate reliability of observers • Interrater Reliability • Percentage Agreement • Kappa • Corrects for chance agreement • 1 (perfect agreement) to –1 (less than chance alone) • Interpreting: • >.75 = “excellent” • .40 to .75 = “fair to good” • < .40 = “poor”

24. Interpreting Reliability • General rule of thumb: • Above 0.70 to 0.80 – good • Higher the stakes, higher the r • Use confidence intervals (from standard error of estimate)

25. Low Reliability • Increase items • Spearman-Brown prophecy formula • Factor item analysis • Omit items that do not load onto one factor • Drop items • Correct for Attenuation (low correlations)

26. Validity • Agreement b/w a test score and what it is intended to measure • Face validity: • Looks like it’s valid • Content-validity • Representative/fair sample of items • Construct underrepresentation • Construct-irrelevant variance

27. Criterion-Related Validity • How well a test corresponds with a criterion • Predictive validity • Concurrent validity • Validity Coefficient • Coefficient of determination

28. Evaluating Validity Coefficients • Changes in cause of relationship • Meaning of criterion • Validity population • Sample size • Criterion vs predictor • Restricted range • Validity generalization • Differential prediction

29. Construct-Related Validity • Define a construct and develop its measure • Main type of validity needed • Convergent evidence • Correlates with other measures of construct • Meaning from associated variables • Discriminant evidence • Low correlations with unrelated constructs • Criterion-referenced tests