1 / 31

Schmidt & Hunter Approach to r

Explore statistical artifacts like range restriction and reliability that impact observed effects in research. Learn about corrections for sampling error and computational error to enhance data accuracy. Discover how to handle discrepancies in reliability across studies for more robust meta-analysis.

clevelandr
Download Presentation

Schmidt & Hunter Approach to r

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Schmidt & Hunter Approach to r Artifact Corrections

  2. Statistical Artifacts • Extraneous factors that influence observed effect • Sampling error* • Reliability* • Range restriction* • Computational error • Dichotomization of variables *Addressed in the analysis Sampling error (in theory) has no systematic effect on ES; just noise. Less than perfect reliability and typical range restriction serve to reduce the observed ES. Differences between studies in reliability and range restriction increase the between studies variance (REVC) in theoretically barren ways.

  3. Effect of a single reliability If we multiply a distribution by a constant (e.g., .77 (sqrt(.6)), the new mean is the old mean times the constant (100*.77) and the new standard deviation is the old SD times the constant (10*.77=7.7). Unreliability reduces the mean and variability. If we correct, increase both. ryy =1, M=100, SD=10 ryy =.6, M=77, SD=7.7

  4. Effect of multiple reliabilities • Study 1 rxtyt = .8, ryy = .9, rxy = .8*sqrt(.9) = .76 • Study 2 rxtyt = .8, ryy = .7, rxy = .8*sqrt(.7) = .67 • Differences in reliability across studies will increase the variance of observed correlations

  5. Range Restriction/Enhancement These are examples of direct RR.

  6. Psychometric Meta-Analysis Disattenuation for reliability Correction for both Correction for IV Correction for DV Suppose rxy = .30, rxx = ryy = .80. Then:

  7. Direct Range Restriction/enhancement Suppose rxy = .33, SD1=12, SD2 = 20. Then: Can also invert by uX = 1/UX

  8. Indirect RR Reliability of IV in restricted sample (job incumbents in I/O validation study). Reliability of IV in unrestricted sample (job applicants in I/O validation study). Ratio of SD of true scores; analogous to uX. You will need rxxa for DIRECT range restriction correction. You will need uT AND rxxi for INDIRECT range restriction correction.

  9. Meta-Analysis of corrected r • If information is available can correct r for each study • Compute M-A on the corrected values • Can also be done with assumed distributions, but I do not recommend it.

  10. Steps (1) Record data (N, r, artifact values rxx, etc.) Compute the corrected correlation for each study: If there is only 1 kind of artifact, disattenuation is simple: Where a is the disattenuation factor. Note ro is observed and rC is corrected. If there is range restriction, things are tricky. If INDIRECT range restriction, then use Ut instead of Ux and disattenuate for reliability before adjusting for range restriction. Use reliabilities from the restricted group. If DIRECT range restriction, adjust for ryy, then range restriction, then rxx, but rxxa, the reliability in the unrestricted group.

  11. Steps (1b) For each study, compute compound attenuation factor: Compute sampling variance of uncorrected r: Note this is sampling variance for one study.

  12. Steps (2) Compute sampling variance of disattenuated r: If there is range restriction, then do the following 2 steps. Compute adjustment for range restriction: Adjust sampling variance of disattenuated r: Compute weights: Note A is the compound attenuation factor.

  13. Steps (3) Compute the weighted mean: Compute the weighted variance: Compute average corrected r sampling error: Compute variance of rho:

  14. Psychometric M-A data We did bare bones already. Now we will analyze 3 ways: (1) just criterion reliability, (2) all artifacts with INDIRECT RR, (3) all artifacts DIRECT rr.

  15. Correct ryy only (1) Suppose we only wish to correct for criterion unreliability. Study 1 r = .20, rxx = .90, ryy = .80, Ux = 1.5 Disattenuation ryy : rC = .2/sqrt(.8) = .223607. Compound attenuation factor A = .20/.223607 = .894.

  16. Correct ryy only (2)

  17. Correct ryy only (3)

  18. Correct ryy only (4)

  19. Correct ryy only (5)

  20. All corrections, Indirect RR Already know bare-bones mean.

  21. Indirect RR (2)

  22. Indirect RR (3)

  23. Indirect RR (4)

  24. Indirect RR (5)

  25. Indirect RR (6)

  26. Direct Range Restriction (1)

  27. Direct RR (2)

  28. Direct RR (3)

  29. Direct RR (4)

  30. Direct RR (5)

  31. Philosophical Issues • Psychometric M-A (Schmidt & Hunter) • Want to know relations among constructs, not the measured variables • Sometimes unrealistic estimates at the end • Problematic prediction intervals • Other methods (Hedges, Rosenthal) • Avoid extrapolating to data that you have not observed • Parameter estimates may be incorporated into weights leading to bias

More Related