1 / 22

Correcting for Indirect Range Restriction in Meta-Analysis: Determining the u T Distribution

Correcting for Indirect Range Restriction in Meta-Analysis: Determining the u T Distribution. Huy Le University of Central Florida In-Sue Oh University of Iowa. The Importance of Correcting for Study Artifacts in Meta-Analysis.

giolla
Download Presentation

Correcting for Indirect Range Restriction in Meta-Analysis: Determining the u T Distribution

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Correcting for Indirect Range Restriction in Meta-Analysis: Determining the uT Distribution Huy Le University of Central Florida In-Sue Oh University of Iowa

  2. The Importance of Correcting for Study Artifacts in Meta-Analysis • The major goal of meta-analysis is to estimate the true relationships between variables (constructs) from observed correlations. • These observed correlations, however, are influenced by the effects of study artifacts  Meta-analysts need to take these effects into account in order to accurately estimate the true correlations.

  3. Range Restriction as a Study Artifact • What is Range restriction? • Occurs when the variance of a variable in a sample is reduced due to pre-selection or censoring in some way (Ree, Carretta, Earles, & Albert, 1994). • Effects of range restriction • Statistics estimated in such a restricted sample (incumbent sample) are biased, attenuated estimates of parameters in the unrestricted population (applicant sample).

  4. Two Types of Range Restriction Direct Range Restriction Explicit Selection on X resulting in distortion of correlation between X and Y X Y Indirect Range Restriction Explicit Selection on a third variable Z resulting in distortion of correlation between X and Y (which are correlated with Z); Always the case for concurrent validation studies Suitability scores Z X Y GATB Multivariate Range Restriction An extension of indirect range restriction where explicit selections occur on several variables

  5. Effects of Direct and Indirect Range Restriction on Correlations Direct Range Restriction (Sr= 20%) Unrestricted Correlation Rho=.60 Rxx =.90 Ryy =.52 R = .22 Indirect Range Restriction (Sr= 20% on Z; Rxz=.66; Ryz=.30) R = .41 R = .32

  6. Correction for Range Restriction • Direct Range Restriction: Thorndike Case II • Indirect Range Restriction: Thorndike Case III Notes: Ux = 1/uX; uX = sdx/SDx = Range restriction ratio of X (the ratio of standard deviation of the independent variable X in the restricted sample to its standard deviation in the unrestricted population). UZ = 1/uZ ; uZ = Range restriction ratio of Z (the third variable where explicit selection occurs).

  7. Correction for Range Restriction • Problems related to correcting for the effect of range restriction in Meta-Analysis: • Most studies are affected by indirect range restriction. • However, information required to correct for this effect of indirect range restriction (shown in the previous) is often not available. • The problem is even worst for meta-analysts who have to rely on information reported by primary researchers.

  8. New Range Restriction Correction Methods • Recently, Hunter, Schmidt, and Le (2006) introduced a new procedure (CASE IV) to correct for range restriction. • The procedure requires information about: • uT: Range restriction ration on the true score T underlying X • Rxxa: Reliability of X estimated in the unrestricted population. • Simulation study shows that the method is accurate (Le & Schmidt, 2006), outperforming traditional approach of using direct range restriction correction (when range restriction is actually indirect) in most situations. • Using this procedure, the researchers showed that traditional estimates of the validity of the GATB were underestimated from 24% - 45%!

  9. New Range Restriction Correction Methods Hunter, Schmidt, & Le (2006) model for the combined effects of indirect range restriction and measurement error: uS S RTP uT T P True validity RTX=(RXX )1/2 RXY X Y uX

  10. New Method for Range Restriction (Case IV) Two key characteristics: Beforeapplying Thorndike’s Case II, + Ut(instead of Ux) + Correction for measurement error before RR correction + Applying Thorndike’s Case II + Reintroducing unreliability in predictor to estimate true validity

  11. Applying the New Correction Approach to Meta-Analysis • Problem: The information needed to apply the new procedure is not available in every primary study. • In the past, meta-analysts addressed that problem by using artifact distributions. • This approach allows corrections to be made even when information of the artifacts is not available in each primary study.

  12. Difficulties in Estimating the uTDistribution Problem when applying the artifact distribution approach to correct for indirect range restriction in meta-analysis: • Need the uT artifact distribution but uT is unknown (unobservable – unlike uX)! • Hunter et al. (2006) suggested uT be estimated from uX and Rxx (reliability of the independent variable in the unrestricted population) using the formula:

  13. THE ARTIFACT DISTRIBUTION OF UT Very Rare Case! Less representative! Dependent on Rxxa! Sometimes, cannot be computed even when Rxxa and Ux are simultaneously available Individual studies Study 1: Rxxa1 uX1 uT1 N1 rxy1 Study 2: Rxxa2 N2 rxy2 Study 3: uX3 N3 rxy3 Study 4: Ryy4 N4 rxy4 ………… ........ ……. …… …… ….. …… Study k: Rxxak Ryyk uXk uTk Nk rxyk Artifact Distributions of Rxx Ryy uX uT

  14. Difficulties in Estimating the uTDistribution • Doing so, however, renders the resulting distribution of uT is highly dependent to Rxxa the assumption of independence of the artifact distributions is violated. • Further, there are values in the distributions of uX and Rxxa which cannot be combined. For example, when uX = .56 (equivalent to selection ratio of 40%) and Rxxa =.60, we cannot estimate the corresponding value of uT . Current practice is to disregard these values.  The uT distribution estimated by the current approach may not be appropriate  Meta-analysis results may be affected.

  15. Estimating the uT Distribution • Our solution: • To go backward: Instead of combining values uX and RXXin their respective distributions to estimate the values of the uTdistribution, we systematically examine the appropriateness of different “plausible uT distributions” in term of how closely they can reproduce the original uX distribution when combined with the RXX distribution. • This approach is logically appropriate because uX results from uT and RXX (see Hunter et al., 2006; Le & Schmidt, 2006), not the other way around as seemingly suggested by the formula.

  16. Estimating the uT Distribution • Procedure: Five steps (1) Selecting a “plausible distribution” for uT ( ). This distribution includes a number of representative values of uT , together with their respective frequencies. (2) The values of are then combined with all the values of Rxxin its distribution using the following equation to calculate the corresponding uX values (equation 8, p. 422, Le & Schmidt, 2006): (3) The resulting values of uX form a distribution with frequency of each value being the product of the corresponding frequencies of and Rxx in their respective distributions.

  17. Estimating the uT Distribution • Procedure: (cont.) (4) This uX distribution is then compared to the observed (original) distribution of uX, based on a pre-determined criterion. If they are close enough, as determined by the criterion, the process terminates and the “plausible distribution” of specified in step (1) becomes the estimated uX distribution. Otherwise, the process continues in step (5); (5) A new plausible distribution is constructed by keeping the original values of but systematically changing their frequencies. A new iteration is then started (by returning to step 2 above). A SAS program was developed to implement the procedure (the program is available from Huy Le).

  18. Estimating the uT Distribution • Demonstration of the procedure: • Note that the uX distribution for cognitive tests derived by Alexander et al. (1989) and the Rxxa distribution derived by Schmidt and Hunter (1977) were used.

  19. Result: The uT Distribution for Cognitive Measures

  20. Result: The uT Distribution for Educational Tests

  21. Discussion • Procedures to correct for indirect range restriction are necessarily complicated, but the procedure described in this paper will allow better, more accurate estimation of the uT distribution  More accurate meta-analysis results. • The uT distributions estimated here can be used by researchers in their future research. • Alternatively, meta-analysts can apply the current procedure to any situations where there are only sparse information about range restriction and reliabilities in their data (i.e., primary studies).

  22. Thank you! • Any Questions or Comments?

More Related