1 / 27

Validity

Validity. Validity. Degree to which inferences made using data are justified or supported by evidence Some types of validity Criterion-related Content Construct All part of unitarian view of validity

keziah
Download Presentation

Validity

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Validity

  2. Validity • Degree to which inferences made using data are justified or supported by evidence • Some types of validity • Criterion-related • Content • Construct • All part of unitarian view of validity • Constructs - theoretical abstractions aimed at organizing and making sense of our environment; they are LATENT

  3. Importance of Criteria • A criterion is any variable you wish to explain and/or predict • They are the key to well-developed theory, good measurement, and strong research design • Ultimate criterion • Multidimensional nature of criteria • Intermediate criteria

  4. Criterion-Related Validity • Process of establishing a relationship between variables • Predictive, concurrent, postdictive • Usually based on correlation or regression equation • Low reliability will attenuate or mask relationships

  5. Usefulness of criterion-related validity estimate • Selection Ratio – proportion of the individuals in the sample who are selected of the total number • Base rate – percent of successful individuals under random selection • Range Restriction • Differential Prediction for different subgroups

  6. DecisionsSR,BR=.50 False Negatives VP Successful FN FN+VP=BR VN+FP=1-BR VP+FP=SR FN+VN=1-SR Yc FP Unsuccessful VN False Positives Reject Accept Xc

  7. DecisionsSR=.15,BR=.50 FN+VP=BR VN+FP=1-BR VP+FP=SR FN+VN=1-SR False Negatives VP Successful FN Yc Unsuccessful FP VN False Positives Reject Accept Xc

  8. DecisionsSR=.85,BR=.50 False Negatives Successful VP FN Yc FN+VP=BR VN+FP=1-BR VP+FP=SR FN+VN=1-SR Unsuccessful FP VN False Positives Reject Accept Xc

  9. DecisionsSR=.50,BR=.80 VP FN FN+VP=BR VN+FP=1-BR VP+FP=SR FN+VN=1-SR Yc FP VN Xc

  10. Criterion-Related Validity • Even low correlations can lead to large increases in selection efficiency • SR and BR have strong influences • When SR is small (choose few), fewer FP and more FN • When SR is large, fewer FN and more FP • When BR is large (many can be successful), SR and validity have little effect on selection efficiency • Most gains in success ratio when BR = .50 and SR is small (e.g., .10) • The tradeoffs depend on purpose of selection

  11. Range Restriction - Direct - Indirect - Ambiguous Y X

  12. Differential PredictionIntercept Bias Y X Same prediction for each group

  13. Differential PredictionSlope Bias Y X Different prediction for each group

  14. Content Validity (Logical Analysis) • Extent to which items or measures cover the content area the test purports to measure • Expert judges determine if a measure came from a particular content domain • Scoring and content is based upon theory • If measures are from same content domain, should demonstrate high reliability • If low internal consistency reliability, low content validity

  15. Construct Validity • Validity of inferences about latentunobserved variables on the basis of observed variables • Does a measure assess what it is intended to assess? Do the variables relate in theoretically meaningful ways? • Low reliability will make it difficult to assess the nature of a particular construct and attenuate relationships with other constructs

  16. Construct Validity What you think Theory True Relationship Cause Construct Effect Construct Observed Relationship Measure or Manipulation Observed Outcomes What you see Can we generalize to the constructs from the measures?

  17. 3 Ability to Learn Vegetarianism Anxiety 2 4 5 Measure of Anxiety (X) Test Score (Y) Salads Eaten (Z) 1 Construct Validity

  18. Ways to Establish Construct Validity • Internal Structure Analysis • Cross Structure Analysis • Nomological network (Cronbach & Meehl)

  19. Internal Structure Analysis • Factor Analysis • Used to identify factors or dimensions that underlie relations among observed variables • Exploratory - Useful When: • No info on internal structure available • Factor structures may look different than original scale • You have reservations about previous factor analyses • Confirmatory - Useful When: • You have some idea of the internal structure • Confirming factor structures from previous studies • Necessary but not sufficient to establish construct validity

  20. Anxiety Ability to Learn X1 X4 Z3 Z1 X2 X3 Z2 e1 e4 e7 e2 e5 e3 e6 Internal Structure Analysis

  21. Cross-Structure Analysis • Embedded in nomological network (nomological validity) • Test of hypotheses by examining relationships between different indicators of underlying constructs • e.g., leadership style based on reports from subordinates and leadership self-report inventory • Relies on multiple methods of measurement

  22. Nomological Network • A representation of constructs of interest in a study, their observable manifestations (measures), and the interrelationships among and between them • Cronbach & Meehl said this is necessary to establish construct validity • Elements include: • Specify linkage between constructs (hypotheses) • Operationalize constructs (specify measurement)

  23. Convergent and Discriminant Validity • Convergent validity - Convergence among different methods designed to measure the same construct • Discriminant validity - Distinctiveness of constructs, demonstrated by divergence of methods designed to measure different constructs • Multi-Trait Multi-Method

  24. MTMM • Heterotrait-Monomethod • Different traits, same method • Heterotrait-Heteromethod • Different traits, different methods • Monotrait-Heteromethod • Same trait, different methods • Validity diagonals • Monotrait-Monomethod • Same trait, same method • Reliability diagonals

  25. MTMM Method1 Method2 Method3 A1 B1 C1 A2 B2 C2 A3 B3 C3 M1 A1 (.89) B1 .51 (.89) C1 .38 .37 (.76) M2 A2 .57 .22 .09 (.93) B2 .22 .57 .10 .68 (.94) C2 .11 .11 .46 .59 .58 (.84) M3 A3 .56 .22 .11 .67 .42 .33 (.94) B3 .23 .58 .12 .43 .66 .34 .67 (.92) C3 .11 .11 .45 .34 .32 .58 .58 .60 (.85)

  26. Steps to Establish Construct Validity • Specify the nomological net (expected + and - relationships) of expected relations • Establish reliability • Check convergence with other preexisting measures of the construct (convergent validity) • Factor analysis • Empirical studies of relatedness • Empirical studies of discriminability

  27. Assignment 4 • Take the hypotheses you developed in assignment 2 and the variables that were included in them. • Draw a picture of what you believe the nomological network of these variables would look like • What alternative measures of each variable might you use (different than those specified in Assignment 3) to establish convergent validity? • Draw what an MTMM construct validity chart would look like that includes each variable in your study and the original and alternative measures you identified for each construct. Specify whether each correlation would be expected to be Hi, Low or Moderate.

More Related