1 / 21

Validity Part II – Applications of Validity and Considerations in the Validation Process

Validity Part II – Applications of Validity and Considerations in the Validation Process. PSY605. Practical Importance of Validity. Legal considerations Equal Employment Opportunity Laws Adverse Treatment & Adverse Impact The I/O ‘Code’

sheera
Download Presentation

Validity Part II – Applications of Validity and Considerations in the Validation Process

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Validity Part II – Applications of Validity and Considerations in the Validation Process PSY605

  2. Practical Importance of Validity • Legal considerations • Equal Employment Opportunity Laws • Adverse Treatment & Adverse Impact • The I/O ‘Code’ • Best practices outlined in the SIOP Principles and Uniform Guidelines

  3. Practical Importance of Validity • Impact on performance and profit • Estimating the ROI of good measurement • Impact on attitudes (satisfaction, commitment, etc.) • Arguably an end in istelf • Also means to an end  profit (e.g.,Huselid, 1995; Subramony, 2009), customer satisfaction (e.g., Koys, 2001), low turnover (e.g., Kristof-Brown, Zimmerman, & Johnson, 2005), low absenteeism (e.g., Scott & Taylor, 1985), and low workplace accidents (e.g., Sherry, 1991)

  4. How to Develop and Implement a Validation Plan

  5. Developing and Implementing a Validation Plan for Performance-Predicting Tests

  6. How to Develop & Implement a Validation Plan – Performance Predictors & Assessments • “Validation begins with a clear statement of the proposed uses as well as the intended interpretations and outcomes and should be designed to determine how well the proposed uses will be achieved” (SIOP Principles, p.8) • When purpose is to predict performance, start with analysis of work • Job analysis – Competency Modeling

  7. How to Develop & Implement a Validation Plan – Performance Predictors & Assessments • Define organizational needs & objectives • how does this test fit with broader scope? • keep in mind different stakeholder groups may have different (even conflicting) objectives • consider workforce size & availability • I.D. organizational constraints • any ‘bad taste’ from prior tests/ assessment systems? • time, money • Communicate and get buy-in from major stakeholders • Estimate ROI for management • Communicate importance/benefits to each group

  8. How to Develop & Implement a Validation Plan – Performance Predictors & Assessments • predicting performance in specific, well-defined job vs. common competencies for groups of jobs or ever-changing jobs • Consider existing evidence – generalizing validity evidence • emphasis on ‘job relatedness’ • emphasis on performance prediction

  9. How to Develop & Implement a Validation Plan – Performance Predictors & Assessments • If using existing evidence, how will you support the appropriateness of validity generalization? • How will you support job-relatedness? • How will you ensure performance prediction? • concurrent or predictive?

  10. How to Develop & Implement a Validation Plan – Performance Predictors & Assessments • How is ‘the best’ defined in your organization? • Find/create performance measure, other criteria measures

  11. How to Develop & Implement a Validation Plan – Performance Predictors & Assessments • Pros & cons of using job incumbents vs. job applicants • Have sample complete new test + any additional measures in validation strategy + gather existing data on relevant criteria (e.g., performance ratings) • Use correlation/regression to calculate reliability and validity coefficients

  12. How to Develop & Implement a Validation Plan – Performance Predictors & Assessments • Use each validation study to build evidence in favor of future validation efforts • What impact does using this test have on the quality of hires? profit of organization? • Compare results for applicants who score very low on test vs. those who score very high • consider wide range of ‘results’ criteria (e.g., sales, burden on management, ethical behavior, performance ratings, leadership potential…) • calculate actual $$ amount corresponding to the difference on results criteria

  13. Developing and Implementing a Validation Plan for Attitude & Affect Tests

  14. How to Develop & Implement a Validation Plan – Attitude & Affect Tests • What kinds of decisions will this test inform? • In what contexts will this test be used? • What is the purpose of this validation effort?

  15. How to Develop & Implement a Validation Plan – Attitude & Affect Tests • Get all groups on board – communicate with those who will be involved about their roles, importance, benefits • Build on existing empirical evidence showing the visible benefits of happy, engaged, employees. • Attitudes & affect  retention, productivity, commitment, customer service…

  16. How to Develop & Implement a Validation Plan – Attitude & Affect Tests • Consider content-related, criteria-related, and other construct-related evidence • Keep purpose in forefront – always!

  17. How to Develop & Implement a Validation Plan – Attitude & Affect Tests • Support content validity through scale development process and SME judgments of content adequacy & dimensionality • Determine nomological network of interest for supporting construct- and criteria-related validity (base on empirical research – meta-analytic where possible) • What should your test be related to? • What should your test be distinct from? • What should your test predict?

  18. How to Develop & Implement a Validation Plan – Attitude & Affect Tests • Find existing reliable & validated measures of the other pieces of the nomological network (connecting with your test) • Look to academic journals for measures in ‘free domain’

  19. How to Develop & Implement a Validation Plan – Attitude & Affect Tests • Administer your new measure + the existing measures of the othernomological network constructs/attributes to a random sample of intended test-takers • Using correlation/regression, calculate reliability & validity coefficients called for in validation strategy

  20. How to Develop & Implement a Validation Plan – Attitude & Affect Tests • Use test scores to make decisions – changes – implement programs • Evaluate the effectiveness of those test score-based decisions – changes – programs • Translate effectiveness into $$$ • Impact on retention  $$ savings of turnover/new hire • Impact on customer satisfaction  $$ profit per customer • Impact on productivity & work quality  $$ profit for worker

  21. Things to Consider in Calculating Validity Estimates • Correlation-based  correlation considerations • sample size • range restriction (limited variance) • linearity assumption • Low Reliability • Criterion Contamination • Test bias

More Related