1 / 13

Validity Evaluation

Validity Evaluation. NCSA Presentation NAAC/NCIEA GSEG CONSTORIA. Validation is “a lengthy, even endless process” (Cronbach, 1989, p.151). Kane’s argument-based framework.

ciqala
Download Presentation

Validity Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Validity Evaluation NCSA Presentation NAAC/NCIEA GSEG CONSTORIA

  2. Validation is “a lengthy, even endless process” (Cronbach, 1989, p.151)

  3. Kane’s argument-based framework • “…assumes that the proposed interpretations and uses will be explicitly stated as an argument, or network of inferences and supporting assumptions, leading from observations to the conclusions and decisions. Validation involves an appraisal of the coherence of this argument and of the plausibility of its inferences and assumptions (Kane, 2006, p. 17).” NAAC/Center for Assessment. 4/30/08

  4. Two Types of Arguments An interpretative argument specifies the proposed interpretations and uses of test results by laying out the network of inferences and assumptions leading from the observed performances to the conclusions and decisions based on the performances The validity argument provides an evaluation of the interpretative argument (Kane, 2006) NAAC/Center for Assessment. 4/30/08

  5. The NAAC-GSEG Builds off of previous projects (NHEAI and first phase of NAAC) that focused on technical documentation Collecting and evaluating validity evidence was a significant challenge This project focuses on having states (5) create and evaluate validity arguments Our focus is NOT on single studies or even sets of studies, but on how the studies support or refute a validity argument NAAC/Center for Assessment. 4/30/08

  6. Process • Theory of Action • Interpretive Argument • Validity Evaluation NAAC/Center for Assessment. 4/30/08

  7. Theory of Action

  8. Types of Studies • Surveys • Teachers and Principals • Skills, resources, supports • Scorer background • Consequential validity • Parent Surveys • Learner Characteristics Inventories • Classroom Visit Protocols • Scoring Observation Protocols • Performance-level judgment assignment • Focus group

  9. Prioritizing the Studies • Based on the Theory of Action… • Which studies will give us the best information to develop an interpretive argument? • How will we synthesize the study results into our interpretive argument?

  10. Evaluating the Validity Argument • Haertel (1999) reminded us that the individual pieces of evidence (typically presented in separate chapters of technical documents) do not make the assessment system valid or not, it is only by synthesizing this evidence in order to evaluate the interpretative argument can we judge the validity of the assessment program.

  11. An approach for organizing synthesis for validity evaluation (Ryan & Marion)

  12. Next Steps States are completing their studies and comparing their results with their interpretive arguments. They will then evaluate the study results interpretive argument comparisons to develop the validity evaluation.

  13. Contact Information Jacqui Kearns, Ed.D. NAAC University of Kentucky www.naacpartners.org Scott Marion NCIEA www.nciea.org

More Related