Roadmap Towards a Validity Argument

1. Roadmap Towards a Validity Argument Peggy Garza Associate BILC Secretary

2. Views of Validity Adapted from Chapelle (1999)

3. Validity Argument Glossary definition: Interpretive argument that presents evidence to make a case justifying the score-based inferences and the intended uses of the test. Validity Argument

5. Testing Purpose • Develop a Validity Argument for a particular test use • NATO Interoperability Capability Targets • Job Descriptions • Training or exercise requirements From STANAG 6001: • Participating nations adopt the appended table of language proficiency levels for the purpose of: • Communicating language requirements for international staff appointments. • Recording and reporting, in international correspondence, measures of language proficiency. • Comparing national standards through a standardized table while preserving each nation’s right to maintain its own internal proficiency standards.

6. How are SLPs Officially Used in NATO? E1101 outlines English Language Capability Targets • For NATO Command Structure staff / participation: • As described by job description, OR • Officers SLP 3 3 3 3 • NCOs SLP 2+ 2+ 2+ 2+ • For NATO operations, exercises or training • As described by job description, OR • Officers in command positions and principal staff officers SLP 2+ 2+ 2+ 2+ • All other officers SLP 2 2 2 2 • NCOs OR-5 and above likely to have contact with personnel from other nations SLP 2 2 2 2 • Enlisted personnel planned to operate tactical comms, or on NATO comms networks or are members of tactical air control elements SLP 2 2 1 1

7. Job Description Requirement for English: SLP 2222 What does this really mean? SLP 2 1 2 1 SLP 2 2 3 2 Not Qualified  Qualified  Test validation is the process of making a case for the interpretation and uses of test scores.

8. Audience for Validity Argument • Determine the audience for your validity argument “An argument is made and judged by an audience” Chapelle (2011) • National stakeholders • BILC: IAW sound practices in STANAG 6001 testing

9. Purpose of assessment Validity evidence is created and collected throughout the test development cycle. Tasks, criteria, instruction Scores Performance Criteria Starting at the planning phase, tasks, criteria and instructions must correspond to STANAG 6001 descriptors, CTA statements, and established test development guidelines. Diagram adapted from Luoma 2004

10. Getting started : collecting & organizing evidence to support a validity argument for National STANAG 6001 Tests Table of Contents: • Test Information for Stakeholders • Testing Personnel Qualifications and Training Records • Outline or Summary of Training Session Content • Test Specifications • Test Moderation Checklists and Documents • Test Validation Documents and Statistical Data • Test Administration Procedures • Test Security Handbook • Records of Norming Sessions for Raters • Statistical Data on Rater Reliability • Records and Summaries of a Priori Research • Records and Summaries of a Posteriori Research • _________________ • _________________

11. Roadmap Activity Tasks • Identify your audience(s) • List evidence or documentation in as many Building Blocks as possible • Share/exchange ideas and examples of evidence

12. Bibliography Chapelle, C. A. (2012). Validity argument for language assessment:  The framework is simple. Language Testing, 29(1), 19–27. Chapelle, C. A. (1999). Validity in language assessment. Annual Review of Applied Linguistics, 19(1), 254–272. Luoma, S. (2004). Assessing Speaking. Cambridge: Cambridge University Press.