Reverse-Engineering Passage Test Specifications in ELL Assessments:

1. Reverse-Engineering Passage Test Specifications in ELL Assessments: The Key for Good Passage Creation for Tests. Luis Perea UTSA ESL Services The problem: The need to create passage specifications when none available that will be clear, concise but informative, but most importantly that are able to address appropriate passage-based items and are congruent to the standards/benchmarks/constructs of the test and the test design. Research questions: “What is the process to reverse-engineer passage test specifications by including key stakeholders in the assessment development (teachers, consultants, content specialists, editors, etc.)?” “Why is passage reverse-engineering indispensable to the test process?” • Reverse-Engineering • Straight RE • Historical RE • Critical RE • Test Deconstruction • Parallel RE • Standards RE • The Basic Components of Creating Passages & Passage Test Specifications: • Passage Types (fictional & non-fictional). • inclusion of Passage Topics & exclusion of specific Taboo Topics • Difficulty levels (Passage Length, Readability Levels, Vocabulary Types / Difficulty) • Linguistic Considerations • Theoretical proficiency levels • Levels of Cognitive Thinking • Grade level or Grade grouping • Syntactic Complexity • Grammar • Discourse features • Genre characteristics The Solution: Propose Model for Passage Specification Creation from Davidson & Lynch (2002) originally for item specifications. This means including illustrative sample passages per level or grade grouping (and if possible by genre) to provide a clear image to passage writers. Adding a specification supplement section in form of sub-sections or appendixes will be beneficial to create informative and accurate passage specs. Davidson (2003) provides the definition of spec-reverse engineering as “the production of test specifications when none exist: all we have are test items or tasks, and from those we induce what the blueprint might be” (p. 1).

2. The Context of the Project & the Gaps • Passage RE of a standardized English assessment for students at grades 3-11 in a U.S. project. • No prior documentation was recorded in terms of item and passage test specifications. • Different stakeholders in both the Department of Education and the test development team rotate or processes get inherited from one team to another, and so on. Appendixes can include: Lists of vocabulary words. Information regarding website evaluation and trustworthy resources. Passage Development as this affects Passage-Based Items. A General Passage Development Plan types of art / photos ) For listening passages, a listening script template A matrix of guidelines to account for cognitive levels • Reading Passage Specification RE • Problem – Long passages • Solution - Teachers evaluated the length and appropriateness of the passages, they looked at the samples brought, made individual notes and in a consensus based approach and recommended word ranges. • Listening Passage Specification RE • Problem – problems with the test design for listening and even future passage development plans. • Length of passage would have an impact on how the 2011 test forms would be created. • Historically previous OP & FT administrations and forms lacked of structure in the number of associated items per passage in listening as well as how many passages and passage length could have been pulled in a single form. • Solution – Use previous administrations as starting point; then re-define the length of passages vis-à-vis number of test items permissible. Finally, redefine test design / blueprints. • Writing Passage Specification RE • Problem - Some passages need to be short and straightforward enough to avoid including the reading construct in assessing writing. • Solution – Re-evaluation of current passages and improve their quality • Conclusion: • Obtain consensus from teacher panels about what constitutes an appropriate passage for word length ranges, vocabulary level, genre types, culturally appropriate topics, readability levels, curricular approach, etc. • - After specs have been created and passages been developed, obtain feedback from item writers’ reactions to newly created passages (based on the RE passage specs) and document observations and comments for improvement, congruence between passages, items and standards so that passage specifications can be redefined.