Integrating learning theory in an era of accountability testing

1. Integrating learning theory in an era of accountability testing Drew Gitomer ETS Conference on Educational Testing in America: State Assessment, Achievement Gaps, Federal Policy and Innovations September 8, 2008

2. The Problem We Are Confronting • Accountability assessments grounded in an outdated scientific model for conceptualizing proficiency, teaching it, and assessing it • Accountability is based on a limited set of limited proxies • Interim assessments, formative assessments, and teacher professional development that are emulating the less desirable characteristics of those accountability tests (aka the irony of alignment)

3. The Vision Can have an assessment system that contributes to educational practice by: • Documenting what students have achieved (“assessment of learning”), • Helping identify how to adjust instruction (“assessment for learning”), and • Engaging students and teachers in worthwhile educational experiences in and of themselves (“assessment as learning”)

4. Coherence of Three Components • Public accountability functions • Formative assessment • Professional development What is coherence?

5. Current Focus of the Project • Reading, writing and mathematics • Middle school • Primary work in Maine (Portland) • Beginning work in other places • Today’s talk focuses on reading and the work cited is led by Kathy Sheehan and Tenaha O’Reilly

6. The Foundation: A Strong Conceptual Base • Built on: • Cognitive-scientific research • Principles • Domain-specific competency models • Developmental models as available • Aligned with State standards

7. Required Skills Reading Strategies Text Conv. & Characteristics Understand & Use Texts The Reading Competency Model: The Skills Branch Learning to Read Reading to Learn Reading to Do Prerequisite Reading Skills Basic Comprehension Skills Applied Comprehension Skills Pre-Reading Strategies Model Building Strategies Strategies for Going Beyond the Text Informational Text Persuasive Text Literary Text

8. Required Skills Reading Strategies Text Conv. & Characteristics Understand & Use Texts The Reading Competency Model: The Strategies Branch Deliberate, conscious, effortful actions that readers can implement to repair breaks in comprehension & enhance understanding. Prerequisite Reading Skills Basic Comprehension Skills Applied Comprehension Skills Ask/answer pre-reading questions Use Graphic Organizers Adopt a Critical Stance Pre-Reading Strategies Model Building Strategies Strategies for Going Beyond the Text Informational Text Persuasive Text Literary Text

9. Required Skills Reading Strategies Text Conv. & Characteristics The Reading Competency Model: The Text Branch Prerequisite Reading Skills Basic Comprehension Skills Applied Comprehension Skills Pre-Reading Strategies Read Model Building Strategies Strategies for Going Beyond the Text Knowledge of conventions that contribute to reading ease or difficulty Informational Text Persuasive Text Literary Text

10. The Text Genre Literary Vocabulary Syn. Complexity Cohesion Structure, Negation, etc. Difficulty Drivers The Text Branch (Based on Sheehan, Kostin & Futagi, in press) Informational Research/Persuasive Ensure that: (1) targeted skills are assessed within a grade-appropriate span of text complexity; and (2) assessment tasks focus on those aspects of text variation that are likely to have a significant impact on reading performance.

11. Integrate Inf. Across Texts Evaluate/Critique Locate/Retrieve Inf. Und. Diff. Voc./Lit. Term. Infer Relations Amg Concepts Und. & Use Discourse Struc. The Skills Tested by Most Existing Reading Comprehension Tests Recognize Common Words Prerequisite Reading Skills Decode Unfamiliar Words Read Fluently Understand & Use Texts Basic Comprehension Skills Applied Comprehension Skills Integrate Inf w Bk. Know.

12. Key Features - Reading Design • Measure the full range of skills, including prerequisite skills • Purposeful, authentic, scenario-based tasks • Support reading across the curriculum • Informational Texts: Excerpts from the introductory chapters of science and social studies textbooks, newspaper & magazine articles, etc. • Persuasive Texts: Editorials, Letters to the editor • Literary Texts: Narratives, memoirs, poems • Select task formats that encourage use of targeted reading strategies • Include a mixture of expressive (CR) and receptive (mc) tasks • Minimize production requirements • Want to test reading not writing

13. Implementation: The Assessment Design • Period accountability assessments (PAAs) • Foundational tasks • Constructed response and selected response • Scaffolded structure • Supplemental tasks • Computer Administration • Automated Scoring

14. CBAL vs. (stereo)typical assessments

15. A Sample PAA • Spoken Module • Provide mastery evidence relative to the Prerequisite Skills • Short Tasks: Word Recognition, Decoding, Oral Fluency • Students speak into a headset, responses are scored as Words Correct per Minute (wcpm) • Comprehension Module • Provide mastery evidence relative to the Model-Building & Applied Comprehension Skills • All texts & tasks revolve around a meaningful scenario • e.g., You have to write a report about the scientific method for your science class • Tasks employ innovative response formats: SR and CR • Multi-step thinking, scored via partial credit scoring rubrics

16. The Scenario • Four Texts • A passage about Ben Franklin adapted from a Social Studies textbook • An article about the Scientific Method extracted from an Encyclopedia • A newspaper article about 3 winners of the Intel Science Competition • A diagram from a student lab report

17. Preliminary Results – Construct Validation TOWRE: a standardized measure of Prerequisite Skills Gates-MacGinitie: a std. measure of Model-Building Skills

18. The Assessment is Also Designed to Yield Diagnostic Information that Teachers can Use to Adjust Instruction

19. Formative System • Is built upon the same conceptual base as the Accountability System • Gets information from the Accountability System • Is used at the district’s and teacher’s option • Is available on demand • Performance in the formative system will not be used for accountability purposes

20. Structure of formative assessment • Library of resources including professional development component • Elaborated foundational tasks – for each task component • Clarification of learning intentions – connection to competency model • Criteria for considering student responses • Common conceptual challenges • Potential pedagogical strategies • Potential additional resources

21. Challenges • Psychometric • Automatic scoring • Educational infrastructure • Political accountability