The Curious Life of Local Reading Items

1. The Curious Life of Local Reading Items Annual Spring WERA Conference March 30, 2007 JoAnne Buiteweg, Peter Hendrickson and Debra Ritchhart Everett Public Schools PETER INTRODUCE SELF AND DISTRICT _ HANDOFF TO JOANNE JOANNE INTRODUCES HERSELF AND BEGINS THE “STORY” CUE: “SO THE STORY BEGINS WITH…” NEXT SLIDEPETER INTRODUCE SELF AND DISTRICT _ HANDOFF TO JOANNE JOANNE INTRODUCES HERSELF AND BEGINS THE “STORY” CUE: “SO THE STORY BEGINS WITH…” NEXT SLIDE

2. Once Upon a Time… BRIEF THEN NEXTBRIEF THEN NEXT

3. In a place not so far, far away… BRIEF – THEN NEXTBRIEF – THEN NEXT

4. There was a department organized under the Executive Director for Curriculum Alignment and Implementation… Unique title but reflects our superintendent’s focus on aligning, articulating, and coordinating our efforts to reach our mission - each student will learn to high standards Core starting place – assure that each student have the opportunity to learn and be provided the experiences necessary to demonstrate core skills, knowledge, and understanding. NEXT SLIDE Unique title but reflects our superintendent’s focus on aligning, articulating, and coordinating our efforts to reach our mission - each student will learn to high standards Core starting place – assure that each student have the opportunity to learn and be provided the experiences necessary to demonstrate core skills, knowledge, and understanding. NEXT SLIDE

5. Opportunity to Learn (Requirements) Align curriculum & instruction to the assessed standards Align assessments to the assessed standards Monitor student progress on the assessed standards Communicate progress on the assessed standards to students and parents Do something additional for students not at standard Each curricular area is led by a specialist and supported by either facilitators or cadre who provide the professional development to advance the initiatives and best practices. The area curriculum and assessment as well as curriculum and instruction are paired and provide generalist support. Each area is curricular area is responsible for planning to account for each one of the Opportunity to Learn standards. Each curricular area has started in a slightly different place on a staggered timeline. Math Curriculum K-8 then 9-12 Instructional strategies that are embedded in curricular design Common assessment development Science Unit – Kit Adoption and Training Instructional strategies that are embedded in curricular design Common assessment development Literacy Common curricular materials & Instructional calendar Common assessment development & curriculum mapping Instructional strategies that are embedded in curricular design NEXT SLIDE Each curricular area is led by a specialist and supported by either facilitators or cadre who provide the professional development to advance the initiatives and best practices. The area curriculum and assessment as well as curriculum and instruction are paired and provide generalist support. Each area is curricular area is responsible for planning to account for each one of the Opportunity to Learn standards. Each curricular area has started in a slightly different place on a staggered timeline. Math Curriculum K-8 then 9-12 Instructional strategies that are embedded in curricular design Common assessment development Science Unit – Kit Adoption and Training Instructional strategies that are embedded in curricular design Common assessment development Literacy Common curricular materials & Instructional calendar Common assessment development & curriculum mapping Instructional strategies that are embedded in curricular design NEXT SLIDE

6. Royal Experts Wiggins, G. P., & McTighe, J. (2005). Understanding By Design. Alexandria, VA: ASCD. Stiggins, R. J. , Arter, J. A., Chappuis, S., and Jan Chappuis. (2004). Classroom Assessment for Student Learning: Doing It Right-Using It Well. Portland, OR: Assessment Training Institute. O’Connor, K. (2002). How to Grade for Learning. Arlington Heights, IL: Skylight Professional Development. Marzano, R. J. (2000). Transforming Classroom Grading. Alexandria, VA: ASCD. DuFour, R. and Eaker, R. (1998). Professional Learning Communities at Work: Best Practices for Enhancing Student Achievement. Alexandria, VA: ASCD. Timeline of experiences with current thinking: Understanding by Design entered our system through many channels including the Teacher Leadership Grants. The work of Judy Arter in rubrics was alive and well in our Middle School Competency Project Work In 2002, brought in by the Curriculum Alignment and Implementation division, Ken O’Connor was a keynote speaker for our entire high school staff. He followed up with building level presentations and his ideas intrigued the principal group and he was brought back to speak during the year to leadership councils. The literacy initiative sparked by O’Connor’s visit began to design a standards-based reporting and instruction initiative. Professional Learning Communities – Dufour and Eakers - Intervention – professional communities We began to focus on helping teachers understand assessment design and knew the WASL and the assessed targets were our starting place. Soon discovered that the alignment work had been to the teacher’s plans not the student’s work as evidence of those standards. Marzano’s work was helpful NEXT STEPTimeline of experiences with current thinking: Understanding by Design entered our system through many channels including the Teacher Leadership Grants. The work of Judy Arter in rubrics was alive and well in our Middle School Competency Project Work In 2002, brought in by the Curriculum Alignment and Implementation division, Ken O’Connor was a keynote speaker for our entire high school staff. He followed up with building level presentations and his ideas intrigued the principal group and he was brought back to speak during the year to leadership councils. The literacy initiative sparked by O’Connor’s visit began to design a standards-based reporting and instruction initiative. Professional Learning Communities – Dufour and Eakers - Intervention – professional communities We began to focus on helping teachers understand assessment design and knew the WASL and the assessed targets were our starting place. Soon discovered that the alignment work had been to the teacher’s plans not the student’s work as evidence of those standards. Marzano’s work was helpful NEXT STEP

7. Standards-Based Core Course Elements Culmination – Re-thinking course design and assuring we have the end in mind before all else. Common core – Assessed Standards NEXT SLIDECulmination – Re-thinking course design and assuring we have the end in mind before all else. Common core – Assessed Standards NEXT SLIDE

8. The Secondary Literacy Initiative began. Previous work had been done with teams in aligning their instruction to standard but as initiative so we started with reporting Accounting for evidence of learning in order to build supplemental report revealed lack of understanding and need for alignment of performance standards rather than attaching standards as labels to units. Grow over time INTRO DEB – Teacher, member of cadre during beginning of pilot and her experiences led to creation of a new leadership role. KEEP HERE UNTIL SHE BEGINS TO TALK AS IFLThe Secondary Literacy Initiative began. Previous work had been done with teams in aligning their instruction to standard but as initiative so we started with reporting Accounting for evidence of learning in order to build supplemental report revealed lack of understanding and need for alignment of performance standards rather than attaching standards as labels to units. Grow over time INTRO DEB – Teacher, member of cadre during beginning of pilot and her experiences led to creation of a new leadership role. KEEP HERE UNTIL SHE BEGINS TO TALK AS IFL

9. Instructional Facilitator in Literacy2004-05 Working with teachers to answer: Does the evidence you are collecting truly reflect what you have taught? Have you gathered and provided feedback on enough formative evidence throughout a unit for students to learn? Do you have enough summative evidence to accurately assess a student? DEB SPEAKS AS FACILITATOR AS SHE ENDS ADVANCE TO NEXT SLIDEDEB SPEAKS AS FACILITATOR AS SHE ENDS ADVANCE TO NEXT SLIDE

10. HAND OFF TO JOANNE Initiative took on a multi-level approach. District Program Goals and Core Training Follow-up, development, modeling, and details as team of facilitators and in building with teams and individuals Starting place: Curriculum Maps & Common District Assessments (UBD & PLC) NEXT SLIDEHAND OFF TO JOANNE Initiative took on a multi-level approach. District Program Goals and Core Training Follow-up, development, modeling, and details as team of facilitators and in building with teams and individuals Starting place: Curriculum Maps & Common District Assessments (UBD & PLC) NEXT SLIDE

11. Development of Common Assessments Writing Annual assessment provided a bank of prompts with annotations Initial target of literacy initiative was to shift from six trait writing scoring to a guide more aligned with WASL but rich enough to help focus instruction Results a 4 x 4 instrument developed over two years Reading WASL released items did not create enough of a focus for classroom instruction Initial target of literacy initiative was to train teacher leaders to write reading items for common assessments for both monitoring teaching and learning as well as develop students’ assessment literacy Writing had a starting place – Development of new instructional rubric bridging 6 trait and state scores Re-annotating bank of assessments Setting up internal scoring rather than out sourcing Scoring conferences utilized to review rubric annually Reading needed a Starting Place Decision curriculum maps and build common assessments NEXT SLIDEWriting had a starting place – Development of new instructional rubric bridging 6 trait and state scores Re-annotating bank of assessments Setting up internal scoring rather than out sourcing Scoring conferences utilized to review rubric annually Reading needed a Starting Place Decision curriculum maps and build common assessments NEXT SLIDE

12. Decisions were made: Determined how often and how many common assessments Looked at assessed reading targets as 10 rather than 20 – balanced information and literary text by pairing up in instruction Calendared instruction to include all 10 targets by end of second trimester (in March – 3 full weeks prior to WASL window) Instructional calendar and assessment plan became foundation for Curriculum Maps. Starting Point Baseline – Provide classroom and individual pictures of students performance on targets Decision point – reflect beginning of year expectations or end of the year expectations Coached – Provide classroom teachers with form and format of WASL-like structure and scripts that help them familiarize students Used grade level text (Prentice Hall) Scripts based on building assessment literacy Independent– Based on instructional targets of previous trimester Unique text Decision point – do you preview future targets or keep tests focused on previously taught targets NEXT SLIDEDecisions were made: Determined how often and how many common assessments Looked at assessed reading targets as 10 rather than 20 – balanced information and literary text by pairing up in instruction Calendared instruction to include all 10 targets by end of second trimester (in March – 3 full weeks prior to WASL window) Instructional calendar and assessment plan became foundation for Curriculum Maps. Starting Point Baseline – Provide classroom and individual pictures of students performance on targets Decision point – reflect beginning of year expectations or end of the year expectations Coached – Provide classroom teachers with form and format of WASL-like structure and scripts that help them familiarize students Used grade level text (Prentice Hall) Scripts based on building assessment literacy Independent– Based on instructional targets of previous trimester Unique text Decision point – do you preview future targets or keep tests focused on previously taught targets NEXT SLIDE

13. Middle School Reading Common Assessments Reading WASL: sets conditions Practice WASL: to simulate rigor (from OSPI) Coached: focus on assessment literacy and current trimester targets Independent: to mirror WASL conditions (simulations) Baseline: Beginning-of-the-Year (Pre-Test) on all 10 targets all multiple-choice questions (opt. short answer & extended response items) 1st Trimester: Focus on targets 1-5 1 of 2 Short Answer should be summarizing 1 Extended Response should be literary elements or text features stem 2nd Trimester: Focus on targets 6-10 2 Short Answer questions Extended Response should be compare and contrast 3rd Trimester: End-of-the-Year (Post-Test) on all 10 targets all multiple-choice questions (opt. short answer & extended response items) Our decisions: Keep the tests doable and of greatest perceived value to the most teachers. Pay attention to patterns and information regarding WASL items. NEXT SLIDE Our decisions: Keep the tests doable and of greatest perceived value to the most teachers. Pay attention to patterns and information regarding WASL items. NEXT SLIDE

14. Role of Item Writers Strong relationship to classrooms, teachers Deep understanding of GLEs, test specifications, item specifications Broad knowledge of accessible text, web sources Literacy specialists, coach/leaders Develop, administer, analyze, translate to instruction Who would write the tests? 21 middle school assessments for 3 grades 420 items 360 multiple choice 42 short answer 18 extended response The best candidates – most accessible for development, monitoring implementation, and providing professional development in responding to the data: Teachers – Instructional Facilitators Now that we’ve written them – what do we do? NEXT SLIDEWho would write the tests? 21 middle school assessments for 3 grades 420 items 360 multiple choice 42 short answer 18 extended response The best candidates – most accessible for development, monitoring implementation, and providing professional development in responding to the data: Teachers – Instructional Facilitators Now that we’ve written them – what do we do? NEXT SLIDE

15. Gathering the Data Utilize Technology Input Collect response selection Easy entry for short answer and extended response Automate multiple choice answer scoring and blend with short answer and extended response Assessment total score, strand scores, and target scores to teachers as immediate as possible Reports & Displays Provide student level, classroom level, grade level by school and by district comparison information for each assessment Item level data by response at all levels Ongoing focus for consistency, efficiency, and maximizing both quality and utility of assessments. In order to know if our work is any good we must… slide We need expertise to help us utilize what we have created so that it best helps students. Final comment – “What is the power of these seemingly simple questions?” NEXT SLIDE Ongoing focus for consistency, efficiency, and maximizing both quality and utility of assessments. In order to know if our work is any good we must… slide We need expertise to help us utilize what we have created so that it best helps students. Final comment – “What is the power of these seemingly simple questions?” NEXT SLIDE

16. What we can learn from items… HANDOFF TO PETER NEXT SLIDE ON HIS CUEHANDOFF TO PETER NEXT SLIDE ON HIS CUE

17. Classical Test TheoryP-values Difficulty of a test item Percentage selecting correct response Example 100 students respond Correct answer is “C” 65 students answer “C” P-value = 65/100=0.65

18. P-values… Lower…harder < .20 too close to guessing if five choices Hard .20 to .40 Confusing language? Teach again? Sweet P-values Around 0.60 Range .40 to .60 Higher…easier = or > 0.90 too easy Easy .61 to 89 Very little information

19. Classical Test TheoryPoint Biserials Strength of association between: Correlation right and wrong scores with Total test score Example 30 item test, each item worth a point Item “5” is correct (1) or incorrect (0) Compute Total student scores minus Item “5”score Correlate Item “5” scores to Total minus “5” scores

20. Point -Biserials… Low Point-Biserials <0.15 reject, =>0.15 minimal, =>0.25 good Capable students missing easy items Sweet spot 0.3 or higher High Point-Biserials…more discriminating 1.0 is max Higher scorers got correct Lower scorers missed it

21. Calculating PBS in Excel Create data matrix “A” Cases in rows Item scores 0,1 in columns Sum case scores in Totals column Create data matrix “B” Mirror of matrix “A” Substitute (Total Score-Item Score) for Item Scores Correlate matrix “A” with matrix “B”

22. Caution: Interpretation Item analysis does not equal validity Good p-values and point-biserials may mask invalid items Very easy or tough items may be needed to sample content Application item among many fact items may not discriminate well…but we need them Item statistics influenced by students sampled

23. Why Bad or Misfitting Items? Poorly written, confusing Unclear, misleading graphics No clear, correct response Obviously wrong distractor Item reflects different content than rest Bias against some gender, ethnic, other subgroup (Differential Item Functioning)

24. Item Analysis DisplayTri1 2006 Reading Test

25. What we can learn from items…

26. Distractor Evaluation Distractor quality influences performance Must be incorrect Appeal to low scorers, not at mastery Infrequent choice of high scorers Poor distractor? Revise Replace Remove

29. Item Analysis References “Test item analysis and decision making”. DIIA, University of Texas at Austin. Accessed 14 February 2007 http://www.utexas.edu/academic/mec/scan/index.html The Measurement and Evaluation Center (MEC) offers tutorials for faculty writing and interpreting tests. Halydna, T.M. (1999). Developing and validating multiple-choice test items (2nd ed.). Mahway, NJ: Lawrence Erlbaum Associates. This has become the standard text for item development, oft cited in measurement articles.

30. Item Analysis References Netsky, Bev. (2001). “Ask Dr. Psi: P-values and Point Biserials”. Bloomington, MN: Pearson VUE Accessed 14 June 2006 http://www.promissor.com/knowledge/askdrPsi/drcat20010223.asp. Pearson VUE’s Promissor is the online test division of Pearson/NCS where Dr. Psi (testing) and Dr. Phi (mathematics) reside. Varma, Seema. (--). Preliminary item statistics using point-biserial correlation and p-values. Morgan Hill, CA: Educational Data Systems. This slim tutorial for educators from a Rasch shop demonstrates the use of Excel to compute simple test statistics. SPSS syntax is also provided. Find them at http://www.eddata.com. ADVANCE SLIDE AS HANDING OFF TO DEBADVANCE SLIDE AS HANDING OFF TO DEB

31. Hand back to DebHand back to Deb

33. Guiding Questions 1.  What do we want each student to learn? 2.  How will we know if they have learned? 3. How do we respond when students don't learn? 4. How do we respond if students already know the content?

34. The moral of the story?

35. One little item can be the seed for changing the landscape for our students!

36. Narrators’ Contact Information JoAnne Buiteweg, Curriculum & Assessment Specialist jbuiteweg@everettsd.org, Educational Service Center Peter Hendrickson, Ph.D., Assessment Specialist phendrickson@everettsd.org, Educational Service Center Debra Ritchhart, Instructional Facilitator for Literacy dritchhart@everettsd.org, Heatherwood Middle School