PARCC Core Leadership Group Item Review Meeting January 2013

1. PARCC Core Leadership Group Item Review Meeting January 2013 Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

2. Overview of Training • Charge as Committee Member • Purpose of PARCC Summative Assessments • PARCC Summative Assessments (PBA, EOY) • Item Review Process • Item Review Criteria • Apply Criteria 1-9 to Item Review

3. Item Review Committee Charge • Your role is to provide expert CONTENT review of items and tasks. • You will use information provided in this item review training and will apply item review criteria to review the items. • You should focus exclusively on PARCC’s item review criteria during your review.

4. Item Review Committee Please also note the following: • Passage review committees have already approved the passages according to PARCC content and bias/sensitivity guidelines. • Bias/sensitivity item review committees will apply bias/sensitivity guidelines to all items at a different time. • Other concerns identified will be placed in “the parking lot” for consideration by PARCC leadership.

5. Purpose of PARCC Summative Assessments • Determine whether students are college- and career-ready or on track • Assess the full range of the Common Core State Standards (CCSS) for reading, writing, and language • Measure the full range of student performance, including the performance of high- and low-performing students • Provide data for accountability, including measures of growth • Incorporate innovative approaches throughout the system

6. Performance-Based Assessment Composed of Three Tasks • Literary Analysis Task (LAT) • Research Simulation Task (RST) • Narrative Task (NT) Eligible Item Types for Performance-Based Assessment (PBA) • Evidence-Based Selected Response (EBSR) • Technology-Enhanced Constructed Response (TECR) • Prose-Constructed Response (PCR)

7. Performance-Based Assessment A. Literary Analysis Tasks—The Literature Task plays an important role in honing students’ ability to read complex text closely, a skill that research reveals as the most significant factor differentiating college-ready from non-college-ready readers. This task will ask students to carefully consider literature worthy of close study and compose an analytic essay. B. Research Simulation Task—The Research Simulation Task is an assessment component worthy of student preparation because it asks students to exercise the career- and college- readiness skills of observation, deduction, and proper use and evaluation of evidence across text types. In this task, students will analyze an informational topic presented through several articles or multimedia stimuli, the first text being an anchor text that introduces the topic.  Students will engage with the texts by answering a series of questions and synthesizing information from multiple sources in order to write two analytic essays. C. Narrative Writing Task—The Narrative Task broadens the way in which students may use this type of writing. Narrative writing can be used to convey experiences or events, real or imaginary. In this task, students may be asked to write a story,  detail a scientific process, write a historical account of important figures, or to describe an account of events, scenes or objects, for example. Pg 13 Item Guidelines

8. End-of-Year Assessment Focused on supporting Reading Comprehension Claims • Item development must be informed by CCSS and Evidence Statements Eligible Item Types for End-of-Year (EOY) • Evidence-Based Selected Response (EBSR) • Technology-Enhanced Constructed Response (TECR)

9. Evidence-Based Selected Response (EBSR) Items: EBSR items: • Are designed to measure Reading Standard 1 and at least one other Reading Standard. • May have one part. • In grade 3, a one part EBSR is allowable because Reading Standard 1 evidence 1 is distinctly different from Reading Standard 1 in grades 4-11. • In grades 4-11, a one part EBSR is allowable when there are multiple correct responses that elicit multiple evidences to support a generalization, conclusion or inference. Pgs 27 -29 Item Guidelines

10. Evidence-Based Selected Response (EBSR) Items: • May have multiple parts. • The Item Guidelines document describes how PART A and PART B may function in a multiple part EBSR. • Among things to consider: • Each part must consist of a selected-response question with a minimum of four choices. • In the first part, students select a correct answer among a minimum of four choices. (continued on next slide) Pgs 27 -29 Item Guidelines

11. Evidence-Based Selected Response (EBSR) Items: • In additional parts, one format requires students to select among a minimum of four choices to demonstrate the ability to locate and/or connect details/evidence from the text that explains, justifies, or applies the answer chosen for the first part of the item. • There is no requirement for one-to-one alignment between Part A options and Part B options. • EBSR items can meet evidence statements that specify “explain,” “provide an analysis,” etc., by selecting options that comprise explanation or analysis.

12. Evidence-Based Selected Response (EBSR) Items: Variable Elements EBSR items may possess the following characteristics: • For items with one correct response, four answer choices are requisite. For those items with two correct responses, six answer choices are requisite. For those items with three correct response (allowed only in grades 6-11), seven answer choices are requisite. Pgs 27 -29 Item Guidelines

13. Evidence-Based Selected Response (EBSR) Items: • For items with one correct response in Part A and one correct response in Part B, there is no partial credit. One part items do not offer partial credit. • For those items with one or more correct responses in Part A and more than one correct response in Part B, partial credit should be available. • To receive partial credit, students must answer Part A correctly AND select at least one correct response in Part B. This will earn the student 1 point. For these items, to receive full credit, students must answer both Part A and Part B correctly. • When an item allows for more than one correct choice, each correct choice must be equally defensible. Pgs 27 -29 Item Guidelines

14. Evidence-Based Selected Response (EBSR) Items: EBSR itemscan be written with an inference to be drawn in Part A and a requirement for students to find another example of how that inference applies in a different part of the text for Part B. Example 1: Part A – What character trait does Snow White reveal when Snow White does X? Part B—Which paragraph from the passage best shows additional evidence of this character trait? Example 2: Part A—What theme is revealed in the passage? Part B—Which paragraph from the passage best shows this same theme? Example 3: Part A—What is the point of view/perspective in this passage? Part B—Which paragraph from the passage best shows this same point of view/perspective (or the opposite point of view/perspective)? See the Item Guidelines document pages 27 -29 for other types of EBSR items. Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

15. Additional Considerations for Distracters • The primary purpose of a distracter is to provide evidence that a student is not able to meet the standard(s) assessed due to student misconceptions. • Distracters must be plausible responses to item stems. • The written style of all distracters in an item should be similar to that of the correct response(s), but need not be “perfectly parallel” in length, grammatical function, or in use of punctuation. • The content of the distracters, rather than the parallelism of style, is the primary focus for distracter choices. • Answer responses are not ordered alphabetically by first word or from short to long, etc. They may be ordered in any sequence as is appropriate to the content measured by the specific item.

16. Additional Considerations for Distracters • If answer responses are quotations or paraphrased textual evidence, place the answers in the order they appear in the passage. • Particular care must be taken for Part B in EBSR items (where students are asked to select evidence from the text) such that achieving parallelism in distracters does not overly influence distracter wording. • In Part B, when writing the distracters for evidences, all of the answer choices must be the same type of citation of evidence (e.g. all quotes or all paraphrases). • All answer choices for Part B (distracters) must be accurate/relevant/from the passage (whether exact citations or paraphrases). All distracters must originate accurately from the text. In Part A, distracters may be written as plausible misreadings of the text.

17. Part A: What does the word “regal” mean as it is used in the passage? generous threatening kingly* uninterested Part B: Which of the phrases from the passage best helps the reader understand the meaning of “regal?” “wagging their tails as they awoke” “the wolves, who were shy” “their sounds and movements expressed goodwill” “with his head high and his chest out”* Sample Evidence-Based Selected Response (EBSR) Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

18. Technology-Enhanced Constructed Response (TECR) Items: TECR items must: • Allow for machine scoring • Award the student two (2) points for full credit • Be delivered and responded to using technology, allowing for a variety of technology-enhanced student responses, including but not limited to the following: • drag and drop • highlighting the text • annotating text • other negotiated methods Pgs 29 -31 Item Guidelines

19. Technology-Enhanced Constructed Response (TECR): TECR items may possess the following characteristics: • When a TECR uses an EBSR structure (e.g. with Part A[measuring one or more of standards 2-9] and Part B [measuring standard 1]), use the same rules as applied for EBSR. • Use the same guidelines for distractors for EBSR items for TECR items where applicable. • For TECR items, partial credit may be offered when an item allows for partial comprehension of the texts to be demonstrated. Pgs 29 -31 Item Guidelines

20. Part A : Below are three claims that one could make based on the article “Earhart’s Final Resting Place Believed Found.” Highlight the claim that is supported by the most relevant and sufficient evidence within “Earhart’s Final Resting Place Believed Found.” Part B : Click on two facts within the article that best provide evidence to support the claim selected in Part A. Sample Technology-Enhanced Constructed Response (TECR) Item Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

21. Prose Constructed-Response (PCR): Required Elements PCR items must: • Visibly align questions/tasks with specific Standards; that is, the actual language of the Standards should be used in the prompts/questions • Elicit evidence(s) supporting the Sub Claim for Written Expression and the Sub Claim for Conventions and Knowledge of Language Pgs 32 -33 Item Guidelines

22. Prose Constructed-Response (PCR): Required Elements PCR items must: • Establish a clear purpose for writing, modeling the language found in the Writing Standards • Specify the audience to be addressed • State clearly the topic, issue, or idea to be addressed • Reference the source text (or texts) serving as the stimulus (or stimuli) for a student response • Specify the desired form or genre of the student response Note: Standardized wording for PCRs is under discussion. Pgs 32 -33 Item Guidelines

23. Prose Constructed-Response (PCR): Required Elements • Elicit evidence(s) aligned with at least one Reading Standard (even when not scored for a sub claim associated with the Major Claim for Reading Complex Text) • Allow students to earn partial credit Pgs 32 -33 Item Guidelines

24. Prose Constructed-Response (PCR): Required Elements • In addition, prose constructed-response items must provide all students the opportunity to demonstrate a full range of sophistication and nuance in their responses. • Prose constructed-response items must be designed to elicit meaningful responses on aspects of a text that may be discussed tangentially or in great detail and elaboration, thereby enabling measurement of the full range of student performance. Pgs 32 -33 Item Guidelines

25. Prose Constructed-Response (PCR): Narrative Description Narrative writing takes two distinct forms in the PARCC assessment system: Narrative Story Narrative Description*

26. Prose Constructed-Response (PCR): Narrative Story Narrative Story – is about imagined situations and characters. It uses time as its deep structure.

27. Prose Constructed-Response (PCR): Narrative Story- Scenarios • Scenarios should be associated with the text and should culminate by stating the objective of the final prose constructed response for the task. • Scenarios for Narrative Writing –Narrative Story Task Model : • Today you will read [fill in the text type/title]. As you read, pay close attention to [fill in general focus of PCRs] as you answer the questions to prepare to write a narrative story. Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

28. Sample Prose-Constructed Response (PCR) Item Use what you have learned from reading “Daedulus and Icarus” by Ovid and “To a Friend Whose Work Has Come to Triumph” by Anne Sexton to write an essay that analyzes how Icarus’s experience of flying is portrayed differently in the two texts. As a starting point, you may want to consider what is emphasized, absent, or different in the two texts, but feel free to develop your own focus for analysis. Develop your essay by providing textual evidence from both texts. Be sure to follow the conventions of standard English. Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

29. Additional Considerations for Vocabulary Items • Several styles for presenting vocabulary words/phrases are viable. In considering which means is best for presenting the vocabulary words/phrases, item writers should use the means that most efficiently directs students to the word/phrase in the text, while allowing students to see the full relevant context for determining the meaning of the word/phrase. • The part of a vocabulary item that asks for word or phrase meaning should not use qualifiers (e.g., best, most likely: Which of the phrases from the excerpt besthelps the reader understand the meaning of “XXX”? • The part of a vocabulary item that asks for support may use qualifiers if needed: Which sentence from the excerpt best supports the response in Part A? • Distractors should always be syntactically plausible. This is especially important in vocabulary items.

30. Additional Considerations for Vocabulary Items • When writing vocabulary items use the following formats: Part A - “What is the meaning of the word XXX as it is used in [paragraph 13, or line, or text]?—Part A wording Part B - “Which of the phrases from the excerpt helps the reader understand the meaning of XXX? Unless referencing in this way creates a problem for the item. In this case, the item may require the use of a text box approach where the options for Part B come only from the excerpted text. • Use of Technology Enhancement: • To measure vocabulary using a drag/drop technology use the following format: • “Drag the sentences from the passage into your notes that help create the meaning of the word “XXX” as it is used in the passage.” • “Drag the words/phrases from the passage into your notes that help create the tone of the passage.” Note: Give a selection of sentences or words/phrases from which to choose where not all are correct.

31. Item Review Process Step 1 - Each reviewer on your team will read the passage/text and the items independently. Step 2 - For each item, check to make sure the item meets the evidence statements/standards noted in the metadata for the item. If the item is aligned, continue to review the item. If the item is not aligned, but can be edited to align with the evidence statements/standards, note in comments how to align the item. If the item cannot be aligned, reject the item and move on to review the next item. Step 3 - Review the item in terms of other criteria on the criteria evaluation sheet. Make comments to reflect needs for revision/strengths noted.

32. Item Review Process Step 4 - The facilitator will go item by item to determine which items are to be accepted, accepted with edits, or rejected. If all members of a group have marked an item “accepted” or “rejected,” no discussion occurs. Participants discuss those items where there is not 100% agreement. Step 5- Allow a maximum of 10 minutes per item for consensus building to determine whether to accept, accept with edits, or to reject an item. If consensus cannot be achieved, move on. The team will return to these items on the last day of the review. (continued)

33. Item Review Criteria • Does the item allow for the student to demonstrate the intended evidence statement(s) and to demonstrate the standard(s) to be measured? • Is the wording of the item clear, concise, and appropriate for the intended grade level? • Does the item provide sufficient information and direction for the student to respond completely? • Is the item free from internal clueing and miscues? • Do the graphics and stimuli included as part of the item accurately and appropriately represent the applicable content knowledge?

34. Item Review Criteria • Are any graphics included as part of the item clear and appropriate for the intended grade level? • If the item has a technology-based stimulus or requires a technology-based response, is the technology design effective and grade appropriate? • Is the scoring guide/rubric clear, correct, and aligned with the expectations for performance that are expressed in the item or task? • If the item is part of a PBA task, does it contribute to the focus and coherence of the task model? (continued)

35. Criterion 1. Alignment to the CCSS and Evidence Statements Does the item allow for the student to demonstrate the intended evidence statement(s) and to demonstrate the CCSS to be measured?

36. Criterion 1. Alignment to the CCSS and Evidence Statements An item should: • be aligned to more than one CCSS and corresponding evidence statements. • Focus is no longer on a one-to-one relationship between an item and tested skill • All reading comprehension items will align to a focus standard as well as RL 1 or RI 1 • elicit the intended evidence(s) to demonstrate ability with the skills expressed in the standard(s). • Items should focus on important aspects of the text and be worth answering • Items are intended to provide evidence to support the claims

37. Criterion 1. Alignment to the CCSS and Evidence Statements Vocabulary items should: • target Tier 2 academic vocabulary • Words with wide use across academic subjects • Words that are important for students to know • have distracters in the same part of speech as the assessed word • elicit evidence from at least one language standard and from reading standard RI1 or RL1 Pgs 25-26 Item Guidelines

38. Criterion 1. Alignment to the CCSS and Evidence Statements Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

39. Criterion 2. Language Clarity and Appropriateness Is the wording of the item clear, concise, and appropriate for the intended grade level?

40. Criterion 2: Language Clarity and Appropriateness Items should: • convey a clearly defined task or problem in concise and direct language • use the language of the evidence statements and standards when appropriate • focus on what is important to learn rather than on trivial content • use a variety of approaches rather than a canned approach

41. Criterion 2: Language Clarity and Appropriateness Example from Grade 10: Poor Clarity What was the problem that Galileo’s experiment did not solve? the influence of air pressure on the tubes’ water level Example from Grade 10: Improved Clarity According to the excerpt, what problem did Galileo’s experiment fail to solve? the influence of air pressure on the glass tube’s water level Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

42. Criterion 3. Clarity of the Student Directions Does the item provide sufficient information and direction for the student to respond completely?

43. Criterion 3: Clarity of the Student Directions Items should: • pose the central idea in the stem and not in the answer choices • provide directions that are specific and direct • require no background knowledge

44. Criterion 3: Clarity of the Student Directions Example of Poor Clarity in Student Directions Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

45. Criterion 3: Clarity of the Student Directions Example of Improved Clarity in Student Directions Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

46. Criterion 4: Avoiding Clueing and Miscues Is the item free from internal clueing and miscues?

47. Criterion 4: Avoiding Clueing and Miscues • Options should NOT echo a stem word. • Distractors should be plausible so that students will consider them carefully. • Items should not be answerable by reading other items in the set.

48. Criterion 4: Avoiding Clueing and Miscues Example of Miscue: How is the life cycle of a frog similar to the life cycle of the butterfly? • Both turn into a pupa. • Both breathe with gills. • Both begin life as an egg.* • Both break out as a chrysalis. Note: all items included in this presentation are for illustrative training purposes only. They are not representative of PARCC assessment items.

49. Criterion 5: Accuracy of Content Represented by Graphics Do the graphics and stimuli included as part of the item accurately and appropriately represent the applicable content knowledge?

50. Criterion 5: Accuracy of Content Represented by Graphics Items should: • not include graphics simply for the sake of including graphics. • use graphics that allow students to understand the problem or task or demonstrate ability to solve a problem or task. • elicit the intended evidence(s) to demonstrate ability with the skills expressed in the standard(s).