2015 MCAS-Alt: How to Submit a “ Complete ” Portfolio

1. 2015 MCAS-Alt:How to Submit a “Complete” Portfolio Karen Orlando MCAS-Alt Teacher Consultant

2. Topics • Top five reasons portfolios were scored Incomplete • Other reasons for an Incomplete score • Checking for “completeness” • Frequently-asked questions Massachusetts Department of Elementary and Secondary Education

3. Top Five Reasons for Incomplete Portfolios Massachusetts Department of Elementary and Secondary Education

4. Top Five Reasons For Incomplete Portfolios • Some or all brief description(s) did not address the measurable outcome. As a result, student did not address measurable outcome on at least 8 dates. • Brief descriptions were not included or did not provide enough information on at least 8 dates. • Titles and/or samples of text used with student were missing in the ELAReading strand. • At least two pieces of primary evidence were not submitted that related to measurable outcome listed on the data chart. • No evidence of self-evaluation in the strand. Massachusetts Department of Elementary and Secondary Education

5. Top Five Reasons For Incomplete Portfolios Some or all brief description(s) did not address the measurable outcome. As a result, student did not address the measurable outcome on at least 8 dates. Massachusetts Department of Elementary and Secondary Education

6. Walter will identify basic forms of energy with 75% accuracy and 75% independence. Massachusetts Department of Elementary and Secondary Education

7. Student will answer simple comprehension questions to demonstrate the understanding of an informational text with 80% accuracy and 80% independence. Massachusetts Department of Elementary and Secondary Education

8. Page 35 2015 Educator’s Manual for MCAS-Alt Brief Descriptions Unmatchedto Measurable Outcome • Page 35: “…include a brief description beneath each data point that clearly describes what the student did and how the student addressed the skill, taking particular care to document that the activity matched the measurable outcome.” • Page 35: “…assess and record data only on those skills directly based on the measurable outcome. The data will be inconclusive, and the chances of scoring Incomplete will increase, when unrelated or multiple skills are included on the same data chart. Massachusetts Department of Elementary and Secondary Education

9. Creating a Measurable Outcome from Entry Points that Include Related Skills Example: Entry Point as written: “Solve number sentences that represent one-step multiplication and division word problems.” Option 1: • Create a measurable outcome that addresses both skills. • All activities must document “solving number sentences involving one-step multiplication AND division problems.” Option 2: • Modify the measurable outcome: “Solve number sentences that represent one-step multiplication word problems.” • All activities must document “solving number sentences involving one-step multiplication problems.” Massachusetts Department of Elementary and Secondary Education

10. Which descriptions are scorable? Measurable Outcome: Student will solve addition and subtraction word problems with 80% accuracy and 100% independence. Massachusetts Department of Elementary and Secondary Education

11. Clear and Specific Descriptions • Score accuracy and independence only on the skill related to the measurable outcome. Clearly indicate this on the work description label. Example: • Skill measured in the measurable outcome: “Answer questions about bar graphs” • Activity: Student creates a bar graph from the results of a class survey, then answers questions about the graph. • Scorable Description: “After creating a bar graph from a class survey, student answered questions about the graph. (Accuracy and independence were based on questions only.)” • Unscorable Description: “Student created a bar graph and answered questions.” Massachusetts Department of Elementary and Secondary Education

12. Top Five Reasons For Incomplete Portfolios 2. Brief descriptions were not included or did not provide enough information on at least 8 dates. Massachusetts Department of Elementary and Secondary Education

13. Walter will compare fractions using the terms “greater than,” “less than, or “equal to” with 80% accuracy and 80% independence. Massachusetts Department of Elementary and Secondary Education

14. Pages 35 and 40 2015 Educator’s Manual for MCAS-Alt Scorable Brief Descriptions • Page 35: “include a brief description beneath each data point that clearly describes what the student did and how the student addressed the skill, taking particular care to document that the activity matched the measurable outcome; Massachusetts Department of Elementary and Secondary Education

15. Pages 35 and 40 2015 Educator’s Manual for MCAS-Alt Scorable Brief Descriptions continued • Page 40: “Brief descriptions are provided below each learning activity that match the measurable outcome, and describe what the student was asked to do to address the measurable outcome, and how they did it. The following are examples of appropriate brief descriptions:… Massachusetts Department of Elementary and Secondary Education

16. Acceptable Brief Descriptions • Measurable Outcome for STE- Technology/Engineering • Walter will identify simple machines with 80% accuracy and 100% independence. Massachusetts Department of Elementary and Secondary Education

17. These brief descriptions DO NOT provide enough information to know what the student was asked to do, or how he/she did it. Massachusetts Department of Elementary and Secondary Education

18. Which descriptions are scorable? Outcome: After listening to story or article, student will answer simple comprehension questions to demonstrate understanding of an informational text with 80% accuracy and 80% independence. Massachusetts Department of Elementary and Secondary Education

19. Top Five Reasons For Incomplete Portfolios Titles and/or samples of text used with student were missing in the ELAReading strand Massachusetts Department of Elementary and Secondary Education

20. Titles of text are missing! Massachusetts Department of Elementary and Secondary Education

21. Pages 15-19 and 41 2015 Educator’s Manual for MCAS-Alt Cite the Title or Include Actual Text • Page 15-19: “The title or a photocopy of the text used for assessment must be included in all brief descriptions for ELAReading; a copy of any teacher-created text must be submitted” • Page 41: “Evidence or brief descriptions in the ELAReading strand must refer by name to the text from which words, phrases, or excerpts were selected for assessment, and must include either: • the title of the published text, or • the actual text (e.g., sentence, passage, narrative, etc.), or • a photocopyor printout, if text is teacher-created, untitled, or includes selections from a print or digital source (e.g., websites) Massachusetts Department of Elementary and Secondary Education

22. Options for Citing the Text • Provide titles. • Provide copy of teacher-created text, worksheets, or selections from websites. • Attach a list of text used to data chart with dates and names of text used. • Include topic of the text if it is from a specific reading program, rather than just name of program. • Include title of text and/or topic if it is from a specific website, rather than just the name of the website. • For untitled texts include a photocopy Massachusetts Department of Elementary and Secondary Education

23. Top Five Reasons For Incomplete Portfolios At least two pieces of primary evidence were not submitted that relate to the measurable outcome listed on the Strand Cover Sheet Massachusetts Department of Elementary and Secondary Education

24. Student will compare and contrast characters in a story with 90%accuracy and 80%Independence Massachusetts Department of Elementary and Secondary Education

25. EVIDENCE #1 Student Massachusetts Department of Elementary and Secondary Education

26. Student EVIDENCE #2 Student is answering questions about characters, not comparing and contrasting them. Massachusetts Department of Elementary and Secondary Education

27. Pages 12-19 and 33 2015 Educator’s Manual for MCAS-Alt Two Pieces of Primary Evidence • Pages 12-19 “…two additional pieces of primary evidence showing the student’s performance of the same targeted skill identified on the data chart.” • Page 33 “A minimum of one data chart and two pieces of additional primary evidence (called the “core set of evidence”)…” Massachusetts Department of Elementary and Secondary Education

28. Top Five Reasons For Incomplete Portfolios 5. No evidence of self-evaluation was found in this strand. Massachusetts Department of Elementary and Secondary Education

29. Self-Evaluation Either… • Evidence was either missing altogether, or • Showed no evidence of student choice • Stickers • Stamp • “Student smiled” • “Student traded in for music” Massachusetts Department of Elementary and Secondary Education

30. Pages 39 2015 Educator’s Manual for MCAS-Alt Self-Evaluation • “Self-evaluation activities document the student’s choices, decisions, and involvement before, during, and after instruction, including evidence that the student performed…” Massachusetts Department of Elementary and Secondary Education

31. Examples of Self-Evaluation include: • planning and goal setting • choosing an activity or next steps in an activity • selecting a problem-solving strategy • monitoring own progress or use of a strategy (e.g., by checking off steps as each is completed) • deciding when to continue or end participation in an activity • identifying and correcting his/her own errors • graphing own performance or progress on a chart • determining own score using a rubric • selecting work for his/her own portfolio • reflecting on his or her performance; for example, teacher asks student to respond to: • What did I do well? What am I good at? Was this too easy? • What did we do during this activity? What did I learn? • How could I do better? Where do I need help? • What should I work on next? What would I like to learn? Massachusetts Department of Elementary and Secondary Education

32. Massachusetts Department of Elementary and Secondary Education

33. Other Reasons for Incomplete • Literature and Informational text were combined in a single strand. The Reading strand must be one or the other. • Data chart did not document student performance on at least 8 different dates. • Both Accuracy and Independence began above 80-100%. • No data chart was submitted. • Data were unclear or could not be interpreted by the scorer. • Percent accuracy and/or independence were not included on primary evidence and/or could not be determined by scorer. • Student's name and/or date (month, day, year) was missing on one or more pieces of primary evidence, or was incorrect on Work Sample Description label. Massachusetts Department of Elementary and Secondary Education

34. Level of Complexity = 1 Massachusetts Department of Elementary and Secondary Education

35. Page 48 Educator’s Manual for MCAS-Alt What Does Level of Complexity = 1 Mean? • Strand must assess standards required for a student in that grade. • Data chart must document student performance of an acceptable measurable outcome in the required strand/domain. (Teachers cannot make up or use outdated entry points!) • ELA Reading is comprehension of words, phrases, sentences, passages based on a text. • Access skill must be taught in the context of an academic activity. Massachusetts Department of Elementary and Secondary Education

36. Skill Not Taught in the Context of an Academic Activity Massachusetts Department of Elementary and Secondary Education

37. Earth and Space Science is not assessed in High School Massachusetts Department of Elementary and Secondary Education

38. Checking for Completeness Massachusetts Department of Elementary and Secondary Education

39. Pages 35-36, Educator’s Manual for MCAS-Alt 12 Key Questions to Check for Completeness • Does the strand include a data chart and at least two pieces of primary evidence • Is the first data point on the data chart below 80 percent accuracy and/or independence? • Does the data chart have at least 8 different valid dates? • Is the student’sname, date, % accuracy, and % independence listed on the data chart and on at least two pieces of primary evidence (or on a Work Description label)? Massachusetts Department of Elementary and Secondary Education

40. 12 Key Questions to Check for Completeness (continued) 5. Do brief descriptions on the data chart state what the student was asked to do (i.e., what was the activity?) and how the activity was conducted (instructional approach, materials, etc.) on at least 8 different dates ? 6. Do brief descriptions on at least 8 different datesaddress the skill identified in the measurable outcome? 7. Do at least two pieces of primary evidence directly address themeasurable outcome? Massachusetts Department of Elementary and Secondary Education

41. 12 Key Questions to Check for Completeness (continued) 8. Are all skills listed in the measurable outcome (e.g., “addition and subtraction”) addressed in at least 8 activities on the data chart and two pieces of evidence (all skills must be addressed in each activity). 9. If photographs or video are included as primary evidence, do they show a final product and are they clearly labeled? 10. If a teacher-scribed work sample is included as primary evidence, does it provide sufficient information to determine what the student did and how the student did it? Massachusetts Department of Elementary and Secondary Education

42. 12 Key Questions to Check for Completeness For ELA-Reading Strands Only: 11. Do brief descriptions include titles of all texts? 12. Do all activities in the strand include eitherInformational or Literary text (but not both)? Massachusetts Department of Elementary and Secondary Education

43. Common Questions Massachusetts Department of Elementary and Secondary Education

44. How Do I Determine Accuracy and Independence? • First, determine what the outcome is. • What are you asking the student to do? • Then, determine the activity the student will perform to address that outcome. • Next, divide that activity into “items” or steps. • Determine the overall accuracy (correctness) and independence (i.e., no prompts needed) for the entire activity: % Independence = Number of “items” performed independently Number of possible responses Massachusetts Department of Elementary and Secondary Education

45. Page 38 2015 Educator’s Manual for MCAS-Alt Examples of “Items”:Outcome:Andre will answer simple comprehension questions in relation to a text with 80% accuracy and 100% independence.Activity:After reading Chapter 1 in “Frog and Toad Are Friends” by A. Lobel, with support of visuals and a reading checklist, student answered 4 comprehension questions on a worksheet. Massachusetts Department of Elementary and Secondary Education

46. Massachusetts Department of Elementary and Secondary Education

47. How many questions? 5 • How many correct responses? 2 P • What is the accuracy score? 2/5 or 40% I I • How many independent responses? 4 I • What is the independence score? 4/5 or 80% I Massachusetts Department of Elementary and Secondary Education

48. Pages 37-39, 50 Educator’s Manual for MCAS-Alt Independence: Prompts vs. Accommodations The following are cues and prompts since they guide the student toward the answer: • Verbalassistance (coach or tell: ”look on page…”) • Gesture (point or indicate where to find answer) • Visual cue (provide a sample problem or model) • Full or partial physical assistance (including hand-over-hand assistance) Any Cue or Prompt =a “non-independent response” Massachusetts Department of Elementary and Secondary Education

49. Page 39, 2015 Educator’s Manual for MCAS-Alt What is Generalized Performance? 2015 Educator’s Manual: • Page 39: “Generalized performance reflects the student’s ability to transfer knowledge and/or skills to other learning situations through application of a range of instructional approaches andactivity formats, including any of the following instructional elements: • media and materials (e.g., using art materials, written text, manipulatives, computer) • activity formats (e.g., classroom projects, small group discussions, paired research, experiments) • presentation formats (e.g., oral, written, multimedia) • method of response (e.g., handwritten, word-processed, oral, creation of a visual display, video)” Massachusetts Department of Elementary and Secondary Education

50. Page 37, 2015 Educator’s Manual for MCAS-Alt What is a Teacher-Scribed Work Sample and When Should it Be Used? • One method of documenting performance of a student who does not produce written work • Indicates what the activity was, what response options were offered to student, and how he/she responded on each item (accurately, independently?) • Documents student’s performance of a series of tasks recorded by the teacher at one time. Massachusetts Department of Elementary and Secondary Education