Research and Theory: Introduction to the Scale or Rubric, Writing Measurement Topics, Score 3.0 and Sample Tasks

1. Research and Theory, Introduction to the Scale or Rubric, Writing Measurement Topics, Score 3.0 and Sample Tasks Ravalli County Curriculum Consortium Marzano Research Laboratory

2. Research and Theory behind the Standards Rubric or Scale

3. Why are we doing this? • Typical State and National Standards Documents simply have too much content to cover effectively and with any depth • Standards lack unidimensionality • The use of formative assessment data allows for the greatest gain in student achievement • Scales or rubrics provide the best feedback to students and are the most accurate way to assess student knowledge and growth

4. Typical Standards documents cover too much content! • If you wanted to teach all of the standards in the national documents, you would have to change school from K-12 to K-22! • 255 standards across 14 subject areas • 3,500 benchmarks • 15,500 hours of instruction needed to cover the 3,500 benchmarks (Kendall and Marzano, 2000) • 9,000 hours of instruction available (Marzano, Kendall and Gaddy, 1999)

5. Typical Standards documents lack Unidimensionality • Standards documents are not written with assessment in mind • Benchmarks typically cover more than one skill or piece of knowledge • In order to assess each individual dimension, standards and benchmarks must be “unpacked” so that they measure only one thing at a time

6. The use of formative assessment data can dramatically improve student achievement Black and Williams (1998) in a synthesis of 250 studies, concluded that the use of formative assessment data, done properly, “…does improve learning. The gains in achievement appear to be quite considerable, and as noted earlier, amongst the largest ever reported for educational interventions.”

7. Types of Feedback from Formative Assessments

9. Why use a rubric or scale for formative assessment and to assess student progress over time? • Formative assessment using rubrics provides the best form of feedback to students • Well written and specific scales or rubrics are more accurate than a set of tests or formal assessments graded on a 100 point scale

10. In a study conducted by Marzano in 2002, a group of science teachers independently developed a 100 point scoring scheme for a test and scored it. The reliability coefficient was only .294 (out of 1.0). • The resulting scores were so variable that the test could not reliably be used to assess student knowledge and/or skills

11. The same teachers scored the same test using a 4 point rubric or scale. • The reliability coefficient was .719 • This means that the resulting scores were significantly more consistent, and as a result the rubric came much closer to measuring what the student actually knows or is able to do than the test scored using the 100 point scale.

12. Using Standards Rubrics or Scales Will… • Create greater reliability in assessment of standards • Create more consistency in teaching, assessment and learning throughout a district or state • Allow for more consistent and better feedback throughout the course of learning • Allow for more focused instruction in appropriate depth • Make true assessment possible due to the creation of unidimensionality • IMPROVE LEARNING!

13. Questions? Comments? Discussion?

14. Writing Standards Scales or Rubrics

15. The Standards Scale or Rubric

16. The Scale

17. 4 Easy(?) Steps • Unpack/Cut Down Standards/Organize /Write Measurement Topics • Specify expected learning by writing Score 3.0, and write sample tasks to further specify content • Identify steps towards mastery of the content by writing Score 2.0, write sample tasks to further specify content • Pilot and Use! Design Assessments, Assess student learning, Report student progress

18. Measurement TopicsCategories of learning within each strand • Organize your standards into more precise categories of study (Measurement Topic) within each strand • The Measurement Topic represent the critical or essential content that you want to measure and record student progress on • You may need to leave some things out! • You will write scales or rubrics for each of the MT’s • Each MT will assess only one specific piece of information or skill for each grade level

19. Writing Measurement Topics • MRL highly recommends limiting the number of MT’s to no more than 30-35 per grade level • Write each measurement topic specific enough that it can be described by only 1 or 2 elements • Combine similar or covarying elements • Covarying: as the ability in one increase, so does ability in the second • Split up statements that are not unidimensional • Unidimensional: measuring only one concept or skill • Delete content that is not essential • Delete content that is not measureable • Reduce! You can’t assess it all!

20. Other Options • Reorganize at the High School level • Organize by class and write measurement topics for each class • Consider including a Teacher Choice rubric allowing teachers to assess something other than what was included

21. Measurement topics do not have to be assessed at every grade level! They can disappear and reappear at various levels

22. Writing Score 3.0

23. Writing Score 3.0 elements • What is a Score 3.0 element? • 3.0’s represent all of content that is explicitly taught in class under a measurement topic • This is what you expect a student to know and be able to do • This represents the parts of the curriculum that are being assessed

24. General Guidelines for Writing 3.0’s 3.0 elements: • have a procedural or declarative focus • Are unpacked – measure only one piece of knowledge or skill • Are learning goals, not tasks • Are measureable! • Are specific! • Are differentiated by grade level

25. Learning Goals vs. Learning Tasks • Score 3.0 elements are written in the form of learning goals, not learning tasks or activities • Learning goals specify the content or skills that the student is expected to know or do • Learning tasks or activities are the classroom activities that a student does in order to accomplish or be assessed on the learning goal.

26. Measurability • 3.0 elements must be measureable • Measureable: • Observable or Quantifiable • Makes predictions about reading • The teacher can observe a student doing this • Performs 10 push-ups with proper form • The teacher can count the number of pushup and observe proper form • Not Measureable • The student enjoys independent reading

27. Specificity of Elements • Elements must be written so that a teacher who is unfamiliar with the process and unfamiliar with the curriculum, knows what the student is expected to know or be able to do and can accurately assess a student! • Avoid terms that can’t be easily defined or allow for subjective interpretation • Terms can be defined within the element • Examples can also be used to provide specificity

28. Differentiation of Elements • Elements must be written to show a increasing level of complexity as the performance level increases • If the element remains the same through several performance levels, examples must be provided to differentiate each level • For Example – ELA will frequently assess the same reading strategies (e.g., inference) at multiple performance levels, but… the complexity of the expectation or text will change. Give examples of the type of inference, or level appropriate text to differentiate

29. Unidimensional Rubrics • For each measurement topic, we are going to write one (maybe 2 if you beg!) Score 3.0 element • Each element should measure only one individual piece of knowledge or skill • If the topic requires more than one (or 2) elements, then break the topic into two or more new topics to maintain unidimensionality • Remember, you can’t measure it all! You will have to make decisions about what to include and measure and what not to include and measure.

30. Writing Sample Tasks

31. What are sample tasks? • Sample Tasks are examples of assessment items or tasks that teachers can use to construct formative assessments • Sample Tasks are written to present a model of an assessment task to help clarify the intent of the element, not as a definitive list of assessment tasks • Assessment items that are written to address the appropriate taxonomic level for each element and score level

32. The Purpose of Writing Sample Tasks Sample tasks: • Help to clarify the expectations of each element • Provide instructional guidance to the teacher • Suggest ways to assess an element • Can be used to design formative assessment • Can be used to construct a common assessment item bank

33. Writing Sample Tasks Ask yourself some basic questions: • How can a student demonstrate success for this element? • What kinds of learning activities or tasks would assess success in this element? • What taxonomic operation would be appropriate for this task at this grade level?

34. General Rules • When explaining or other higher level processes are involved in sample tasks, ensure that the students are explaining how or why something is happening and not just that it happens. If the elements are worded correctly this should not be a problem. • If an element refers to memorization, a sample task must say something like, produce from memory. • Each bulleted element will generally have a corresponding sample task. • Be specific!!!!

35. Organizing the Work • We suggest that you split the work up into several teams, each with representatives from lower elementary, upper elementary and middle school. High school reps are helpful, if possible. • Each team would be responsible for writing the scales or rubrics K-8 within a measurement topic. This helps to ensure vertical alignment. A high school rep is helpful to make sure that the student ends up where they need to be! • High school teams can work to create rubrics by course.

36. Our Job today: • Review Draft MT’s and take a first cut at revising them. You can and will continue to revise as you develop the rubrics • Develop the Learning Goal (3.0) for each of the MT’s at each grade level • YOU WILL NOT FINISH THIS TODAY!