TILSA Alignment Tool Dissemination Workshop

1. TILSA Alignment Tool Dissemination Workshop July 25 and 26, 2005 WYNDHAM Hotel Boston, Massachusetts Funded by the U.S. Department of Education through a contract to the state of Oklahoma and subcontracts to CCSSO, WCER, HumRRO, and Tindal.

2. Alignment Powerful Tool for Focusing Instruction, Curricula, and Assessment

3. AgendaJuly 25, 2005 8:30 to Noon Speakers: Norman L. Webb Lauress L. Wise Gerald Tindal Noon Lunch 1:00 to 5:00 PM Concurrent sessions Session A: Using the WAT Session B: Interpreting reports and coordinating an alignment study

4. AgendaJuly 26, 2005 8:30 to 12:30 PM Concurrent sessions Session A: Using the WAT Session B: Interpreting reports and coordinating an alignment study 12:30 Lunch 1:30 to 3:00 Plenary Technical issues with the CD alignment system General questions and closing

5. Alignment Issues Vertical Alignment Grade to grade content linkages Lauress Wise Alternate Assessment Alignment Operationalize the process Gerald Tindal Webb Alignment Process Norman Webb, Rob Ely, Meredith Alt, & Brian Vesperman

6. Workshop Expectations Set up of an alignment study Responsibilities of a group leader Responsibilities of reviewers Coding procedures Special features How to conduct an alignment analysis of standards and assessments with the WAT

7. Alignment The degree to which expectations and assessments are in agreement and serve in conjunction with one another to guide the system toward students learning what is expected.

8.  Standards  Curriculum Assessment

9. Degree of Alignment Standards Standards Assessment Assess-ment Assessment Items Standards Assessment Standards

10. Alignment Process • Identify Standards and Assessments • Select 6-8 Reviewers (Content Experts) • Train Reviewers on DOK Levels • Part I: Code DOK Levels of the Standards/Objectives • Part II: Code DOK Levels and Corresponding Objectives of Assessment Items

11. Specific Criteria Content Focus A. Categorical Concurrence B. Depth-of-Knowledge Consistency C. Range-of-Knowledge Correspondence D. Balance of Representation and Source of Challenge

12. Depth of Knowledge Level 1 Recall Recall of a fact, information, or procedure. Level 2 Skill/Concept Use information or conceptual knowledge, two or more steps, etc. Level 3 Strategic Thinking Requires reasoning, developing plan or a sequence of steps, some complexity, more than one possible answer. Level 4 ExtendedThinking Requires an investigation, time to think and process multiple conditions of the problem.

13. Items by Objectives Report

14. What alignment is good enough?

15. Alignment Levels Using the Four Criteria

16. Categorical Concurrence State B Grade 8 Mathematics

17. Depth-of-Knowledge Consistency State B Grade 8 Mathematics

19. Balance IndexState B Grade 8 Mathematics

24. Coding Process Tips • One Primary Objective and up to Two Secondary Objectives (if necessary) • Source of Challenge (a correct/incorrect response for the wrong reason) • Notes (any insights to share) • Consider Full Range of Standards • Use generic objectives sparingly

25. Structure of the Automated Alignment Process Registration Group Leader Reviewers Standards/Goals/Objectives Entry Process Training on Depth-of-Knowledge Levels Phase I Consensus Process on Assigning DOK Levels to Objectives Phase II Coding of Assessment Tasks Phase III Analysis of Coding Phase IV Reporting

26. Web Sites http://facstaff.wcer.wisc.edu/normw/  Alignment Tool http: //www.wcer.wisc.edu/WAT/index.aspx Survey of the Enacted Curriculum http://www.SECsurvey.org 

27. EXAMPLE OF STANDARDS AND DEPTH-OF-KNOWLEDGE LEVELS CONTENT AREA: GEOMETRY

28. EXAMPLE OF STANDARDS AND DEPTH-OF-KNOWLEDGE LEVELSCONTENT AREA: PROBABILITY AND STATISTICS

35. New Cubes Your school is planning a casino night to raise funds to construct a wall aquarium in your school. As a mathematics student, you are given the job of developing a dice game for this event. A regular pair of “number dice” consists of two cubes, each with its faces numbered 1 through 6. Often, dice games are played by rolling the two dice and then finding the sum of the two numbers turned upward. 1. Show that, with a regular pair of number dice, the probability of rolling a sum of 7 is greater than the probability of rolling any other sum.

37. Coordination of an Alignment Institute Identify • Content areas • Grade levels • Number of test forms • Number of reviewers • Computer facilities • Standards and their structure

38. Coordination of an Alignment Institute Ask if • Tests include field test items • Items have different point values • Alternate assessments will be included • English Language Learners will be included

39. WAT Adoption to State Needs • Assessment development (front end alignment) • District and local assessments • Test to test comparison analysis • Curriculum to standard analysis

40. Grades 9–12 Science Objectives and Depth-of-Knowledge (DOK) Levels for Michigan Alignment Analysis

41. Comparison of Six Science Assessments on Categorical Concurrence

42. Comparison of Six Science Assessments on Balance of Representation

43. Cross-Assessment Summary

44. Three Analytic Methods • Common Framework • Expert Consensus • Common Criteria

45. Survey of the Enacted Curriculum

47. Enacted, Intended, and AssessedCurriculum Intended—What standards require Enacted—What teachers teach Assessed—What state tests

48. Achieve MatrixGrade 3 MathematicsData Analysis and Probability

49. Achieve Alignment Criteria

50. Alignment Process • Identify Standards and Assessments • Select 6-8 Reviewers (Content Experts) • Train Reviewers on DOK Levels • Part I: Code DOK Levels of the Standards/Objectives • Part II: Code DOK Levels and Corresponding Objectives of Assessment Items