Integrated Document Analysis Workshop

1. Integrated Document Analysis Workshop Washington, DC May 5-7, 2008 Describing the Content & Language Demands of Standards, Assessments & Student Work Peter Worth, Assessment Standards & Development Services, WestEd, CA John L. Smithson, Director, Measures of the Enacted Curriculum Wisconsin Center for Education Research, University of Wisconsin-Madison

2. A neutral linguistic taxonomy A neutral content language Topics by Cognitive Demand Linquistic Complexity by Language Demands Analyzing Alignment The intended curriculum: State content standards—What students should learn The assessed curriculum: State (and other) assessments—tested learning The learned curriculum: Student outcomes based on school learning The enacted curriculum: What teachers teach

3. Content and Language Coding Form

4. Content and Language Coding Grade 8 mathematics item

5. Content and Language Coding Form

6. K-12 Mathematics Content

7. K-12 Mathematics Content Basic Algebra

8. The Goal To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of the three domains of a standards-based approach to education reform and their relationship to one another.

9. To Facilitate Comparisons Academic Content ELL Lang. Acquisition Assessments Academic Content ELL Lang. Acquisition Curriculum Alignment Academic Content ELL Lang. Acquisition Standards

10. The Two Dimensions Of Content What students should know [Topics] And… Be Able to Do [Expectations for student performance]

11. Describing Instructional Content SEC utilizes a two-dimensional taxonomy based on: Topic by Cognitive Demand

12. The Content Matrix

13. The Content Matrix

14. … adding levels of relative emphasis yields a 3-D construct

15. Describing the Cognitive Domain How Many Categories? 3 SCASS Science 4 DOK (Webb) SEC 5 6 Bloom’s

16. Dimensions of Knowing & Inquiry Acquire Use Extend (From: Dimensions of Knowing and Inquiring about Science, State Collaborative on Assessments & Student Standards Science Project, Council of Chief State School Officers, 1997)

17. Depth of Knowledge Level 1 2 3 4 Recall Skill/Concept Strategic Thinking Extended Thinking * Webb, N. 1999. Alignment of Science and Mathematics Standards in Four State. NISE Research Monograph #18. Madison:Wisconsin Center for Education Research.

18. Exploring Cognitive Demand Acquire Use Extend Recall Skill/Concept Strategic Thinking Extended Thinking

19. Cognitive Demand (or Expectations for Student Performance) Acquire Use Extend Recall Skill/Concept Strategic Thinking Extended Thinking Memorize Perform Procedures Demonstrate Understanding Conjecture, Generalize Prove Solve non- routine/ make connections Memorize Conduct Investigations Communicate Understanding Analyze Information Apply concepts /make connections Recall Perform Procedures Generate /Demonstrate Analyze/ Investigate Evaluate

20. Cognitive Demand (or Expectations for Student Performance) Acquire Use Extend Recall Skill/Concept Strategic Thinking Extended Thinking Memorize Perform Procedures Demonstrate Understanding Analyze Information Evaluate/Apply Recall Understanding Application Analyze Evaluate Create

21. Practice Session #1 Exploring Cognitive Demand But first…. Let’s take a 10 minute break!

22. Exploring Cognitive Demand CgD Immersion Activity • Organize into Groups/Tables • Each Table w/ CgD Pie • Each Person w/ Cgd Descriptors Step 1: Place CgD cards on Pie Slices face-down Step 2: Turn cards over: ID agreements e.g. 2 cards w/ same descriptor in same slice if Group Agrees ... discuss key words if not … Discuss … operational definition to distinguish Step 3: Discuss disagreements if consensus reached put in envelope / if not, set aside

23. Content Analysis Materials • Cognitive Demand List • Topics Lists • Comments & Suggestions Worksheet • Coding Forms • Documents to be analyzed

24. Cognitive Demand Lists • Five categories of cognitive demand • Slightly different for each subject • Each category is defined by a list of descriptors • The list of descriptors are not exhaustive • Each category stands on its own • Each category has an associated letter (B-F)

28. Dimension A: Content Topics Topics List (In your packet of material) Organized at two levels: Content Areas (16 for Mathematics) (27 for Science) (14 for ELAR) Topics (identified by number) (182 Mathematics Topics) (211 Science Topics) (114 ELAR Topics) Plus: non-specific & other

29. Topics Lists • Topics Lists • Mathematics • Science • English Language Arts & Reading • Cover grades K-12 • Organized into Content Areas • Topics & Content Areas have an associated #

30. Comments & Suggestion Worksheet • One for each reviewer - more available • Use to: • Record coding conventions/decision rules • Suggest/identify additional topics not listed • Suggest/identify additional CGD descriptors • Provide other comments & suggestions • Be sure to turn in at end of workshop (and with mail-in materials, as necessary).

36. The Content Analysis Process ‘Coding’ Teams of 4-5 Content Experts Independent Coding by each Analyst w/ Group Discussion Should not be necessary to discuss every item – select by team Goal for Process: Generalizability not Inter-rater Reliability Pick-up and return documents / coding sheets to Alissa Sign & return to Alissa non-disclosure forms

37. Practice Exercise #1 (Math)

38. Practice Exercise #2 (ELAR)

40. A neutral content language Topics by Cognitive Demand Content Analysis Workshop The intended curriculum: State content standards—What students should learn The assessed curriculum: State (and other) assessments—tested learning The learned curriculum: Student outcomes based on school learning The enacted curriculum: What teachers teach

41. Coding Forms • Assessment Coding Forms • Standards/Benchmark Coding Forms • Each is used to record content descriptions • Each content description consists of • A topic number • A cognitive demand category letter • The Integrated Analysis Coding Form • Combines Content & Linguistic Information in a single data collection form

42. Assessment Coding Forms

43. Standards Coding Forms

44. Standards Coding Forms

45. Gr. 5-8 National Science Standards