Using Curricular Measures for Description & Analysis

1. Surveys of Enacted Curriculum Using Curricular Measures for Description & Analysis SEC Collaborative Meeting Tampa, FL February 21, 2005 John L. Smithson, Director, Measures of Enacted Curriculum Wisconsin Center for Education Research, University of Wisconsin-Madison johns@education.wisc.edu

2. The SEC Data - Sets Distinctions On-LineOff-line • Data collection/processing/reporting • Descriptive Data • Limited Reporting Options • Easy Access / Indiv. Results • Analytic analyses • Unlimited reporting options • Requires data manipulation

3. Conducting Inquiry Using SEC Data Forms of Inquiry Collaborative or Evaluative Program Evaluation Indicator Reporting Program Management Performance Modeling Teacher Enrichment School Improvement Professional Lrng. Comm.

4. A short history of SEC research • Reform-Up-Close(Porter, Kirst, Osthoff, Smithson, Schneider, 1993) • Validation of teacher self-report survey data. • Upgrading Mathematics(Gamoran, Porter, Smithson, White, 1997) • First content analysis of assessment using content language. • Predictive validity of alignment index comparing instruction & assessments • Data on Enacted Curriculum(Blank, Porter, Smithson, 2004) • Use of SEC data to facilitate school improvement efforts • First content analysis of state standards • MSP-PD Study(In progress: Blank, Smithson, Porter, Garet, Birman) Use of SEC data for program evaluation

5. Alignment Relationships in Standards-based Reform Assessments Curriculum Alignment Standards

6. Alignment Relationships in Standards-based Reform Intersection of what is taught with what is tested. Assessment Instruction Taught, tested, and in the standards Intersection of what is taught with what is in the standards Standards Intersection of what is taught with what is in standards.

7. A Quantitative Approach to Alignment Porter-Smithson/SEC Alignment Process Content analyses of curriculum documents and reports of practice by content experts using two-dimensional content language. Multiple raters (w/ content & assessment expertise) using independent ratings in combination with team discussions. Content Description [Topic(s) by Cognitive Demand(s)] Yields Alignment Index based on:

8. Alignment Analyses The analytic power of quantitative alignment measures • Prediction • Program Evaluation* • Planning • Performance Modeling * Not teacher evaluation

9. Alignment Analyses Students perform best when tested on material for which they have been provided the opportunity to learn. Correlation of Alignment Index to Achievement Gains From: Gamoran, et.al. (1997). Uprgading High School Mathematics Instruction. EEPA v19n4pp325-338.

10. Explaining variation in student learning gains

11. Alignment Analyses for School Improvement Using alignment as an outcome measure Alignment Index: Instruction to Standards Mathematics Across 4 Districts Counts Treatment 99 Control 124 Leaders 16 (Measuring change in alignment over time)

12. Alignment Analyses for Planning Purposes

13. Mapping Curriculum Materials

14. SEC - Online (www.seconline.org)

15. Administrative Functions: Administration Set-up Review Registrants, Completion Rates Administrator Report Generator

17. SEC Reports Dynamic Sample Selection Dynamic Data Disaggregation Floating Bars Sample and Disaggregate Counts Comparison Charts Individual Results

18. SEC Reports: Instructional Content Coarse Grain Maps Content Areas by Cognitive Demand

19. SEC Reports: Instructional Content Fine Grain Maps Topics by Cognitive Demand

20. Alignment as a Systemic Tool Assessments 0.19 Curriculum 0.27 Alignment 0.18 Standards Fine Grain

21. Calculating Alignment 1-((|x-y|(1-n))/2)