1 / 47

Oregon Content Standards and Assessment System Evaluation

Oregon Content Standards and Assessment System Evaluation. Prepared for the Oregon Department of Education by WestEd Dr. Stanley Rabinowitz Dr. Edynn Sato May 18, 2007. One of several key projects funded by the Gates Foundation

ron
Download Presentation

Oregon Content Standards and Assessment System Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Oregon Content Standards and Assessment System Evaluation Prepared for the Oregon Department of Education by WestEd Dr. Stanley Rabinowitz Dr. Edynn Sato May 18, 2007

  2. One of several key projects funded by the Gates Foundation Independent evaluation by WestEd, a non-profit educational research, development, and service agency Background: Overview

  3. Content Standards and Assessment System Evaluation: Why? • Content standards signal what knowledge and skills are valued (breadth and depth) • Current state content standards have undergone periodic revision • Need to continuously revisit and re-evaluate on a set schedule

  4. Content Standards and Assessment System Evaluation: Why? • Rigorous standards and assessments are major levers for improving student achievement • Signal goals • Focus instruction • Provide information for improvement to the state, districts, schools

  5. Content Standards and Assessment System Evaluation: What? WestEd will: • Review the content standards (all grades, academic content areas) • Evaluate the structure of the content standards • Evaluate alignment between the state assessments and the content standards • Make recommendations for improvement of structures and systems

  6. What? • WestEd will address questions regarding: • the structure and quality the state content standards • the alignment of state assessments to state standards • Recommendations will be based on data from the analyses that have practice and policy implications • Research-based information related to assessment and accountability models/practices will be provided as available

  7. Key Deliverables • Preliminary report of an initial review of selected content standards and grades using initial protocol/criteria • Final report for the comprehensive review of the content standards • Final report for the alignment of assessments to content standards • Final report on the structure and quality of Oregon’s content standards and assessments

  8. Content Standards and Assessment System Evaluation: How? WestEd’s independent evaluation will be conducted in three phases: • Preliminary Review of Selected Content Standards (already presented to Board) • Comprehensive Evaluation of Content Standards • Alignment Studies of Content Standards and Assessments

  9. How? Comprehensive Evaluation of Content Standards • English language arts • Mathematics • Science • Social sciences • Arts • Second languages • Physical education • Health education • Technology • English language proficiency

  10. How? Alignment Studies of Content Standards and Assessments Content • English language arts—Reading and Literature • Mathematics • Science Items • Oregon’s multiple-choice knowledge and skills state test items Tests • One test blueprint/specification for English language arts, mathematics, and science

  11. How? WestEd’s independent evaluation will involve analysts with expertise in • large-scale test development • standards development • alignment • measurement and statistics • the content areas • curriculum and instruction • the K-12 student population • No Child Left Behind (NCLB) • Oregon programs and history

  12. How? Stakeholder engagement: • Discussions—These discussions will not influence WestEd’s independent and objective analyses, rather they will inform the analyses • Surveys (online) • Parents & Guardians (427 completed/593 total) • Educators (286 completed/535 total) • Policy Makers (18 completed/38 total)

  13. How? Stakeholder groups include: • ODE staff, such as - Management Team - Curriculum and Assessment staff • State Board of Education • OAESD Instructional Leader Council • PK-20 Coordination Advisory Group • Literacy Leadership State Steering Committee • Critical Friends Advisory Group

  14. Content Standards and Assessment System Evaluation: When? December 1, 2006 through July 31, 2007 • Report of a preliminary review of selected content standards and grades using initial protocol/criteria (January 2007) • Report for the alignment studies (March/April 2007) • Report for the comprehensive review of the content standards (April-June 2007) • Final report on the structure of Oregon’s content standards and assessments (July 2007)

  15. Status: Alignment Studies • The analyses aimed to address the following key questions: • To what degree do the State assessment items reflect the concepts and skills embodied in the State’s academic content standards? • To what degree do the assessment items cover the breadth, depth, and range of complexity of content intended by the State?

  16. Status: Alignment Studies (continued) ELA: Item Pool-Level Analyses Categorical Concurrence • For all grades except for grade 4, the State’s assessment items had a high level of categorical concurrence overall as well as with each CCG in the grade-level standards. • At grade 4, items had a high level of categorical concurrence overall and with each CCG in the grade-level standards, except for DGUL, DAIL, and ECSI, which had a low level of categorical concurrence. Range of Depth of Knowledge • Overall, at each grade level, items covered a range of DoK. However, the range of DoK appeared restricted for some CCGs at each grade level.

  17. Status: Alignment Studies (continued) ELA: Item Pool-Level Analyses(continued) Range of Knowledge (Comprehensiveness/Breadth) • Overall, items covered the range of content represented in the standards at each grade level. With the exception of Grade 3, items appeared to address the breadth of content of each CCG, where the standards were assessable according to test specifications. Balance of Representation (Emphasis) • Overall, at each grade level there was a balance of representation. However, the actual emphasis of content related to specific CCGs in the item pool and the intended emphasis of content related to the ELA “score reporting categories” per the test specifications were not entirely consistent.

  18. Status: Alignment Studies(continued) ELA: Test-Level Analyses • The examination of items on the Grade 8 ELA assessment shows coverage of content across all six score reporting categories. The emphasis of content appears relatively consistent for VOCA, READ, and ECSL, less so for DGU, DAI, and ECSI.

  19. Status: Alignment Studies (continued) Math: Item Pool-Level Analyses Categorical Concurrence • For all grades, the State’s assessment items had a high level of categorical concurrence overall as well as with each strand in the grade-level standards. Range of Depth of Knowledge • Overall, at each grade level, items covered a range of DoK. However, the range of DoK appeared restricted for some strands at each grade level.

  20. Status: Alignment Studies (continued) Math: Item Pool-Level Analyses (continued) Range of Knowledge (Comprehensiveness/Breadth) • Overall, items covered the range of content represented in the standards at each grade, with the exception of Grade 6. At the strand level, items for Grades 7, 8, and CIM appeared to address the breadth of content of each strand; however, this was not the case for Grades 3, 4, 5, and 6. Balance of Representation (Emphasis) • Overall, at each grade level there was a balance of representation. However, the actual emphasis of content related to specific CCGs in the item pool and the intended emphasis of content per the test specifications are not entirely consistent.

  21. Status: Alignment Studies (continued) Math: Test-Level Analyses • The examination of items on the Grade 5 mathematics assessments shows that overall the intended and actual distributions of content coverage at the strand level were comparable; that is, no percentage difference exceeded 4%.

  22. Status: Alignment Studies (continued) Science: Item Pool-Level Analyses Categorical Concurrence • For all grades, the State’s assessment items had a high level of categorical concurrence overall as well as with each CCG in the grade-level standards. Range of Depth of Knowledge • For all grades and for all strands, except Earth and Space Science at CIM, the items represented a range of depth of knowledge levels from Recall to Strategic Thinking. In CIM, items in the Earth and Space Science strand were aligned to two DoK levels only: Recall and Basic Application.

  23. Status: Alignment Studies (continued) Science: Item Pool-Level Analyses (continued) Range of Knowledge (Comprehensiveness/Breadth) • For Grade 5, the items in the test pool generally address the entire breadth of the strand content as operationalized in the benchmark standards and eligible content. • For Grade 8, the items in the test pool generally address the entire breadth of the strand content as operationalized in the benchmark standards and eligible content. • For CIM, the items in the test pool generally address the entire breadth of the strand content as operationalized in the benchmark standards and eligible content.

  24. Status: Alignment Studies (continued) Science: Item Pool-Level Analyses (continued) Balance of Representation (Emphasis) • For Grade 5, there was generally a balanced representation of content, although Life Science had slightly more items than either Physical or Earth Sciences. • For Grade 8, each CCG had a balanced representation of content in the item pool items; however, as was the case with Grade 5, Life Science had slightly more items than either Physical or Earth Sciences. • For CIM, each CCG had a balanced representation of content in the item pool items. • The content for Physical Science, Life Science, and Earth Science were fairly evenly emphasized throughout the entire item pool.

  25. Status: Alignment Studies (continued) Science: Test-Level Analyses • A comparison of the intended and actual distribution of items found that overall, the percentages of items at each strand were comparable; that is, no percentage difference exceeded 6%.

  26. Status: Alignment Studies(continued) General Recommendations • The ODE should evaluate the small set of items that WestEd analysts determined had either partial or no alignment (categorical concurrence) to the standards and consider replacing or revising the items as appropriate in order to strengthen the relationships between the items and standards. Current procedures for item development and review should be analyzed and modified to limit future occurrences of partial or no alignment. • The ODE should verify that the content coverage of the assessed domains is purposeful in terms of breadth, depth, emphasis, and complexity.

  27. Status: Alignment Studies (continued) General Recommendations (continued) • The ODE should consider the assessed content and how content interacts with item format (multiple choice) in order to articulate more clearly its intended range of depth of knowledge (i.e., whether the emphasis and range of DoK reflected in its assessments for each grade level and content area are consistent with expectations for each grade level/content area as well as across grade levels for a content area) and to ensure that the item format appropriately and effectively lends itself to students’ demonstrations of what they know and can do vis-à-vis the assessed content.

  28. Status: Alignment Studies (continued) General Recommendations (continued) • Although the online assessment engine (TESA) is designed to select and administer to students a set of items that satisfy the test specifications, the ODE should consider examining a random sample of these assessments to ensure that the actual sets of items administered are indeed consistent with the test specifications in terms of breadth, depth, emphasis, and complexity. Additionally, documentation of the reliability and accuracy with which the test engine adheres to the state’s test specifications should be produced.

  29. Status: Alignment Studies (continued) General Recommendations (continued) • The ODE should consider examining the full range of test blueprints vis-à-vis relevant sets of assessment items. • The DoK definitions should be considered during item development to help ensure the full range of complexity of the assessed content, as appropriate.

  30. Status: Standards Evaluation Studies The analyses aimed to address the following key questions: • Do Oregon’s content standards adequately represent the knowledge and skills that all students should know and be able to do? • Do Oregon’s content standards reflect the appropriate breadth and depth of the content area? • Do Oregon’s content standards have the clarity and consistency needed to adequately guide instruction and assessment?

  31. External Referents

  32. External Referents (continued)

  33. External Referents (continued)

  34. External Referents (continued)

  35. Status: Standards Evaluation Studies (continued) Overall Quality • Oregon’s English Language Arts (ELA) standards overall are of good quality and provide breadth and depth of coverage of Reading, Literature, Writing, Speaking, and Listening. The standards generally are clearly written, focus on important skills/concepts for instruction, and are assessable. • Oregon’s mathematics standards overall are of good quality and reflect a breadth and depth of content coverage. The standards generally are clearly written, focus on important skills/concepts for instruction, and are assessable.

  36. Status: Standards Evaluation Studies (continued) Overall Quality (continued) • Oregon’s science standards overall are of good quality in terms of clarity and importance for instruction. However, in all grades except for grade 8, analysts determined that the standards did not reflect a range of depth of knowledge. For all grades except for grade 1, analysts determined that the standards did not reflect a breadth of knowledge. With regard to consistency, analysts determined that standards were inconsistent (language, skills, knowledge) in grades 4, 8, and CIM. And, in all grades except for grades 3, 5, 8, and CIM, there were issues of assessability. • Oregon’s socialsciences standards overall are of good quality in terms of depth, breadth, consistency, importance for instruction, and assessability. However, overall these standards lack clarity.

  37. Status: Standards Evaluation Studies (continued) Overall Quality(continued) • Oregon’s arts standards overall are of good quality and cover a breadth and depth of content. The standards generally are clearly written, focus on important skills/concepts for instruction and are assessable. • Oregon's physical education standards overall are of good quality in terms of breadth and depth of content coverage, consistency across benchmark grades, importance for instruction, and degree of assessability. However, analysts determined that these standards are not easily understood due to lack of clarity and precision of language. • Oregon’s health standards overall are of good quality in terms of depth, consistency, importance for instruction, and assessability. For all benchmark levels except for Benchmark 2, the standards overall are of good quality in terms of breadth of content. And, overall these standards lack clarity.

  38. Status: Standards Evaluation Studies (continued) Overall Quality(continued) • Oregon’s educational technology standards overall are of good quality in terms of depth, breadth, consistency, importance for instruction, and assessability. However, overall these standards lack clarity. • Oregon’s second languages standards overall are of good quality in terms of depth, breadth, clarity, consistency, importance for instruction, and assessability. • Generally, Oregon’s English language proficiency (ELP) forms and functions are presented in a clear and useful format. The proficiency level descriptors clearly describe and differentiate the language skills of students at each level. However, the definitions of language function and forms of language appear incomplete. All the language functions except one are appropriate—literary analysis is typically considered English-Language Arts content rather than a language function. All the language forms except one are appropriate—the language of propaganda is not a form consistent with other elements of form presented by the state; rather it appears to be a genre.

  39. Status: Standards Evaluation Studies (continued) Alignment to External Referents • Oregon’s ELA standards generally aligned to the Indiana, NAEP and McREL standards in terms of overall content order/sequence and depth of content. However, while the breadth of the Oregon standards appeared comparable to the NAEP and McREL standards, they were not as comparable to the Indiana standards. • Oregon’s mathematics standards were compared to Indiana’s mathematics standards as well as to the NCTM Principles and Standards for School Mathematics (2000) and Curriculum Focal Points for K–8 Mathematics (2006). Oregon’s mathematics standards generally aligned to the NCTM standards and to the Indiana standards with regard to overall depth and breadth of content coverage. However, there was not overall congruence between Oregon and Indiana’s content ordering and sequencing.

  40. Status: Standards Evaluation Studies (continued) Alignment to External Referents(continued) • Oregon’s science standards were compared to Indiana’s science standards and the AAAS benchmarks (2001). • Generally, the Earth and Space Science standards aligned in terms of overall depth of content; however, they did not align in terms of content order/sequence or overall breadth of content. • Generally, the Physical Science standards aligned to the Indiana standards in terms of content order/sequence, overall depth, and overall breadth of content. The Oregon standards also aligned to the AAAS standards in terms of overall depth and breadth of content; however, they did not align in terms of content order/sequence. • Generally, the Life Science standards aligned to the Indiana standards in terms of content order/sequence, overall depth, and overall breadth of content. However, they did not align to the AAAS standards in terms of content order/sequence, overall breadth, or overall depth of content.

  41. Status: Standards Evaluation Studies (continued) Alignment to External Referents (continued) • Oregon’s socialsciences standards were compared to Washington’s social science standards and the NAEP frameworks. • Generally, the Geography standards aligned in terms of overall content order/sequence with Washington’s standards. However, they did not align in terms of content order/sequence with the NAEP framework or in terms of the overall depth or breadth of content with the Washington standards or the NAEP framework. • Generally, the History standards did not align to the NAEP framework in terms of content order/sequence, depth or breadth of content. Nor did these standards align in terms of overall breadth with the Washington U.S. History and World History standards. However, the Oregon standards aligned to the Washington standards in terms of content order/sequence and overall depth.

  42. Status: Standards Evaluation Studies (continued) Alignment to External Referents(continued) • Oregon’s socialsciences standards (continued). • Generally, the Civics standards aligned in terms of overall content order/sequence with Washington’s standards and the NAEP framework. However, they did not align in terms of overall depth or breadth of content with either the Washington standards or the NAEP framework. • Generally, the Economics standards aligned to the Washington standards and the NAEP framework in terms of content order/sequence, depth and breadth of content. • There was a high degree of alignment between Oregon’s arts standards and those of Washington and McREL. In terms of content order / sequence, Washington’s standards spiral content across different benchmarks in a manner similar to Oregon; many of McREL’s benchmark level standards are not spiraled across different benchmarks. Both external referents show similar depth of content to Oregon’s standards.

  43. Status: Standards Evaluation Studies (continued) Alignment to External Referents(continued) • Generally, Oregon's physical education standards, the National Association of Sports and Physical Education Standards (NASPE), and the Indiana standards share a common purpose. In broad terms, the nature of the content knowledge/cognitive information and the types of skills students are expected to learn and be able to demonstrate are comparable across the three sets of standards. Oregon differs in depth of content coverage from the NASPE and Indiana State standards. • Oregon’s health standards were compared to Indiana’s health standards as well as to the National Health Education Standards (NHES). Oregon’s standards generally aligned to the Indiana and NHES standards with regard to overall content order/sequence and breadth of content coverage. However, there was not overall congruence in terms of depth of content.

  44. Status: Standards Evaluation Studies (continued) Alignment to External Referents (continued) • Oregon’s educational technology standards generally aligned to the Washington and NETS standards in terms of overall content order/sequence, depth of content and breadth of content. • While Oregon and Indiana both have the same number of proficiency levels, a comparison of the depth and breadth of the second languages standards across levels in both states shows that the Indiana standards start at a slightly higher level than those in Oregon, and reach a higher level. Oregon’s proficiency levels appear to be based on ACTFL levels Novice-Low through Intermediate-Mid. The standards statements differ in level of detail. Generally, Oregon standards are written more broadly, and Indiana’s are more detailed and specific. Oregon’s set of Functions Supporting Standards describe a similar level of detail as Indiana’s standards.

  45. Status: Standards Evaluation Studies (continued) Alignment to External Referents(continued) • Oregon’s English language proficiency (ELP) forms and functions were compared to current national research and literature on forms and functions (e.g., TESOL, CALLA, Butler, et al.). All but one of Oregon’s language functions (literary analysis) are appropriate and relevant for ELD instruction. The external referents include functions that are currently not present among those in the Oregon ELP standards.

  46. Next Steps • Address questions regarding the state’s standards and assessment system—quality, structure, etc. • Make recommendations based on data from the analyses (standards evaluation and assessment alignment) that have practice and policy implications • Make recommendations that are research-based or based on promising/best assessment and accountability models and practices • Consider the feedback/information provided key stakeholders through surveys, meetings, etc.

  47. For more information about the standards and assessment evaluation: ode.state.or.us/go/real For more information about WestEd: www.wested.org

More Related