2009 NAEP Science Framework and Specifications CAESL Annual Conference May 16, 2006 Steve Schneider, Alice Fu, Ri

1. 2009 NAEP Science Framework and Specifications CAESL Annual Conference May 16, 2006Steve Schneider, Alice Fu, Richard Shavelson, Mike Timms

2. Presentation Overview NAEP Overview Project Overview Science Content Science Practices Types of Items Interactive Computer Tasks (ICTs) The following slides will address these general topics. Steve will discuss NAEP and Project Overviews Alice will discuss Science content and practices Video of Rich discussing Types of Items Mike will discuss ICTsThe following slides will address these general topics. Steve will discuss NAEP and Project Overviews Alice will discuss Science content and practices Video of Rich discussing Types of Items Mike will discuss ICTs

3. Congressionally authorized since 1969Congressionally authorized since 1969

4. NAEP Overview: Components NAEP has 2 major components: Long-Term Trend NAEP (over 30 years) and Main NAEP. Main NAEP is what FW addresses Reports student performance at three levels – national, state, and urban district. Differences among the three levels (national, state, and TUDA) are only in the students sampled, not in the assessment itself – the same FW is used. NAEP does NOT report student performance at student and school levels. Science given every 4 years at national and state levels Other information if asked… State-level assessments in reading, mathematics, science, and writing (grades 4 & 8) Reading and math administered every 2 years at the state-level (grades 4 & 8) Trial urban district NAEP started in 2002; continued in 2003 and 2005. TUDA results for reading and mathematics are available for Atlanta, Boston, Charlotte, Chicago, Cleveland, Houston, Los Angeles, New York City, and San Diego, & District of Columbia. NAEP has 2 major components: Long-Term Trend NAEP (over 30 years) and Main NAEP. Main NAEP is what FW addresses Reports student performance at three levels – national, state, and urban district. Differences among the three levels (national, state, and TUDA) are only in the students sampled, not in the assessment itself – the same FW is used. NAEP does NOT report student performance at student and school levels. Science given every 4 years at national and state levels Other information if asked… State-level assessments in reading, mathematics, science, and writing (grades 4 & 8) Reading and math administered every 2 years at the state-level (grades 4 & 8) Trial urban district NAEP started in 2002; continued in 2003 and 2005. TUDA results for reading and mathematics are available for Atlanta, Boston, Charlotte, Chicago, Cleveland, Houston, Los Angeles, New York City, and San Diego, & District of Columbia.

5. NAEP Overview: Booklet Design All sampled students: 50 minutes – cognitive items 10 minutes – background questions Subset of sampled students: 30 minutes – hands-on performance tasks, ICTs This is the student booklet design for Main NAEP. All sampled students take a 60-minute assessment, which consists of: Cognitive questions: 50 minutes total (two 25-minute blocks) Background questions: 10 minutes total (two 5-minute blocks) Subset of sampled students will receive 1 extra 30-minute block for hands-on performance tasks and ICTs. (See presentations given by Mike Timms and Rich Shavelson.) Two Forms of Sampling Student Sampling 2. Item Sampling: Matrix sampling design—each student takes only a small portion of the items in a subject areaThis is the student booklet design for Main NAEP. All sampled students take a 60-minute assessment, which consists of: Cognitive questions: 50 minutes total (two 25-minute blocks) Background questions: 10 minutes total (two 5-minute blocks) Subset of sampled students will receive 1 extra 30-minute block for hands-on performance tasks and ICTs. (See presentations given by Mike Timms and Rich Shavelson.) Two Forms of Sampling Student Sampling 2. Item Sampling: Matrix sampling design—each student takes only a small portion of the items in a subject area

6. Project Overview $1.3M National Assessment Governing Board (NAGB) contract Contractor: WestEd Promote excellence, achieve equity, & improve learning Subcontractor: Council Chief State School Officers Leadership, advocacy, and technical assistance Collaborators: American Association for the Advancement of Science (AAAS) Council of State Science Supervisors (CSSS) National Science Teachers Association (NSTA) The contract for this Framework and Specifications project was awarded in September 2004. The contract for this Framework and Specifications project was awarded in September 2004.

7. Project Overview: Purposes Develop the recommended framework and specifications for NAEP Science 2009 Recommend important background variables associated with student achievement in science that should be included in NAEP Science Propose a design for one or more small-scale special research studies There are 3 overall purposes of this project. On 2. Many background variables such as demographics are already mandated by Congress. This project will recommend additional background variables that are specific to student achievement in SCIENCE. There are 3 overall purposes of this project. On 2. Many background variables such as demographics are already mandated by Congress. This project will recommend additional background variables that are specific to student achievement in SCIENCE.

8. Project Overview: Development Process Framework and Specs Development (2005-2006) Test development (2006 - 2007) Item reviews (content, bias, etc.) (2006 - 2007) Field testing; more reviews (2007 - 2008) Operational testing (2009) Setting achievement levels (2009) Report preparation and review ( 2010) Release of NAEP results (2010) Why are we developing an assessment framework now in 2005 for an assessment that won’t take place until 2009? The current framework has been in place for 15 years. Developments in science education that have since taken place include: Publication of National Science Education Standards (NRC) and Benchmarks for Science Literacy (AAAS), used widely by states throughout the country. Field knows more about how students develop and represent their knowledge in science. Assessment methods have also grown increasingly sophisticated. No Child Left Behind has placed the spotlight on states’ science assessments. International science standards and assessments have placed additional focus on science assessment in the United States. The National Assessment Governing Board requires that new assessment frameworks be developed periodically, so frameworks have a naturally limited shelf-life. This is not just a revision of the current framework – it is an entirely new framework. (Science trend lines will be broken.) There is a roughly 5-year cycle for the development of NAEP assessments. SEE SLIDE. Why are we developing an assessment framework now in 2005 for an assessment that won’t take place until 2009? The current framework has been in place for 15 years. Developments in science education that have since taken place include: Publication of National Science Education Standards (NRC) and Benchmarks for Science Literacy (AAAS), used widely by states throughout the country. Field knows more about how students develop and represent their knowledge in science. Assessment methods have also grown increasingly sophisticated. No Child Left Behind has placed the spotlight on states’ science assessments. International science standards and assessments have placed additional focus on science assessment in the United States. The National Assessment Governing Board requires that new assessment frameworks be developed periodically, so frameworks have a naturally limited shelf-life. This is not just a revision of the current framework – it is an entirely new framework. (Science trend lines will be broken.) There is a roughly 5-year cycle for the development of NAEP assessments. SEE SLIDE.

9. Project Overview: Development Process Another way to look at where we are in the assessment development process: We have finished with both the Framework (a “broad strokes” description of what students should know and be able to do for a general audience) and Specifications (a “detailed blueprint” for audience of item writers and assessment developers). Specs is aligned to the Framework. Another way to look at where we are in the assessment development process: We have finished with both the Framework (a “broad strokes” description of what students should know and be able to do for a general audience) and Specifications (a “detailed blueprint” for audience of item writers and assessment developers). Specs is aligned to the Framework.

10. Project Overview: Committees Steering Committee: Oversight Oversee Policies, goals, principles Education, industry, & science organization representatives Co-Chairs: Gerald Wheeler (NSTA), Rolf Blank (CCSSO) Planning Committee: Production Produce Recommended Framework Recommended Assessment and Item Specifications Recommendations for background variables Designs for small-scale studies Educators, measurement experts, scientists Co-chairs: Richard Shavelson (Stanford), Senta Raizen (WestEd) This project operated with a 2-tiered development process. SEE SLIDE. SC has 33 members (including co-chairs) PC has 25 members (including co-chairs)This project operated with a 2-tiered development process. SEE SLIDE. SC has 33 members (including co-chairs) PC has 25 members (including co-chairs)

11. Project Overview: Broad-Based Input Diverse Steering and Planning Committees External Framework Review Panel Presentations and Feedback Sessions American Association for the Advancement of Science Center for Assessment and Evaluation of Student Learning Council of Chief State School Officers Council of State Science Supervisors NAEP State Coordinators National Research Council National Science Teachers Association NAGB Invitational Business and Industry Forum NAGB Public Hearing Stakeholders include scientists, teachers, educators, policymakers, state education officials, and public Given importance of document, many stakeholders, and high visibility, FW went through an extensive review process. Major outreach efforts in Spring 2005. CSSS activities included 368 participants at 13 regional meetings NSTA activities included 1, 769 responses to online teacher survey NAGB public hearing (Oct. 2005): testimony from Achieve, AAAS, Intel Foundation, ITEA, Museum of Science, NAE, NSTA, Fordham Foundation. In addition, 100s of individual letters and e-mails with comments received. Stakeholders include scientists, teachers, educators, policymakers, state education officials, and public Given importance of document, many stakeholders, and high visibility, FW went through an extensive review process. Major outreach efforts in Spring 2005. CSSS activities included 368 participants at 13 regional meetings NSTA activities included 1, 769 responses to online teacher survey NAGB public hearing (Oct. 2005): testimony from Achieve, AAAS, Intel Foundation, ITEA, Museum of Science, NAE, NSTA, Fordham Foundation. In addition, 100s of individual letters and e-mails with comments received.

12. Science Content: Topics and Subtopics Science content organized into three main areas: Physical, Life, Earth/Space. On the surface, this is similar to old FW, but subtopics and specific content within each area have changed. 2009 content is based on Standards and Benchmarks. In particular, Earth/Space Science has taken more of a systems perspective (rather than a “story in the rocks” perspective). Content determined by Committees as well as EXTENSIVE REVIEW PROCESS from stakeholders, professional associations, all 50 states. Science content organized into three main areas: Physical, Life, Earth/Space. On the surface, this is similar to old FW, but subtopics and specific content within each area have changed. 2009 content is based on Standards and Benchmarks. In particular, Earth/Space Science has taken more of a systems perspective (rather than a “story in the rocks” perspective). Content determined by Committees as well as EXTENSIVE REVIEW PROCESS from stakeholders, professional associations, all 50 states.

13. Science Content: Earth/Space Statements In the Framework, each content area is described by a narrative introduction and a table of content statements. Note that this presentation format allows one to see clearly the progression across grades. Committees wanted to capture increase in sophistication of science knowledge. Within each content area, content is grouped by major topic (Earth Systems) and subtopic (Climate and Weather). Content statements are coded for easy reference. In the Framework, each content area is described by a narrative introduction and a table of content statements. Note that this presentation format allows one to see clearly the progression across grades. Committees wanted to capture increase in sophistication of science knowledge. Within each content area, content is grouped by major topic (Earth Systems) and subtopic (Climate and Weather). Content statements are coded for easy reference.

14. Content Boundaries Examples Measurement and representations Technical vocabulary Clarification New to the Specifications (does not appear in the Framework): boundaries that elaborate on the content statements, which are written in general terms. Intended as “notes to the item writer.” Boundaries serve the function of informing judgments about the appropriateness of an item for a given grade level and subtopic content area. Boundaries do not stand alone—they should be considered in conjunction with the content statement tables and narrative introductions just described. Boundaries were developed by a subgroup of the PC in January and were further refined based on review by SC, PC, and state science and testing coordinators. Four categories of content boundaries—will talk more about them and provide examples on next slides. New to the Specifications (does not appear in the Framework): boundaries that elaborate on the content statements, which are written in general terms. Intended as “notes to the item writer.” Boundaries serve the function of informing judgments about the appropriateness of an item for a given grade level and subtopic content area. Boundaries do not stand alone—they should be considered in conjunction with the content statement tables and narrative introductions just described. Boundaries were developed by a subgroup of the PC in January and were further refined based on review by SC, PC, and state science and testing coordinators. Four categories of content boundaries—will talk more about them and provide examples on next slides.

15. Content Boundaries—Earth/Space Excerpts Here is an example of content boundaries for the same subtopic shown earlier—Climate and Weather in Earth/Space. Note that this is not a complete table—we are just showing excerpts from sections on (1) Examples/Observations/Phenomena; and (2) Instruments/Representations. EXAMPLES: Because the content statements are written in general terms, this category is concerned with the types of specific examples, observations, and phenomena that should appear on the assessment. Concerns the breadth and depth of topic coverage. There may be limits to the phenomena, examples, and observations that should be used in assessment items. It is impractical to list every phenomenon, example, and observation that is appropriate for inclusion so words such as “common” or “familiar” are used; “such as,” “including,” “e.g.,” and “etc.” are used to denote suggestions. INSTRUMENTS: Describes the use of instruments and the level of precision expected of students in measuring or classifying phenomena, or in interpreting measurements. Moreover, scientific information can be represented in different forms, e.g., kinds of mathematical operations or equations and chemical formulas that should appear on assessment items. Next slide: examples of Technical Vocabulary and ClarificationHere is an example of content boundaries for the same subtopic shown earlier—Climate and Weather in Earth/Space. Note that this is not a complete table—we are just showing excerpts from sections on (1) Examples/Observations/Phenomena; and (2) Instruments/Representations. EXAMPLES: Because the content statements are written in general terms, this category is concerned with the types of specific examples, observations, and phenomena that should appear on the assessment. Concerns the breadth and depth of topic coverage. There may be limits to the phenomena, examples, and observations that should be used in assessment items. It is impractical to list every phenomenon, example, and observation that is appropriate for inclusion so words such as “common” or “familiar” are used; “such as,” “including,” “e.g.,” and “etc.” are used to denote suggestions. INSTRUMENTS: Describes the use of instruments and the level of precision expected of students in measuring or classifying phenomena, or in interpreting measurements. Moreover, scientific information can be represented in different forms, e.g., kinds of mathematical operations or equations and chemical formulas that should appear on assessment items. Next slide: examples of Technical Vocabulary and Clarification

16. Content Boundaries—Earth/Space Excerpts Here is an example of content boundaries for the same subtopic shown earlier—Climate and Weather in Earth/Space. This slide shows excerpts from sections on (3) Technical Vocabulary; and (4) Clarification/Other. VOCABULARY: In general, the NAEP Science Assessment will concern itself with science principles and their use and application. Yet, vocabulary is important to science communication. Consequently, knowing definitions of science terms has a place in the science assessment. Particularly at grades 8 and 12, students should know the meanings and be able to use science vocabulary associated with the topics to which they have been exposed. Unless otherwise stated, it should be assumed that the science terms included in assessment items are generally limited to those that appear in the content statements. CLARIFICATION: May include restatement of a “key idea” encompassed by the content statement(s); specification of the intent of a boundary statement; elaboration on suitable types of representations; suggestion of what might be the basis for appropriate assessment tasks; or reference to related content statements, subtopics, or other specific sections of this document. Note the term “Exclusion(s)” is used to denote content that is not appropriate for inclusion on the 2009 NAEP Science Assessment. Here is an example of content boundaries for the same subtopic shown earlier—Climate and Weather in Earth/Space. This slide shows excerpts from sections on (3) Technical Vocabulary; and (4) Clarification/Other. VOCABULARY: In general, the NAEP Science Assessment will concern itself with science principles and their use and application. Yet, vocabulary is important to science communication. Consequently, knowing definitions of science terms has a place in the science assessment. Particularly at grades 8 and 12, students should know the meanings and be able to use science vocabulary associated with the topics to which they have been exposed. Unless otherwise stated, it should be assumed that the science terms included in assessment items are generally limited to those that appear in the content statements. CLARIFICATION: May include restatement of a “key idea” encompassed by the content statement(s); specification of the intent of a boundary statement; elaboration on suitable types of representations; suggestion of what might be the basis for appropriate assessment tasks; or reference to related content statements, subtopics, or other specific sections of this document. Note the term “Exclusion(s)” is used to denote content that is not appropriate for inclusion on the 2009 NAEP Science Assessment.

17. Science Content: Crosscutting Content Abstract themes from old Framework (such as Systems, Models, Patterns of Change) proved difficult to assess, so Committees wanted to highlight crosscutting principles that were more rooted in the content statements themselves. This is an example of crosscutting content that is highlighted in the FW—note how concepts are embedded in the content statements themselves. Other crosscutting content includes energy sources and transformations. Abstract themes from old Framework (such as Systems, Models, Patterns of Change) proved difficult to assess, so Committees wanted to highlight crosscutting principles that were more rooted in the content statements themselves. This is an example of crosscutting content that is highlighted in the FW—note how concepts are embedded in the content statements themselves. Other crosscutting content includes energy sources and transformations.

18. Science Content: Item Distribution This distribution was informed by NAEP data on students’ course-taking patterns. equal in grade 4 emphasis on Earth/Space in grade 8 emphasis on Physical and Life in grade 12 This distribution was informed by NAEP data on students’ course-taking patterns. equal in grade 4 emphasis on Earth/Space in grade 8 emphasis on Physical and Life in grade 12

19. Science Content: 1996-2005 ? 2009 Primary sources for Framework content were National Standards and Benchmarks (as per SC charge) As these are curriculum documents, the breadth of content needed to be narrowed. In general, content common to both Standards and Benchmarks was given priority. Additions and exclusions made on a case-by-case basis. Tried to focus on central/foundational principles. This was an iterative process! Major categories of science content remain roughly the same as the old Framework (Physical, Life, Earth) Primary sources for Framework content were National Standards and Benchmarks (as per SC charge) As these are curriculum documents, the breadth of content needed to be narrowed. In general, content common to both Standards and Benchmarks was given priority. Additions and exclusions made on a case-by-case basis. Tried to focus on central/foundational principles. This was an iterative process! Major categories of science content remain roughly the same as the old Framework (Physical, Life, Earth)

20. Science Practices: Summary Practices informed by: Standards, Benchmarks, SRI’s “Validities of Science Inquiry” research project, a LOT of deliberation by both Committees. NOTE: “science principles” refers to not only the principles but also the facts, concepts, theories, laws of science. Note that these categories are not distinct and some overlap is to be expected. Although not explicit, some aspects of nature of science are embedded in these practices (especially Using Science Principles and Scientific Inquiry). FW definition of scientific inquiry is limited—tried to be realistic in terms of what can be achieved in a large-scale assessment. Some aspects that are traditionally included in a definition of inquiry appear under Using Science Principles (such as explain observations). Communication runs across the practices; expect that items requiring communication skills will be represented at all three grade levels Though not explicit here, students are expected to use quantitative reasoning on NAEP Science. These expectations are largely embedded in the content statements. Other skills (collaboration, creativity, social responsibility) are outside bounds of NAEP assessment. Practices informed by: Standards, Benchmarks, SRI’s “Validities of Science Inquiry” research project, a LOT of deliberation by both Committees. NOTE: “science principles” refers to not only the principles but also the facts, concepts, theories, laws of science. Note that these categories are not distinct and some overlap is to be expected. Although not explicit, some aspects of nature of science are embedded in these practices (especially Using Science Principles and Scientific Inquiry). FW definition of scientific inquiry is limited—tried to be realistic in terms of what can be achieved in a large-scale assessment. Some aspects that are traditionally included in a definition of inquiry appear under Using Science Principles (such as explain observations). Communication runs across the practices; expect that items requiring communication skills will be represented at all three grade levels Though not explicit here, students are expected to use quantitative reasoning on NAEP Science. These expectations are largely embedded in the content statements. Other skills (collaboration, creativity, social responsibility) are outside bounds of NAEP assessment.

21. Science Practices: Item Distribution Note increase in Using Science Principles from grades 4 to 8 to 12 and corresponding decrease in Identifying Science Principles. The expectation is that, as students move up through the grades, their critical response skills and methodological and analytical capabilities will increase. Note increase in Using Science Principles from grades 4 to 8 to 12 and corresponding decrease in Identifying Science Principles. The expectation is that, as students move up through the grades, their critical response skills and methodological and analytical capabilities will increase.

22. Generating Items: Performance Expectations Neither the content statements nor the practices will be assessed in isolation. All assessment items will be derived from a combination of the two (i.e., from performance expectations). Performance expectations are created by crossing a content statement with a science practice. Performance expectations describe the performances expected of students on the assessment.Neither the content statements nor the practices will be assessed in isolation. All assessment items will be derived from a combination of the two (i.e., from performance expectations). Performance expectations are created by crossing a content statement with a science practice. Performance expectations describe the performances expected of students on the assessment.

23. Generating Items and Interpreting Responses Once performance expectations are generated, items can be developed, and student responses are interpreted. These interpretations constitute evidence of what students know and can do in science. The four science practices—Identifying Science Principles; Using Science Principles; Using Scientific Inquiry; and Using Technological Design—articulate what students should know and be able to do with the science principles presented in Chapter Two. Certain ways of knowing and reasoning—cognitive demands—underpin these four science practices. Cog. Demands (From Shavelson and colleagues): “knowing that,” “knowing how,” “knowing why,” and “knowing when and where to apply knowledge” The goal is to further elucidate the descriptions of the science practices, to facilitate item specifications and item writing, and to provide a framework for interpreting students’ responses. That is, the set of four cognitive demands can be used as a lens to facilitate item development and to analyze student responses The FW provides an example of this process. Additional examples are provided in the Specifications. Once performance expectations are generated, items can be developed, and student responses are interpreted. These interpretations constitute evidence of what students know and can do in science. The four science practices—Identifying Science Principles; Using Science Principles; Using Scientific Inquiry; and Using Technological Design—articulate what students should know and be able to do with the science principles presented in Chapter Two. Certain ways of knowing and reasoning—cognitive demands—underpin these four science practices. Cog. Demands (From Shavelson and colleagues): “knowing that,” “knowing how,” “knowing why,” and “knowing when and where to apply knowledge” The goal is to further elucidate the descriptions of the science practices, to facilitate item specifications and item writing, and to provide a framework for interpreting students’ responses. That is, the set of four cognitive demands can be used as a lens to facilitate item development and to analyze student responses The FW provides an example of this process. Additional examples are provided in the Specifications.

24. Science Practices: 1996-2005 ? 2009 In 2009, Conceptual Understanding has been further specified into Identifying and Using Science Principles. NOTE: “science principles” refers to not only the principles but also the facts, concepts, theories, laws of science. Practical Reasoning in old FW was broad (included social and personal considerations) and very difficult to assess. Technological design is generally defined as “systematic process of applying science knowledge and skills to solve problems in a real-world context”—it is more specific than Practical Reasoning and is limited to science-based considerations. NEXT: Rich could not be here with us today, but he has recorded a video in which he talks about the types of items that should appear on 2009 NAEP Science Assessment. In 2009, Conceptual Understanding has been further specified into Identifying and Using Science Principles. NOTE: “science principles” refers to not only the principles but also the facts, concepts, theories, laws of science. Practical Reasoning in old FW was broad (included social and personal considerations) and very difficult to assess. Technological design is generally defined as “systematic process of applying science knowledge and skills to solve problems in a real-world context”—it is more specific than Practical Reasoning and is limited to science-based considerations. NEXT: Rich could not be here with us today, but he has recorded a video in which he talks about the types of items that should appear on 2009 NAEP Science Assessment.

25. 2009 NAEP Science Framework and Specifications CAESL Annual Conference May 16, 2006Steve Schneider, Alice Fu, Richard Shavelson, Mike Timms

26. Websites NAEP: http://nces.ed.gov/nationsreportcard/ National Center for Education Statistics: http://nces.ed.gov/ National Assessment Governing Board: http://www.nagb.org/ NOTE: Prepublication edition of the Framework is available on the NAGB website. Click on “Publications.” Scroll down to “Frameworks.” NOTE: Prepublication edition of the Framework is available on the NAGB website. Click on “Publications.” Scroll down to “Frameworks.”