Evaluating the Conceptual Coherence and Rigor of Instructional Materials

1. Evaluating the Conceptual Coherence and Rigor of Instructional Materials Florida Association of Science Supervisors 5 May 2005 Jody Bintz, BSCS Center for Professional Development

2. Think of a district science team working together to select or recommend instructional materials… • What are they saying? • What is influencing their decision? • What are they looking at in the materials? • What context issues are being considered? • How will they make their decision?

3. Goal • Provide insights into an evidence-based process for evaluating instructional materials that: • Builds the awareness that high-quality instructional materials matter in the learning process for students. • Develops common understandings among teachers about the characteristics of high-quality instructional materials. • Requires teacher dialogue around evidence to build consensus during the decision-making process. • Serves as a professional development strategy to support implementation of instructional materials.

4. We believe that we found powerful evidence that textbooks exert a strong influence on what teachers teach. This seems to be true in most countries despite differences in the nature and use of textbooks. Textbook coverage is important both for what topics are taught and for the levels of performances and accomplishments expected of students. Whatever should be the case, textbooks have an impact almost everywhere, and it is perilous to ignore the ways in which they at least partially shape what is taught. Why Schools Matter, Schmidt, et. al., {date}

5. Premises… • Student learning is directly linked to the quality of the instructional materials selected. • Teachers are more effective in the classroom when they implement high quality instructional materials. • Teachers make better decisions about instructional materials when the dialogue is based on evidence. • The process of selecting instructional materials presents an opportunity for professional development.

6. Teacher Dialogue

7. AIM A process and tools for Analyzing Instructional Materials

8. Development of the AIM Process and Tools • Collaboration between the BSCS Center for Professional Development and the K-12 Alliance of WestEd • Adapted and field-tested as part of the NSF funded National Academy for Curriculum Leadership (NACL) • Used by thousands of teachers across the country

9. It is… A process and set of tools to select instructional materials A professional development strategy A collaborative process that uses consensus-building Based on National Science Education Standards Linked to the research findings described in How People Learn It is NOT… A checklist; goes beyond the thumb test Overly complicated and prescriptive What is AIM?

10. Standards and Research

11. Today we will… • Pique • Challenge • Engage • Model

12. Research on How People Learn Based on How People Learn, National Research Council, 2000

13. Application of Key Findings

14. Facts and Concepts(Our Definitions) • Facts or definitions are pieces of information. Focus is on verifiable and discrete details. • Concepts are over-arching ideas that clearly show the relationships between facts. Frequently abstract and often define the discipline.

15. Science Content: Gathering Evidence • Scan the entire unit/chapter and note the overarching concept. • Read your assigned section carefully. • Write the big ideas or concepts addressed in your section on sticky notes. Write concepts in complete sentences. Limit yourself to one concept per sticky note. • Note how the concepts in your section build from those in previous sections, how concepts in your section are extended in subsequent sections, and how the concepts in your section connect to the overarching concept of the unit or chapter • Note the context (e.g., real-world, engaging to students) in which the concepts are learned.

16. Gathering Evidence Science Content Work Students Do Assessment

17. NACL Team’s Work Pittsburgh Team, Summer 2001

18. Conceptual Flow Diagram Major Concepts in a Unit from BSCS Biology: A Human Approach

19. Looking at the Science Content Rubric • Note the criteria addressed in the rubrics. • Are any of these similar to those we brainstormed previously? • For your assigned row, visualize what would be present in “ideal” instructional materials that would justify a score of “5”. In other words, what would the evidence look like?

21. Applying the Science Content Rubric • With your group, give this unit/chapter a score for the bottom three rows in this rubric. • Note two or more pieces of evidence that justify each score. • On the AIM Summary Sheet, note strengths and limitations for the bottom three rows of the rubric.

22. AIM Process: Paper Screen Score Sheet

23. AIM Summary Sheet

24. Paper Screen Identify Criteria Gather Evidence Analyze Evidence & Apply Rubric Score Components Summarize Results Implementation Identify Criteria Gather Evidence Analyze Evidence & Apply Rubric Score Components Summarize Results AIM includes . . . Pre-Screen Select Scaling-Up for Full Implementation

26. Identify Criteria Student Understanding Teacher Implementation Science Content Work Students Do Assessment Work Teachers Do Gather Evidence Gather Evidence Gather Evidence Gather Evidence Gather Evidence Gather Evidence Analyze Evidence and Apply Rubric Analyze Evidence and Apply Rubric Analyze Evidence and Apply Rubric Analyze Evidence and Apply Rubric Analyze Evidence and Apply Rubric Analyze Evidence and Apply Rubric Score Components Score Components Summarize Results Score Components Score Components Score Components Score Components Summarize Results Merge Paper Screen and Implementation Results Select Instructional Materials The AIM Process: Paper Screen & Implementation

28. AIM Summary Sheet Content Work Students Do Assessment Work Teachers Do

29. Customize AIM • Establish your own criteria • Adopt or adapt existing rubrics • Vary the weighting system • Adjust to fit your schedule and needs

30. Reflection • What insights into instructional materials could AIM provide? • What questions have been raised by this brief introduction to AIM?

31. Why does AIM work? • Informed, justifiable, evidence-based decision-making • Customized and flexible • Leads to rich conversations about learning and teaching science • Builds common understandings about high-quality instructional materials • Informs PD for scaling-up implementation • Develops teacher-leaders

32. What people say about AIM... • “AIM really validates the money spent on curriculum.” • “Profitable to team building.” • “Teachers liked the process and it made sense to them.” • “Very useful for conceptual teaching and understandingmaterials. It also plays a role in understanding inquiry.” • “Vital to our ability to implement back in our district.” • “Analyzing materials based on data, not on feeling.” • “For the first time (in 32 years), I feel I have some insight into selecting instructional materials.” • “It makes great sense. Why would you choose curriculum any other way?”

33. Options… • FASS Leadership Team • Come to BSCS and learn to facilitate the process • BSCS Staff • Come to Florida to teach leadership teams around the state how to facilitate the process • Work directly with district teams

34. Contact Us: Center for Professional Development BSCS 5415 Mark Dabling Blvd. Colorado Springs, CO 80918 719.531.5550 X 119 pdcenter@bscs.org www.bscs.org