1 / 53

Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations

This report evaluates the quality of evaluations on the effectiveness of NSF-supported and commercially generated mathematics curriculum materials, providing recommendations for designing more reliable evaluation studies.

mariecollins
Download Presentation

Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations Steering Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials Mathematical Sciences Education BoardCenter for EducationDivision of Behavioral and Social Sciences and Education National Research Council of the National Academies The National Academies Press Sponsored by the National Science Foundation

  2. Committee Members • Jere Confrey, (Chair, Washington University, St. Louis • Carlos Castillo-Chavez, Arizona State University • Douglas Grouws, University of Missouri • Carolyn Mahoney, Elizabeth City State University • Donald Saari, University of California, Irvine • William Schmidt, Michigan State University • Patrick Thompson, Vanderbilt University • William Velez, University of Arizona

  3. Reviewers of the Report REVIEWERS Doug Carnine, University of Oregon Mary Lindquist, Columbus State University R. James Milgram, Stanford University Leona Schauble, Vanderbilt University Alan Schoenfeld, University of California at Berkeley Norman Webb, University of Wisconsin at Madison Iris Weiss, Horizon Research Inc. Adam Gamoran, University of Wisconsin-Madison, Review Coordinator William Howard, Independent Consultant, Review Monitor

  4. The Committee’s Charge • Evaluate the quality of the evaluations of the thirteen National Science Foundation (NSF)-supported and six of the commercially generated mathematics curriculum materials; • Determine whether the currently available data are sufficient for evaluating the efficacy of these materials, and if not; • Develop recommendations about the design of a project that could result in the generation of more reliable and valid data for evaluating these materials.

  5. Reminder-- The committee emphasizes that it was not charged with and therefore did not: • evaluate the curriculum materials directly; or • rate or rank specific curricular programs.

  6. Primary Contribution-- …to clarify the proper elements of an array of evaluation studies designed to judge the effectiveness of mathematics curricula and clarify what standards of evidence would need to be met to draw conclusions on effectiveness.

  7. NSF-Supported Mathematics Curriculum Materials Elementary School: • Everyday Mathematics (EM), Grades K-6 (SRA/McGraw-Hill) • Investigations in Number, Data and Space, Grades K-6 (Scott Foresman) • Math Trailblazers, Grades K-6 (Kendall Hunt Publishing Company)

  8. NSF-Supported Mathematics Curriculum Materials Middle School: • Connected Mathematics Project (CMP), Grades 6-8 (Prentice Hall) • Mathematics in Context (MiC), Grades 5-8 (Holt, Rinehart and Winston) • MathScape: Seeing and Thinking Mathematically, Grades 6-8 (Glencoe/McGraw-Hill) • MathThematics (STEM), Grades 6-8 (McDougal Littell) • Middle School Mathematics Through Applications Project (MMAP) Pathways to Algebra and Geometry, Grades 6-8 (currently unpublished)

  9. NSF-Supported Mathematics Curriculum Materials High School: • Contemporary Mathematics in Context (Core-Plus), Grades 9-12 (McDougal Littell) • Interactive Mathematics Program (IMP), Grades 9-12 (Key Curriculum Press) • MATH Connections: A Secondary Mathematics Core Curriculum, Grades 9-12 (IT’S ABOUT TIME, Inc.) • Mathematics: Modeling Our World (MMOW/ARISE), Grades 9-12 (W.H. Freeman and Company) • Systemic Initiative for Montana Mathematics and Science (SIMMS) Integrated Mathematics, Grades 9-12 (Kendall Hunt Publishing Company)

  10. Commercially Generated Mathematics Curriculum Materials Elementary School: • Math K-5 (Addison Wesley/Scott Foresman) • Harcourt Math K-6 (Harcourt Brace)

  11. Commercially Generated Mathematics Curriculum Materials Middle School: • Applications and Connections (Glencoe/McGraw-Hill) • An Incremental Development (Saxon)

  12. Commercially Generated Mathematics Curriculum Materials High School: • Larson Series (Houghton-Mifflin/McDougal Littell) • University of Chicago Schools Mathematics Project (UCSMP) Integrated Mathematics (Prentice Hall)

  13. Work of the Committee • Collect and review evaluation studies. • Solicit expert testimony. • Develop evaluation framework. • Classify studies, and establish dimensions and criteria for each. • Draw conclusions on corpus of studies. • Prepare report with findings and recommendations.

  14. Winnowing Evaluation Studies • Eliminate “background” and informative studies (225) and studies not included as research reports such as project reports, studies that did not meet established methodological criteria, etc. (281) • From 698 studies to 192 • Classify 192 by program type and by type of evaluation

  15. Distribution of Studies Reviewed * University of Chicago School Mathematics program was treated as a hybrid in some analyses

  16. Further Winnowing of Evaluation Studies • For each methodology, we set criteria for further inclusion and more rigorous study

  17. Studies Examined

  18. “...at least minimally methodologically adequate” • Required:  • Include quantifiably measurable outcomes such as test scores, responses to specified cognitive tasks of mathematical reasoning, performance evaluations, grades, and subsequent course taking; and • Provide adequate information to judge the comparability of samples. • At least one of the following additional design elements: • A report of implementation fidelity or professional development activity; • Results disaggregated by content strands or by performance by student subgroups: and/or • Multiple outcome measures or precise theoretical analysis of a measured construct, such as number sense, proof, or proportional reasoning.

  19. First Conclusion “…the corpus of evaluation studies as a whole across the 19 programs studied does not permit one to determine the effectiveness of individual programs with high degree of certainty, due to the restricted number of studies for any particular curriculum, limitations in the array of methods used, and the uneven quality of the studies.”

  20. Caveat “This inconclusive finding should not be interpreted to mean that these curricula are not effective, but rather that problems with the data and/or study designs prevent confident judgments about their effectiveness.”

  21. Note • We set a high evidentiary standard • The development of method is a gradual process • Need to study base and work to accumulate knowledge • “There are studies that provide compelling data concerning the effectiveness of the program in a particular context.” (p.197)

  22. Framework for Evaluating Curricular Effectiveness Two Parts: • 1. Articulation of Program Theory • 2. Methodological Choices and Critical Decisions

  23. Framework for Evaluating Curricular Effectiveness

  24. Methodological Choices and Critical Decisions

  25. A Scientifically Valid Study… • Address components in the framework • Conform to methodological expectations of appropriate category of evaluation (content analysis, comparative study, case study)

  26. Curricular Effectiveness • “…cannot be established by a single scientifically valid study: instead, a body of studies is needed.” • “…is an integrated judgment based on interpretation of a number of scientifically valid evaluations that combine social values, empirical evidence, and theoretical rationales.”

  27. General Principle • “… a curricular program’s effectiveness should be ascertained through the use of multiple methods of evaluation, each of which is a scientifically valid study. Periodic synthesis of the results across evaluation studies should also be conducted. ”

  28. Explicit Standard for Programs “Scientifically Established as Effective” A collection of studies that use a combination of methodologies that meet these specified criteria: • content analyses by at least two qualified experts (a Ph.D.-level mathematical scientist and a Ph.D.-level mathematics educator) (required); • comparative studies using experimental or quasi-experimental designs, identifying the comparative curriculum (required); • one or more case studies to investigate the relationships among the implementation of the curricular program and the program components (highly desirable); and • a final report, to be made publicly available, should link the analyses, specify what they convey about the effectiveness of the curriculum, and stipulate the extent to which the program’s effectiveness can be generalized (required).

  29. Independence and Impartiality of Evaluations “…summative evaluations [should] be conducted by independent evaluation teams with no membership by authors of the curriculum materials or persons under their supervision.”

  30. Representativeness “Evaluations of curricular effectiveness should be conducted with students that represent the appropriate sampling of all intended audiences.”

  31. Documentation of Implementation “Evaluations should present evidence that provides reliable and valid indicators of the extent, quality, and type of the implementation of the materials. At a minimum, there should be documentation of the extent of coverage of curricular material (what some investigators referred to as “opportunity to learn”) and of the extent and type of professional development provided. ”

  32. Curricular Validity of Measures “A minimum of one of the outcome measures used to determine curricular effectiveness should possess demonstrated curricular validity: it should comprehensively sample the curricular objectives in the course, validly measure the content within those objectives, ensure that teaching to the test (rather than the curriculum) is not feasible or likely to confound the results, and be sensitive to curricular changes. ” (differentiate from curricular alignment to systemic factors)

  33. Recommendations for Dimensions of Content Analysis • clarity, comprehensiveness, accuracy, depth of mathematical inquiry and mathematical reasoning, organization, and balance (disciplinary perspectives). • engagement, timeliness and support for diversity, and assessment (learner-oriented perspectives). • pedagogy, resources, and professional development (teacher- and resource-oriented perspectives).

  34. Recommendations for Dimensions of Content Analysis • Connoisseurial assessment (Eisner, 2001) • Full disclosure • Statement of expertise • Clear choice of standards or comparative curricula • Explicit decision on whether to include content and/or pedagogy

  35. Recommendations for 7 Criteria for Comparative Analyses • identify comparative curricula; • if not random assignment, establish adequate comparability; • select the appropriate unit of analysis; • document extent of implementation fidelity; • select outcome measures that can be disaggregated by content strand; • conduct appropriate statistical tests and report effect size; • disaggregate by gender, race/ethnicity, socio-economic status (SES), and performance levels, and express constraints to generalizability of study.

  36. Investigation into the Impact of Rigor in Comparative Analyses • Pry apart issues of method from patterns of results • Consider alternative interpretation, to test results for sensitivity or robustness to changes in design • Tease out among the myriad of variables, the ones most likely to produce, interfere with, suppress, modify and interact with the outcomes • Build on results of previous studies.

  37. Investigation into the Impact of Rigor in Comparative Analyses • Conducted a methodological examination of how the patterns of results varied between those meeting one or more of the 7 criteria and the overall average results by program type. (FILTERING STUDY) • Method: coded all outcomes of comparative studies by program type into triplets (a, b, c) representing the proportion of the findings that (a) were positive and statistically significantly stronger than the comparative program, (b) were negative and statistically significantly weaker than the comparison program, and (c) showed no significant difference between the control and comparative group. • Example: (1,0,0)

  38. Investigation into the Impact of Rigor in Comparative Analyses Report coded results by program type in recognition of different program design specifications No adjustments made for sample size or for weakness in design (such as units of analysis chosen) • On average the evaluations of the NSF-supported curricula (n=46) in this subgroup had reported stronger patterns of outcomes in favor of the experimental curricula than had the evaluations of commercially generated curricula (n=17).

  39. Investigation into the Impact of Rigor in Comparative Analyses

  40. Investigation into the Impact of Rigor in Comparative Analyses “…due to our call for increased methodological rigor and the use of multiple methods, this result is not sufficient to establish the curricular effectiveness of these programs as a whole with adequate certainty. “ “Testable hypothesis [for] further studies”

  41. Investigation into the Impact of Rigor in Comparative Analyses • Methodological conclusion from filtering studies: Increased rigor leads in general to less strong outcomes, but…never led to completely contrary results.

  42. Further Analyses in Chapter 5 Analysis by program type by content strand Analysis of equity outcomes (race, gender, SES and Limited English proficiency) Analysis by program, by grade band level, and by content strand and equity

  43. Recommendations for Criteria for Case Studies • stipulate clearly what they are cases of, how claims are produced and backed by evidence, and what events are related or left out and why; and • identify explicit underlying mechanisms to explain a rich variety of research evidence.

  44. Recommended Actions At the federal level: • Specify explicit expectations and methodologies in calls for curriculum evaluation proposals • Deny continued funding for curricular programs that fail to present recommended evaluation criteria • Charge a federal agency to collect and maintain district- and school-level data on curricula • With state and local agencies, provide training in conducting and interpreting studies of curricular effectiveness

  45. Recommended Actions At the state level: • Develop resource for collection and maintenance of data on issues of curricular implementation • Provide districts with training to conduct feasible, cost-efficient, and scientifically valid studies of curricular effectiveness

  46. Recommended Actions At the district level: • Improve methods of documenting curricular use and link it to student outcomes • Maintain records of teachers’ professional development activities related to curricula and content learning • Ensure that all study participants have a fair opportunities to learn curricular units

  47. Recommended Actions For publishers: • Differentiate market research from scientifically valid evaluation studies • Make evaluation data available to potential clients using federal funds to purchase curriculum materials

  48. Need for Further Research On development of outcome measures • at upper secondary level • non-numerical topics at the elementary level

  49. Need for Further Research On topics of • Interplay among curricular implementation and professional development • Methods of observing and documenting types and quality of instruction • Methods of parent and community education and involvement • Targets of curricular controversy

More Related