Considering the Roles of Research and Evaluation in DR K-12 Projects

1. Considering the Roles of Research and Evaluation in DR K-12 Projects December 3, 2010

2. Discovery Research K-12Program Solicitation All DR K-12 proposals must have a plan for formative and summative evaluation of project research and development work. The evaluation should focus on the validation of, fidelity to, and the usefulness of the development and research processes to achieve the targeted outcomes. Individually write down: One idea you are confident you know about evaluation of research in DR K-12 projects, and One question you have about evaluation of research in DR K-12 projects

3. Discuss: 10 minutes • At your tables, share with your group the two things you wrote down • As a group, identify one idea and one question to share with the room

4. Discovery Research K-12Project Evaluation • Projects focused at different stages in the DRL cycle of research and development will naturally choose different development and implementation strategies and different research and project evaluation methods. • The research, development, implementation and evaluation activities need to be appropriately rigorous.

5. Discovery Research K-12 Project Evaluation The objectives of the evaluation include: • ensuring that the project is making satisfactory progress toward its goals; • recommending reasonable, evidence-based adjustments to project plans; • determining the value of the outcomes of the project; and • attesting to the integrity of outcomes reported by the project.

6. DR K-12 Project Evaluation Roles, by Project Component and Evaluation Purpose

7. An Example For two projects that are developing and testing mathematics teacher professional development programs, the evaluation: • Works with the projects to devise logic models of the assumptions, activities, and intended outcomes of the PD • Uses the logic model in Design Critique and Critical Friend roles to raise concerns and offer considerations about conceptual framing, sampling, instrumentation, data collection, analysis, and interpretation of findings

8. An Example For a project that is studying efficacy of a mathematics teacher professional development program in settings with large proportions of ELL students, the evaluation: • Documents processes by which the project is preparing facilitators to use the PD program • Documents processes in terms of the nature of implementation of the program in field sites, both in terms of fidelity to the PD program and adaptation to the particular opportunities and challenges of the local ELL contexts

9. An Example For a project that is studying impact of school-based mathematics coaches on teaching and learning in elementary schools, the evaluation: • Audits the research by conducting standards of evidence reviews of the draft research report, and coordinating preliminary peer reviews by selected experts in the field

10. Evaluation of Research Components Task • You will receive two handouts describing hypothetical DR K-12 projects, with an emphasis on the project’s research components • Decide together to begin with one of the two examples

11. Evaluation of Research Components Task: 30 minutes • Consider together the important evaluation purposes for the project’s research component • Discuss the benefits/drawbacks of having different parties (internal, external, advisory board) serve each evaluation purpose you identify for the project • Repeat for the other project

12. Wrap-Up • What revisions or additions might you now suggest for the matrix of evaluation roles by project component and evaluation purpose? • Remaining questions about evaluation of research components of DR K-12 projects.