120 likes | 311 Views
Differential Effects of Participatory Evaluation in a National Multi-site Program Evaluation. Frances Lawrenz University of Minnesota. Collaboratives for Excellence in Teacher Preparation.
E N D
Differential Effects of Participatory Evaluation in a National Multi-site Program Evaluation Frances Lawrenz University of Minnesota
Collaboratives for Excellence in Teacher Preparation • CETP program promotes comprehensive change in the undergraduate education of future teachers by supporting cooperative, multiyear efforts to increase substantially the quality and number of teachers well-prepared in science and mathematics, especially members of traditionally underrepresented groups.
CETP Characteristics • Each CETP was unique • All engaged faculty from STEM and education • All included several institutions of higher education (including community colleges) in a particular geographic area • All included mechanisms for improving undergraduate education in the sciences and mathematics. • All had some sort of relationship with schools within the geographic area. • All offered scholarships to students from underrepresented groups. • The ranges within these were broad however e.g., from 3 institutions to 13 or more, and all had unique elements like liaisons with Tribal Colleges or industrial/science internship placements, etc.
History of CETP Evaluation • Initially, project level emphasis on evaluation • Program monitoring evaluation required—negative experience • First conference with evaluators—Corridors for Collaboration • Second conference with evaluators • Funding of CETP CORE evaluation
CORE Evaluation Questions • Three main areas: • Institution • K-12 Teachers • K-12 Students • Foci: • Collaborations • Comparisons to standards • Comparisons to other groups
Institutions a. How supportive of SMET reform education policies and procedures are the participating CETP institutions? SOURCES: Deans/Department Chairs, Faculty, PI b. How successful have the CETPs been at course reform? SOURCES: Deans/Department Chairs, Faculty, College Students, PI c. What impact has CETP had on the system or structure of the teacher education systems at the participating institutions?SOURCES: Deans/Department Chairs, Teachers, Principals, PI, Faculty, QRC
Teachers a. How well do CETP teachers demonstrate the knowledge and skills espoused by the National Science, Mathematics and Technology Education standards?SOURCES: K-12 Students, CETP Teachers, Classroom Observation Protocol, Rubric, Principals of CETP b. How do CETP teachers and the classrooms they create differ from non-CETP teachers and the classrooms they create?SOURCES: K-12 Students, Teachers, Classroom Observation Protocol, Rubric, Principals c. What outcomes have the participating higher education institutions, their faculty members, or the CETP contributed to the K-12 schools?SOURCES:CETP Teachers, Deans/Department Chairs, Principals, PI, Faculty
Students a. Are students learning what is expected in the SMET education standards?SOURCES: K-12 Students, PI, Classroom Observation Protocol, Rubric b. Are there differences in student outcomes for CETP and non-CETP teachers?SOURCES: K-12 Students, Classroom Observation Protocol, PI
Positives • Good working relationships • Shared expertise; website, list serve, emails, and meetings • Quality data collection instruments • Group negotiations with NSF • More effort on unique aspects • Incentives; $15,000, print and mail out instruments, help with IRB, data entry and return
Negatives • Instruments not matched exactly to CETP • Time and labor intensive data collection • All levels of participants required to participate • Training required to do classroom observations • Not required by NSF
Recommendations • Plan for the program evaluation before funding begins • Require participation in the evaluation • Consider partial participation—either only some sites participating or some sites collecting only some data • Have project evaluators in the loop from the beginning • Substantively involve PIs • Have the program evaluation data play a critical role in project evaluation