Meeting of Education Deans at the Chancellor ’ s Office

1. CTQ Item 1 Some Effects of CSU Teacher Education on K-12 Student Learning:A Preliminary Look at Initial CTQ Findings Meeting of Education Deans at the Chancellor’s Office Presentation by the CSU Center for Teacher Quality November 9, 2011

2. CTQ Purposes Today and This Winter • Today we will compare the CSU system’seffects on K-12 student learning with the effects of demographic and socioeconomic conditions that are known to be influential factors in students’ lives. • In the next 3-4 months, CSU Deans will receive reports of campuseffects on K-12 student learning. Question: Are demographic factors good benchmarks for assessing the magnitude of campus effects? • While CTQ analyzes K-12 data, we also consider alternative ways to report complex statistical findings to campuses. • Deans’ feedback about statistical reporting options was valuable last June. Feedback about new reporting formats would be valuable today. CTQ Item 1, Slide 1

3. Teacher Education FactorsBeing Examined Today and This Winter • Factor One: Today we will examine the overall effects of the CSU system on K-12 academic learning by comparing CSU teachers with non-CSU teachers. • Factor Two: Within the CSU, we will compare the impact of campus internship programs with that of supervised teaching programs on the same campuses. • To assess the magnitude of these teacher education factors on K-12 learning, we will compare them with concurrent effects of important demographic variables. • Later, when CSU Deans receive CTQ reports showing campuseffects on K-12 learning, one way to evaluate those effects would be to compare them with the demographic effects that we will look at today. • Your comments about today’s presentation will be welcome. Your feedback will shape the way CTQ reports campus effects to you in the next 3-4 months. CTQ Item 1, Slide 2

4. K-12 Students in CTQ Reports Today and This Winter • Our conversation today focuses on students who attend urbanschools in California’s largest metropolitan districts. • District names cannot be disclosed until CTQ satisfies the districts that its reports will not embarrass them. • In today’s analyses, students are predominantly low in parenteducation and income, high in ethnicdiversity, and include disproportionate numbers of ELs. • We are searching for teacher education practices that serve low-performing students who, in the past, have been under-served by education at all levels. • Teacher education practices that are most effective for these students will substantially help to close achievement gaps in our schools and universities. • This winter, campus-specific reports to Deans of Education will focus on the same groups of students as today. • Subsequent CTQ presentations and reports to campuses will focus on specific subgroups of urban students, and on students in rural regions of California. CTQ Item 1, Slide 3

5. Teachers in CTQ Reports Today and the Next 3-4 Months • Today’s primary focus is on first-year and second-year CSUteachers who were hired by the urban districts that provided our academic and demographic data about K-12 students. • We specifically focus on teachers of mathematics in urban middle schools & high schools. • At your subsequent meetings, CTQ presentations could focus on CSU elementary teachers and on secondary teachers of English, science, and history-social science. • Campus-specific reports to Deans will assess the effectiveness of MS-SS Programs in terms of student progress in English-language arts, math, science and history-social science. • CTQ limited today’s analysis to CSU teachers who earned Single Subject Credentials in Mathematics or Foundational Mathematics on CSU campuses or outside the CSU system. • CTQ data about each CSU teacher’s preparation came from campuses. CTC provided data about the qualifications of non-CSU teachers. • CTQ limits its analyses to first-year and second-year teachers based on employment history data provided by employing school districts. • CTQ checks each teacher’s assignments in relation to his/her preparation. CTQ analyses do not include teachers who were not prepared or authorized for their teaching assignments. CTQ Item 1, Slide 4

6. Subsequent Presentations Could Examine the Effects ofProgram Features and Teacher Attributes • When candidates enter CSU preparation programs, they have individual attributes that may influence their practices and effectiveness as classroom teachers. • Today we focus on the effects of CSU programs on student learning in secondary schools, without considering the attributes that teachers bring to the profession. • But the effects of internships on subsequent student learning may be influenced by teachers’ academic capabilities, which we don’t have time to explore today. • Future presentations will examine the concurrent academic effects of CSU program features and teacher attributes on the students of first- and second-year teachers. • Different analyses may have different implications about how CSU campuses might make teacher edu- cation more effective for K-12 pupils. CTQ Item 1, Slide 5

7. How CTQ Measures Student Academic Learning • Today’s findings are based on a set of norm-referenced exams named the California Achievement Tests, Version Six (commonly called the CAT-6s). • CTQ recognizes the inequalities that occur in standardized test scores, and concurs that California’s current tests measure only a narrow range of important school outcomes. • CTQ would welcome opportunities to assess teacher education’s effects on exciting outcomes such as creativity, critical thinking, and civic participation by K-12 students. • Currently, however, the state’s existing tests are the only available measures of student outcomes that are uniform for all teacher education programs. • Before initiating this project, CTQ reviewed the psychometric properties of the CAT-6 tests. • These tests are strong in measurement reliability. They also measure each student’s growth in each core subject and school year, which is valuable in evaluating teacher-ed. • While acknowledging the tests’ limitations, CTQ believes the CAT-6s are sufficiently sound for evaluating teacher education programs. • It would be valuable for CSU to rely on standardized tests inconjunctionwith other methods for assessing programs, including TPAs and systemwide surveys of teachers & principals. • In the future, new K-12 assessments that are currently being developed for California and other states may also serve as excellent measures of teacher education’s effectiveness. CTQ Item 1, Slide 6

8. An Evaluator’s Problem: Before Students are Taught byCSU Teachers, They Vary in Their Prior Learning of Math • CTQ does not evaluate CSU programs by looking only at test scores that students earn after being taught by CSU-prepared teachers. • Instead, CTQ analyzes every student’s end-of-year score in conjunction with that student’s score in the same subject at the end of the prior school year. • CAT-6 exams yield vertically-scaled scores, so CTQ subtracts each pupil’s pre-instruction score from her/his post-instruction score, and measures how much math each student learned when taught by a CSU-prepared teacher. • Today’s findings focus entirely on the magnitude of learning gains that students realize, in the aggregate, when they are taught by newly-prepared teachers. • Today and in campus-specific reports, CTQ uses statistical procedures that specialists in educational measurement and statistics endorse for evaluations of university programs for prospective teachers. CTQ Item 1, Slide 7

9. In this Project, CTQ Benefits from Expert Advice • At Stanford University, Linda Darling-Hammond has guided and assisted CTQ in this value-added assessment since 2007. • More recently, CTQ has conferred with Henry Braun, Professor of Research and Measurement in Education at Boston College, who is the Principal Investigator on a National Research Academy-funded study of alternative methods for evaluating teacher education’s effects on K-12 student achievement. • Another of CTQ’s frequent advisors is George Noell, Professor of Educational Psychology at Louisiana State University, who is the PI in annual evaluations of all Louisiana institutions, which focus entirely on K-12 student achievement. • CTQ also benefited from expert reviews of a series of CTQ grant applications to the Carnegie Corporation of New York, which were funded from 2006 through 2009. • The Center strives to incorporate the advice of these experts in its work, but all errors in CTQ reports are the responsibility of the CSU authors. CTQ Item 1, Slide 8

10. Another Problem for CTQ: Economic and Demographic Factors Significantly Influence Student Academic Learning • CTQ uses statistical methods that multiple experts recommended for evaluating teacher preparation programs based on measures of K-12 student learning. • Hierarchical linear modeling (HLM) enables us to account for economic and demographic factors while we assess teacher preparation programs in the CSU and outside the CSU. • Using HLM, CTQ minimizestheeffects of family conditions and student circumstances that influence student learning but are outside the control of educators at all levels. • CTQ data describing each student’s demographic circumstances and economic status were recorded administratively by school districts. CTQ protects these data securely. • Example: CTQ minimizes the effects of varying levels of education attained by the parent(s) of each student, using a six-level scale from no high school diploma to post-graduate degree. • Each CTQ analysis also minimizes the effects of family incomes in terms of each family’s eligibility or non-eligibility for free or discounted school meals (a three-level scale). • Each analysis also includes a six-level measure of each student’s English proficiency, including English learners as well as English speakers. • CTQ evaluations of teacher education also include a two-level record of each student’s disability status as recorded administratively by the school district. • Gender effects and race/ethnicity effects are also minimized in CTQ analyses. • Each analysis examines the effects of teacher preparation on student learning while holding each of these economic/demographic factors statistically constant. CTQ Item 1, Slide 9

11. Disclosure: CTQ’s Statistical Accommodations forDemographic and Economic Effects are Imperfect • CTQ’s capacity to minimize the effects of economic and demographic factors depends substantially on the quality of data provided by school districts. • California measures the English proficiency of English learners relativelywell, so this factor is minimized reasonably well in CTQ evaluations of teacher education. • Other factors such as family income are minimizedpoorly, however, because eligibility for school meals is an inexact measure of family resources. • The effects of parent education levels are also minimized imperfectly because districts sometimes record incorrect information. • Complex statistical analyses cannotovercomemeasurementerrors that reside in administrative data provided by our K-12 partners. • When CTQ reports the effects of teacher preparation, economic and demographic factors have been statistically minimized but their effects cannotberemovedentirely from the available evidence of student learning. CTQ Item 1, Slide 10

12. More Limitations: In Each Analysis,CTQ Includes Individual Students Selectively • In grades 7-11, CTQ routinely limits its analyses to students who took tests that were aligned with the courses they had completed. • Example: Fifty ninth-grade pupils took classes named Basic Math. Then they took a test in Algebra I. CTQ does not include their scores in an algebra analysis. • In all grades, CTQ evaluation results are limited to individual students who attended school more than half of the school year. • CTQ also limits each analysis to students whose CSU graduates taught the students for an entire school year (not a semester or quarter). • Schools try to administer make-up exams to students who were absent on test dates. Some make-up tests do not occur, however. • To minimize the effects of missing make-up tests and of district exclusions of low-performing students, CTQ evaluations are limited to elementary classes with 15-32 scores, and secondary classes with 20-40 scores. CTQ Item 1, Slide 11

14. Non-Disclosure of Today’s Results • CTQ continues to rely on school districts for new data that are needed for CTQ’s work. • The Chancellor’s Office asks campuses to restrict circulation and distribution of CTQ Slides 12 and 14. • These graphs should be shared only with CSU persons who are expected to avoid premature disclosure to non-CSU persons. CTQ Item 1, Slide 13

16. Many Made Important Contributions to This Work • All of the results reported today are built on extensive data-preparation work by Rachelle Fox in the Center for Teacher Quality. • All of today’s statistical findings were produced by Dr. Mary Wang, a professional statistician in the Center for Teacher Quality. • Campuses of the California State University provided valuable teacher education data that were essential for today’s analysis. • Urban school districts in California provided student data and teacher data that were also necessary. • CTQ is also grateful for patience by Assistant Vice Chancellors Bill Wilson and Beverly Young, who waited a long time for this information. dwright@calstate.edu