You’ve Got a Great Teacher Education Program so Why Doesn’t Anyone Know About it?

1. You’ve Got a Great Teacher Education Program so Why Doesn’t Anyone Know About it? Mark Girod Chair, Teacher Education Western Oregon University AASCU Academic Affairs Summer Meeting Friday, July 29th, 2011 Portland, Oregon

2. Strengths and challenges in your teacher preparation programs • Identify 2-3 strengths of the teacher preparation programs on your campus • Identify 2-3 challenges of your programs • Can you identify the essential challenges in teacher preparation today? • Bottom line: we need evidence driven programs, with shared outcomes, systematic programs of research and structured dissemination plans

3. Identifying an outcome measure: articulating the inference chain To get from teacher education to impact on pupil’s learning requires a chain of evidence with several critical links: empirical evidence demonstrating the link between teacher preparation programs and teacher candidates’ learning, empirical evidence demonstrating the link between teacher candidates’ learning and their practices in actual classrooms, and empirical evidence demonstrating the link between graduates’ practices and what and how much their pupils learn. Individually, each of these links is complex and challenging to estimate. When they are combined, the challenges are multiplied (p. 303). Cochran-Smith, M. (2005). Studying teacher education: What we know and need to know. Journal of Teacher Education, 56, 301-306.

4. Multi-leveled model… This is a complex and multilayered inference chain that requires examination of data at multiple levels including: • Candidate level variables (GPA, prior coursework, exam scores…) • We have much of this information about our candidates – but is it in a form we can use? • Program level variables (structure, courses, fieldwork, outcomes…) • What is the nature of these variables? Are these grouping variables? Independent variables? • Candidate learning (performance on anchor assignments…) • Variability? How do we rate scaffolding in our failure free systems? • Candidate practices in classrooms (observations of teaching, strategies…) • TWS? Observations? Data quality? Both dependent and mediating? Aggregate? • Contextual factors (classroom, school, community variables…) • Most salient? And are we measuring the right ones? • P-12 learning (kind and complexity of outcomes targeted and met...) • As evidenced by (assessment)? Aggregation? Definitions? Judgment?

5. Necessary conditions… At least five conditions seem to be needed if a teacher preparation program is going to serve as context for research: Persons responsible for the management and operation of the program must be inclined toward experimentation Persons responsible for the management and operation of the program must view it as subject to continuous change, and view a systematically designed program of research on its effectiveness as a major data source for its change. Data of a quality that will support trustworthy research must be collected as a normal part of program operation. Sophisticated data management, storage, retrieval and display capabilities must be available. There must be an advisory structure to insure the research that is pursued has value to people in the program as well as to the profession at large.

6. How to run the system… Need to establish a conceptual model linking the inference chain: • P-12 learning is the product of… • Relationship between teacher/curriculum/learner… • Nested in contexts… Kinds of studies needed: • Policy oriented – cost/benefits, program evaluation • Practice oriented – follow-up studies, short and long-term effect • Basic research – hypothesis testing, hypothesis generating • Quality assurance – form use, rating patterns, inter-rater reliability • Data display – development of procedures to display outcomes • Instrumentation – refinement of instruments

7. And the results… • Teacher preparation programs as contexts for research • Empirically validated systems of teacher preparation • Model building and theory building that would lead to an empirically validated field of teacher preparation • Policy and practices informed by evidence (best case scenario…) • Better teachers and more and better P-12 learning • The argument suggests we have too much to lose to not attempt to become an empirically driven business… and TWS is uniquely well positioned to help move us ahead if we can improve the quality of the data and our ability to aggregate • 1975…

8. So what barriers have kept us from getting there? Discussion: • What are some of the barriers to data driven teacher preparation programs that connect to P-12 student learning? • What will be the consequences if we don’t get there? • How can we move toward this?

9. What is a Teacher Work Sample? A Teacher Work Sample is an authentic performance assessment completed in a real-world setting that demonstrates a candidate’s ability to assess, plan, instruct, and reflect in a standards-based educational system and impact student learning in a positive manner.

10. The revolution of standards-based schooling… The standards set for learning in today’s schools define the successive bars to be reached by students as they progress in their learning, and standards-linked assessments indicate where students stand at a particular point in time with respect to a particular bar, but it is each student that needs to reach each bar and the main job of teachers is to help each student in each classroom make steady progress toward each bar that lies immediately ahead. (Del Schalock, 2006)

11. Use for teacher licensure • Since 1989, in Oregon, each teacher candidate has been required to successfully implement two teacher work samples prior to being awarded initial licensure. • A teacher work sample is an empirically validated performance assessment that includes: • A setting description • Pre-assessment • Learning outcomes • Lesson plans • Post-assessment • Analysis of learning data • Reflective analysis

12. Principles… • An instructional program needs to be aligned with and supportive of what candidates are asked to do, including the documentation and reporting that is required in completing a work sample. • School contexts that model and are supportive of what candidates are asked to do need to be available for practicum and student teaching placements. • A supervision, evaluation, and feedback system needs to be in place that provides guided practice in applying and carrying out the tasks teacher work sampling demands of candidates.

13. Principles… • Judgments about a candidate’s effectiveness as a teacher need to take into account the gains in learning made by every student taught. • Documentation of a candidate’s effectiveness as a teacher needs to be accompanied by observations of practice and descriptions of context, as well as evidence of learning gains by students. • Multiple lines of evidence need to be considered in reaching a recommendation for licensure, only some of which come through teacher work sampling.

14. Principles… • Multiple reviewers of evidence need to be involved in preparing a recommendation for a license to teach, only some of whom represent a teacher education faculty. • Evidence needs to be assembled and reported by a teacher education faculty on the confidence that can be placed in all lines of evidence collected through teacher work sampling that inform a licensing decision (the reliability and validity of information used). • A conceptual map is needed to help inform and give meaning to candidates regarding the way in which the previous 8 principles inform the TWSM.

16. Undergirding skills Candidates prepare products or components of a work sample as evidence of their developing skillfulness. These skills, when employed with acumen, facilitate connections between teaching and learning. It is connecting teaching and learning that we value… and we judge it by examining the products of the work sample. Therefore we must distinguish between skills and products. See handout distinguishing between the undergirding skills and the products developed in a teacher work sample.

17. Weaving a tapestry When a candidate successfully weaves together the underlying skills of the teacher work sample with real children, in real 21st century schools, we argue that they have maximized opportunities for P-12 student learning. The teacher work sample stands as evidence of this connecting teaching and learning.

18. TWS Multi-purposing Use of TWS results by programs • Disaggregated data • Candidates’ ability to teach to state and national standards • Candidates’ ability to enact best practices in content based pedagogy linked to national professional standards • Candidates’ ability to impact student learning • Aggregated data • Program accountability • Program improvement • Context for research See handout on empirical foundations and supports for TWS

19. Optimism about TWSM Reasonableness: hard to argue that we should not value P-12 student learning Feasibility: can be used in any context, at all levels, and is practically familiar Serves multiple purposes: both a pedagogical model as well as an assessment tool Empirically validated:TEP-II, a longitudinal study of teachers used TWS and explained more than 40% of the variance in student learning* Schalock, H.D., Schalock, M., & Girod, G. (1997). Teacher work sample methodology as used at Western Oregon State College. In J. Millman (Ed.), Grading teachers, grading schools. Thousand Oaks, CA: Corwin. *For more information on TEP-II see http://www.tr.wou.edu/tep/products.html

20. Trustworthiness • Validation of TWSM extended at Idaho State University and Western Kentucky University • Dependable rating system and rater training • Bias free rating • Validation of frequency, criticality, necessity, and representativeness of TWS actions to actual practices • Linked TWS performance to levels of P-12 learning • Denner, P., Norman, A., Salzman, S., Pankratz, R., & Evans, S. (2004). The Renaissance Partnership teacher work sample: Evidence supporting score generalizability, validity, and quality of student learning assessment. In E. Guyton and J. Rainer Dangel (Eds.), Research linking teacher preparation and student performance. Dubuque, Iowa: Kendall.

21. Systems-level approach • Teacher work sampling as a compass: • Keeps programs focused on P-12 student learning • Keeps faculty focused on teaching skills, knowledge, and dispositions that help to maximize P-12 student learning • Keeps institutions focused on university teaching as a valued outcome • Codifies a systems-level commitment to “connecting teaching and learning”

22. Teacher development… Students responded to the prompt: Did the process of completing a teacher work sample help you think differently about teaching and learning? If so, how? If not, why not? Analyzing my teaching using the work sample really helped me understand the importance of making adaptations and modifications as needed. It made me realize the importance of alignment between context, instruction, adaptations, and assessment. I don't think I really understood this before.

23. Teacher development And from another student: The teacher work sample has helped me develop a sense of necessity when it comes to being sensitive and attentive to details – especially the needs and abilities of my students. It helped me recognize the importance of paying attention to pre-assessment results when designing instruction. I can modify instruction to better work with my students' needs.

24. NCATE’s focus on student learning A centerpiece of the NCATE performance-based system is collecting and aggregating data to show that candidates have the knowledge and skills to teach effectively so that students learn, a requirement that directly impacts approximately two-thirds of all new teacher graduates nationally. (Westat, 2006)

25. TEAC’s focus on student learning Although less explicit, TEAC also references P-12 student learning as found in Quality Principle 1.3 Teaching Skill which states that teachers must “act on their knowledge in a caring and professional manner that would lead to appropriate levels of achievement for all their pupils.” NOTE: Less than one year ago, NCATE and TEAC consolidated and teacher preparation accreditation is now under a new organization called Council for the Accreditation of Education Preparation (CAEP)

26. Art or science? • Teacher education lacks a unifying theory founded on empirically defensible assertions • If developing a science of teacher education is possible it includes mapping a complex set of interactions, nested in multiple levels, resting on shifting contextual milieu

27. TWS well positioned • Given attention to each stage of Cochran-Smith’s inference chain, and sensitivity to multiple levels of context… teacher work sampling is methodologically well positioned to more systematically explore empirical connections between preparation, practices, and P-12 student learning • Though VAM is powerful… it is not yet particularly useful for informing the work of teacher education beyond giving us hope that what we do matters!

28. Future directions • Maturation of TWSM: • Ongoing instrument validation predictive validity • Dimensionality of undergirding constructs • Codifying scoring procedures • Codifying non-negotiables at multiple levels • Contextualizing methodology • Aggregation of TWS information • Contributing to a scholarship of teaching education • Linking to data warehouses cross-validation • Etc…

29. Research Questions… Descriptive study: How does the learning of P-12 students vary, as evidenced by data from the TWS, across: • Kind and complexity of outcomes pursued and met • Individuals and groups of students (ethnicity and instructional program) • Variations in preparation experiences - planned variation studies • Variations in context, candidate level variables, program level variables… • Variations in instructional strategies, assessment strategies, and efforts to differentiate

31. Initial observations… • More variance within groups than between groups - for ethnicity and academic program • Between program variance illustrates pedagogical differences not program quality • Need for a web-based, data-entry portal that aggregates P-12 learning gain data • Will require within-program conversations about common language and expectations • Will afford programs the ability to link the evidentiary chain and become data-driven settings for the improvement of teacher education… at the program level

32. P-12 student impact • Oregon Collaborative Research Initiative: • Through examination of 98 teacher work samples representing the learning of 2,400 children: • Overwhelmingly, students were focused on low-level, cognitive outcomes • Only small, non-significant differences for student-level effects (race/ethnicity, academic program) • Only small, non-significant differences for school-level effects (SES, urban/rural, size)

33. Learning Gain Aggregator… • Tension in balancing pedagogical goals with evaluation/research goals • Replace existing efforts don’t add work to students or faculty • Getting instructors on board • Using FileMaker Pro for a web-accessible database

34. Candidate data…

35. Learning outcome data…

36. Outcomes by student data…

37. Reporting… cuts by fields…

38. And a real example…

39. And one more real example…

40. What will we be able to say and do? • Descriptive studies of the kind and complexity of outcomes pursued and met by: (1) candidates with particular qualities (2) in particular kinds of programs (3) in particular kinds of placement settings (4) working with particular kinds of P-12 children Ex: What do the learning profiles look like for candidate seeking initial licensure in science, working in urban schools, with African American girls, receiving special education services, when pursuing performance outcomes for learning?!! • Planned variation studies: Programs become laboratories for experimentation with a fixed dependent variable

41. Challenges and opportunities…and this is the last slide! • Getting faculty to agree to a common vision of their work • Having faculty leadership to establish proof of concept • Protecting time and resources to get there • Gives faculty a defensible outcome variable for all research • Provides evidence on program effectiveness • Moves teacher preparation ahead Mark Girod – girodm@wou.edu Western Oregon University