Lynn Holdheide and Dan Reschly, Ph.D. Vanderbilt University July 20, 2010

1. TQ Research & Policy Brief: Challenges in Evaluating Special Education Teachers and English Language Learner Specialists Lynn Holdheide and Dan Reschly, Ph.D. Vanderbilt University July 20, 2010 OSEP Project Director’s Conference

2. About the National Comprehensive Center for Teacher Quality • The National Comprehensive Center for Teacher Quality (TQ Center) is a federally funded partnership whose mission is to help regional comprehensive centers and states carry out the teacher quality mandates of the Elementary and Secondary Education Act (ESEA) as reauthorized by the No Child Left Behind Act. • Partners: • Learning Point Associates • Vanderbilt University • ETS

3. Today’s Goals • Seeks to build the capacity of participants to • Articulate the challenges identified with evaluating special education teachers through value-added and other measures of teacher evaluation. • Actively participate in the creation or redesign of teacher evaluation models that support the development of strong, valid and reliable teacher evaluation policies and practices that recognize and promote the unique contribution of special education teachers.

4. The Purpose Identify the specific challenges in evaluating this population of teachers. Determine the current status of state policy and practice. Identify promising evaluation practices and instruments. Provide guidance and policy recommendations to districts and states.

5. The Inquiry Review of policy/literature Survey inquiry Series of interviews with state- and district-level practitioners and researchers Data collection period: December 2009–April 2010 • Designed in collaboration with Council for Exceptional Children (CEC) and national experts • State and local survey • Respondent pool: state and local directors (identified within CEC’s Council of Administrators of special education listserve) 1,143 total respondents

6. Modification of Evaluation Processes for Special Educators Among the local administrators, reported that contractual agreement prevented modification in the evaluation process. 81%

7. Opinions Regarding Special Education Teacher Evaluation Strongly Agree or Agree 92% 32% 84%

8. Evidence-Based Practices • Meeting the needs of “diverse” learners may not attend to the following: • Special skills (individualized education program [IEP] facilitation, collaboration, secondary transition, social and behavioral interventions, compliance with legal mandates) • Evidence-based instructional methods (direct/explicit instruction, scientifically based reading instruction, learning strategy instruction)

9. Multiple Measures Use more than one measure. 95+% Survey inquired about current practice. Respondents indicated value-added models in future evaluation efforts.

10. Practical Example: District of Columbia IMPACT Special Education 10% 50% 5% 5% 15% 15% • Individual Teacher Value-Added Scores • Non-Value-Added Achievement • Teaching and Learning Framework • Commitment to the School • School Value-Added Scores • Core Professionalism • IEP Quality Plan • IEP Timeliness

11. Opinions Regarding Use of Student Achievement for Special Educators Strongly Agree or Agree 60% 21% 73%

12. Student Growth MeasuresPractical Examples • Austin Independent School District, Texas • Student Learning Objectives • One is targeted toward classroom performance. • One is targeted toward particular skills or subgroups of students. • Norwell Public Schools, Massachusetts • Progress on the IEP is factored into evaluation of special educators. • Both districts are heavily dependent on teacher training and support.

13. Observation Protocols Use the same observation instrument as that of general education teachers. Use a modified or different observation instrument. Align to the state’s professional teaching standards. 26% Didn’t know. 51% 85% “Our evaluation tool was developed in the district over 40 years ago.” “Our current evaluation system is outdated and applied to nothing.” 12%

14. Observation ProtocolPractical Example • Alabama Department of Education’s Professional Education Personnel Evaluation Program • Slightly modified for the following: • Specialty area systems (speech paths, library specialist) • Teachers of students with significant cognitive disabilities • Competencies added in certain areas (e.g., classroom is expanded to include community settings, and academic content is expanded to include functional life skills.)

15. Expert Opinions Regarding Evaluators Strongly Agree or Agree 77% 61% 60% Require training for evaluators. Require specialized training. 12%

16. Practical Examples • Toledo’s Peer Assistance and Review • School-based teams evaluate. • Norwell Public Schools, Massachusetts • All teachers are evaluated using the same instrument. • Two formative assessments are conducted: • One with principal • One with special education administrator • Each evaluator focuses on expertise areas. • Both work collaboratively to develop summative evaluation.

17. Opinions Regarding Attribution in Coteaching Setting Strongly Agree or Agree 85% 75% 13%

18. Policy and PracticeRecommendations Include special education administrators when revamping/designing evaluation frameworks. Identify a common framework that defines effective teaching for all teachers, differentiating for special educators as appropriate. Integrate evidence-based practices for students with disabilities into evaluation models.

19. Policy and PracticeRecommendations Improve data quality. In addition to─or, in some situations, in the absence of─appropriate standardized assessment data, incorporate other reliable evidence of teachers’ contributions to student learning into the teacher evaluation system, such as progress toward accomplishing IEP objectives and student learning objectives across broad academic and behavioral domains.

20. Policy and PracticeRecommendations Ensure that evaluator training includes explicit training for evaluators of special educators and/or consider establishing a model of peer-to-peer observations or a model in which evaluators are matched to specific disciplines. Collaborate with teacher preparation programs to ensure that evidence-based practices are incorporated into teacher preparation coursework and professional development activities.

21. Evidence-based Practices in Special Education and Practice • Scientifically based instruction ESEA (2002) and IDEA (2004) • IES criteria and evolution to evidence-based practices • Research supported evidence-based practices in special education • ABA and its many variations/application • Direct instruction, big D and little d, reading and mathematics • Formative assessments with instructional decision making • Learning strategies

22. What Works? Research Foundations From Meta-Analysis TreatmentEffect Size • Applied Behavior Analysis + 1.00 • Formative Evaluation: Curriculum-Based Measurement+Graphing+Decision Rules+Reinforcement + 1.00 • Explicit Instruction and PS + .70 to 1.50 • Comprehension Strategies + 1.00 • Mathematics Interventions +.60 to 1.10 • Writing Interventions +.50 to .85 • Matching instruction to learning styles? 0.00 Note: These effect sizes are stable across cultural groups. Sources: Kavale, 2005

23. Scientifically Based Instruction in Reading • Reading Curricula content (Snow, Burns, & Griffin, 1998) • Phonemic Awareness • Phonics • Fluency • Vocabulary • Comprehension • Problem of teacher preparation • Vanderbilt University/TQ Center innovation configurations: reading, classroom behavior, inclusive services, learning strategies (Reschly, Smartt, & Oliver, 2007) • PLUS • Direct, systematic instruction • Universal screening and formative evaluation 23

24. Research review 1900–1965 Early reading, K–3 Code versus meaning emphasis Phonics or whole word Code superior, especially for struggling readers Lamented the generally poor preparation of teachers to teach reading Learning to Read: The Great Debate (Chall, 1967) 24

25. National Council on Teacher Quality: Reading Components Taught Well Percentage 43% N=31 13% N=9 11% N=8 11% N=8 7% N=5 Source: Walsh, Glaser, and Wilcox, 2006 Number of Components 25

26. NCES Reading Report Card 2009: Categories < Basic: Less than partial mastery of prerequisite knowledge and skills fundamental to proficient work at the grade level Basic: Partial mastery of … Proficient: Solid academic performance and demonstrated competency over challenging subject matter Advanced: Superior performance Source: National Center for Education Statistics, 2009 26

27. Reading 2009 Grade 4 Percentage Source: National Center for Education Statistics, 2009, Table A-12 27 27

28. Preparation of Special Education Teachers in Scientifically Based Reading Instruction in 27 Institutions of Higher Education (IHEs) 28

29. Reading Course Syllabi: Projects • Explain your philosophy of literacy. • Develop bulletin board to motivate children to read. • Produce journal explaining personal experience in learning to read. • Analyze the social justice implications of literacy. 29

30. TQ Research & Policy Brief • Smartt, S. M., & Reschly, D. J. (2007). Barriers to the preparation of highly qualified teachers in reading (TQ Research & Policy Brief). Washington, DC: National Comprehensive Center for Teacher Quality. http://www.tqsource.org/publications/June2007Brief.pdf 30

31. Current Special Education Practice With High-Incidence Disabilities Review of special education case files for the state of Washington for a court case Randomly selected 900 special education students Ten districts were represented. All students were in special education for at least 12 months. Case files varied from 50 to 1,100 pages. The review evaluated individualized education programs (IEPs) using checklist for required components and evidence of formative evaluation. How many graphs? 31

32. Review of Special Education Case Files: Results Little evidence of systematic, direct instruction or behavior interventions using problem solving Assessment and formative evaluation nearly nonexistent (11 of 870 cases had graphs.) Lots of test protocols documenting weaknesses Little objective evidence of positive outcomes (i.e., benefits of special education are largely undocumented in high incidence.) No assessment of progress 32

33. National Mathematics Advisory Panel (2008) Report Key Findings • Conceptual understanding, computational and procedural fluency, and problem solving skills are equally important and mutually reinforce each other. • Students should develop immediate recall of arithmetic facts to free the “working memory” for solving more complex problems. • Teachers’ regular use of formative assessmentscan improve student learning in mathematics. 33

34. National Mathematics Advisory Panel (2008) Report Key Findings • Explicit instruction for students who struggle with mathematics is effective in increasing student learning. • Teachers should understand how to provide clear models for solving a problem typeusing an array of examples, offer opportunities for extensive practice, encourage students to “think aloud,” and give specific feedback. 34

35. Summary: Teacher Preparation and Practice Insufficient use of evidence-based practices in teacher preparation/comprehensive professional development and practice TQ Center use of evidence-based innovation configurations to address these issues (See the TQ Center Special Education Resource List.) Improved implementation of evidence-based principles leading to improved outcomes Major Challenge: Narrowing the gap between what is known about evidence-based instruction and teacher preparation and special education practice

36. References Chall, J. S. (1967). Learning to read: The great debate. New York: McGraw-Hill. Kavale, K. (2005). Effective intervention for students with SLD: The nature of special education. Learning Disabilities,13(4), 127–138. National Center for Education Statistics. (2009). The nation’s report card: Reading 2009─National Assessment of Educational Progress at Grades 4 and 8 (NCES 2010-458). Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved July 1, 2010, from http://nces.ed.gov/nationsreportcard/pdf/main2009/2010458.pdf National Mathematics Advisory Panel. (2008). The final report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education. Retrieved July 1, 2010, from http://www.ed.gov/about/bdscomm/list/mathpanel/report/final-report.pdf

37. References Reschly, D. J., Smartt, S. M., & Oliver, R. M. (2007). Innovation configurations to improve teacher preparation in reading, behavior management, and inclusive practices. In C. A. Dwyer (Ed.), America’s challenge: Effective teachers for at-risk schools and students (pp. 23–45).Washington, DC: National Comprehensive Center for Teacher Quality. Smartt, S. M., & Reschly, D. J. (2007). Barriers to the preparation of highly qualified teachers in reading (TQ Research & Policy Brief). Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved July 1, 2010, from http://www.tqsource.org/publications/June2007Brief.pdf Snow, C. E., Burns, M. S., Griffin, P. (Eds.). (1998). Preventing reading difficulties in young children. Washington, DC: National Academy Press. Walsh, K. Glaser, D., & Wilcox, D. D. (2006). What education schools aren’t teaching about reading and what elementary teachers aren’t learning. Washington, DC: National Council on Teacher Quality. Retrieved July 1, 2010, from http://www.nctq.org/p/docs/nctq_reading_study_app.pdf

38. Lynn Holdheide, Vanderbilt University P: 615-322-8150 E-Mail: lynn.holdheide@vanderbilt.edu Dan Reschly, Vanderbilt University P: 615-322-8169 E-Mail: dan.reschly@vanderbilt.edu