Developing a Program and Music Teacher Evaluation Tool for Music Education - Creating Your Own Accountability Report

1. Developing a Program and Music Teacher Evaluation Tool for Music Education - Creating Your Own Accountability Report Douglas C. Orzolek University of St. Thomas St. Paul, Minnesota

5. Developing Evaluation Tools for Music Education Session Outline • External Factors • Internal Factors • Administrators/Supervisors Literature • On Developing an Evaluation Tool • The Teacher Performance Assessment • An Accountability Report • What Next?

6. Developing Evaluation Tools for Music Education External Factors • Attitude (Peck) * • Need for Composite Rating / “% Problem” (Burling) * • Inconsistency Policy from State to State, District to District • VAMs (Darling-Hammond, 2012) • “Transparency or Accuracy?” (Prince, Friedman) • Evaluator training, understanding of music, criteria don’t apply • $ to develop a Proper Assessment • Reliability and Validity • Legal Issues and Policy Validity (Prince)

8. Developing Evaluation Tools for Music Education Evaluation Components for Teachers in Non-tested Subjects or Grades – D.C. IMPACT Program (Steele, 2010) • 75% Administrator or Master Educator Observation • 10% Student Growth on a Teacher-Chosen Measure • 10% Commitment to the School Community • 5% School-wide Value-Added

9. Developing Evaluation Tools for Music Education Internal Factors • How Our Work is Percieved (Orzolek) * • Subjective/Affective (Maranzano, Barrett) * • Student Achievement? Connected to Learning? (Bowman) • One Size Does Not Fit All - Multifaceted Jobs – Create, Respond, Perform; Classroom, Ensemble, Individual (Grant, Taebel) • Criteria for Excellent Music Teaching? (Brophy, Taebel); “Successful” Teaching versus “Good” Teaching (Barrett) • More Research – Connect to Other Non-tested Areas? • The “My Program” Problem – Evaluate me on Program Success • “If we say that music kids are smart, then why are we afraid of VAMs? Wouldn’t that prove our point?” (Sean Barker, UST ‘12)

10. Developing Evaluation Tools for Music Education MSBA Survey: Accountability in Music(Orzolek, 2004) No Complaints “Good” Concerts Successful Trips No Letters to the Editor Trophies Athletic Bands Kids Seem Happy C&I Person is Happy A Local “Expert” Mostly at Budget Time “What else is there?

11. Developing Evaluation Tools for Music Education

12. Developing Evaluation Tools for Music Education Reminders from Music Leaders Literature • Clear Goals and Objectives – Cowden and Klotman (1991), 97 • “Evaluation ought to involve looking critically at the process of gathering and weighing evidence that will reveal changes in terms of a desirbable musical product.” – Weyland(1960), 295 • Very Subjective – Snyder (1959), 114 • Multifaceted – Dykema and Gehrkens (1941), 368 • “Seek competent evaluation of your work.” – Prescott (1938), 259

13. Developing Evaluation Tools for Music Education On Developing Evaluation Tools – External Burling (2012) (aka Pearson Corporation) • Define a construct – What is an effective educator? • Deploy multiple indicators – What evidence characterizes good teaching and school leaderhip? • Develop a clear composite rating – What weights should each indicator have and who should be involved in the decision? • Clarify differentiated performance levels – What distinguishes varying levels of educator effectiveness? • Build strong data analysis and reporting tools – What does the information reveal about student, educator, and school performance? • Improve Instructional Leadership and Practice – How can the information target professional development to boost educator practice, student learning outcomes, and school efficacy?

14. Developing Evaluation Tools for Music Education On Developing Evaluation Tools – External Others • Promote consistency in any student measures used to assess teachers (Steele) • Implement tools after experimentation (Steele) • Should supply teachers with feedback – “close the loop” (Prince) • Teachers should determine the extent to which math/reading scores are included (Prince) • Portfolio (Prince) • Always use tools that are valid and reliable – Charlotte Danielson. (Peck)

15. Developing Evaluation Tools for Music Education On Developing Evaluation Tools - Internal Brophy (1993) • Personal characteristics, as evidenced through student-teacher, teacher-faculty, and teacher administration interactions; • Music competence and performance skill, as evidenced when teaching a lesson; • Effective use of nonverbal strategies, such as modeling/demonstrating; • Effective use of verbal strategies, such as eliciting performance from students;

16. Developing Evaluation Tools for Music Education On Developing Evaluation Tools - Internal Brophy(1993) • Classroom management, including discipline, group participation, and the creation of a positive learning environment; • Effective planning for concept learning and aesthetic appreciation across a wide age span; • An objective assessment of teaching style based on empirically supported criteria; • Appropriate professional development activities should be recommended.

17. Developing Evaluation Tools for Music Education Three Categories of Teacher Evaluation NAfME (2011) • Teacher Evaluation linked exclusively to student outcomes. • Teacher evaluations linked exclusively to teacher practice. • Teacher evaluations that combine measures of student outcomes and teacher practice.

18. Developing Evaluation Tools for Music Education Question How do we help prepare teachers for these evaluations?

19. Developing Evaluation Tools for Music Education TPA Background • Teacher Performance Assessment – Stanford, AACTE, CCSSO, Pearson, MACTE. 21 states actively involved. • Assesses “Effectiveness” of Pre-service Educators • The entire assessment is outlined in a “TPA Handbook” – “The Performing Arts”. • Completed during Student Teaching • Scored by Trained Scorers via E-folio

20. Developing Evaluation Tools for Music Education How does the assessment work? “An Authentic Assessment of Teaching Competence” • Students complete 4 Tasks: • Describe their Plans for Instruction & Assessment; • Engage Students in Learning; • Assess Student Learning; • Analyze Teaching. • Parameters • “Central Focus of Learning Segment” • 3-5 Lessons • Evidence provided on Video • Use of “Academic Language” • Detailed Rubrics Provided

21. Developing Evaluation Tools for Music Education Starting Point: A Voluntary Accountability Report

22. Goals/Results for 2003-2004 Increase number of national standards completed / Implemented composition unit this year Improve student rankings in state portfolio assessments / See below Enhance music theory understanding by adapting new software / New software purchased and implemented Provide outstanding performance opportunities for students / Performances were well received - CD attached! National Standards This year we added a composition (Standard 4) unit to our instruction. Several guests spoke of their experiences as composers, we took a field trip to the American Composer’s Forum and wrote our own pieces that were performed during our recital at the end of the year. State/District Standards 11th grade portfolios were sent to our neighboring schools for evaluation and assessment by music educators in those districts. The following are the results based upon established rubrics: 4 (Exemplary) - 15% (10% in 2002) 3 (Proficient) - 45% (40% in 2002) 2 (Developing) - 35% (45% in 2002) 1 (Poor/Incomplete) - 5% (5% in 2002) The results suggest that our students are performing well in these assessments of their work. This year’s scores show an improvement from the previous year. Comments from Professional Reviews “Strongest performing group I have heard in years.” - Leonard Bernstein Comments from Student/Peer Evaluations “I feel like I have a better understanding of my instrument after I’ve studied a composer and her intentions.” “”Billy’s perceptions of the performance were much different than mine. But, I appreciated his points - they reminded me that music can be something different for everyone.” “Opera - I still don’t like it.” Testing Results for 2003-2004 9th Graders completed the NAEP music skills written portion and scored in the 72 percentile nationally. Music reading skills measured by the Watkins-Farnum performance scale continue to improve. The mean etude completed by students in grades 9-12 was #6 in 2002 and rose to #8 in 2003. Average SAT score of a student in band was 1425 (of 1600.) Other Highlights 97% of our students participated in the MN solo/ensemble festival this year. 35% received Superior and 40% received Excellent ratings. Our Concert Band earned the top ranking of Superior at the large group festival. Our Marching Band was selected to perform in the Rose Parade. We commissioned a new work from composer Libby Larsen. 6 band members were selected to perform with the All-State band. Our students gave 60 public performances during this past year. 80% of senior band members are planning to attend a 4 year college. 2002 - 2003 Accountability ReportInstrumental Music Education Department - George Washington High SchoolContact: Douglas C. Orzolek (dcorzolek@stthomas.edu)Mission Statement : High Quality, Comprehensive Music Education

23. Developing Evaluation Tools for Music Education What Next? • Top of NAfME Agenda – Policy Analysis, Leadership • Stay Informed – Websites, Readings • Share Stories – Forum, Meetings • Coordinate and Advance Research Efforts - Grants • The Reality - Multifaceted and Messy – Experiment. • This is about People not Composite Ratings. • Until we agree on what constitutes effectiveness, it will be difficult to measure and reward it.

24. Developing Evaluation Tools for Music Education Darling-Hammond & Bransford (2005) Effective Teachers: • Understand subject matter deeply and flexibly; • Connect what is to be learned to students’ prior knowledge and experience; • Create effective scaffolds and supports for learning’ • Use instructional strategies that help students draw connections, apply what they’re learning, practice new skills, and monitor their own learning; • Assess student learning continuously and adapt teaching to student needs; • Provide clear standards, constant feedback, and opportunities for revising; • Develop and effectively manage a collaborative classroom in which all students have membership.

26. Developing Evaluation Tools for Music Education “When a call for accountability elicits discussions about education, the results produced are usually positive.” R. Colwell, (2003) “The Status of Arts Assessment: Examples from Music”, Arts Education Policy Review, 105 (2), 16.