Evaluating the effectiveness OF PROFESSIONAL LEARNING

1. Access Handouts at: http://tinyurl.com/catxtsh EvaluatingtheeffectivenessOFPROFESSIONAL LEARNING Jacqueline Kennedy Associate Director of Strategic Initiatives Learning Forward Raleigh, NC December 12-13, 2012

2. Our Agenda

3. Our Outcomes • Define evaluation in relationship to professional learning; • Examine the process of teacher change and its impact on learning; and • Acquire strategies, tools, and resources to assist in evaluating professional learning.

4. Our Essential Question • How can evaluating professional learning leverage school, school system, and state improvement effort? • How will I align professional learning objectives to measurable short, medium and long-term results for educators and students? • How will I collaborate with others to construct a framework that outlines a detailed plan for evaluation? • How do I incorporate evaluation into my work and normative practice?

5. Our Norms • Share experiences to enrich others. • Ask questions. • Learn by doing. • Set aside any preconceived notions about evaluating professional learning. • Apply to your own work. THINK S – A – L – S – A !

6. Where are we now? • Independently, read the case study. • At your table discuss components of your professional learning evaluation. • Chart your process, considering actions, steps, and information needed to address questions posed. • Identify questions that emerge about evaluating professional learning. • Be prepared to discuss your work.

7. Burning Questions • Select one question shared at your table that might have implications for the whole group. • Be prepared to share it. • Post all other questions (one per sticky note) on the chart.

8. The Standard Professional learning that increases educator effectiveness and results for all students uses a variety of sources and types of student, educator, and system data to plan, assess, and evaluate professional learning. -Standards for Professional Learning, 2011

9. Evaluation is: A systematic, purposeful process of studying, reviewing, and analyzing data gathered from multiple sources in order to make informed decisions about a program. – Killion, 2008 The systematic investigation of merit and worth. – Thomas Guskey, 2000

10. Link to Student Results

11. 4 Key Components

12. Which best describes your experience?

13. Shifting Perspectives

14. Group Think • How do identified shifts influence you and your work in North Carolina? • What aspect of evaluating professional learning do you find essential?

15. 4 Purposes • Determine merit and/or worth – determine overall value of program • Assess impact – determine overall effectiveness of a program • Identify improvements– determine program strengths and weaknesses • Provide accountability – determine evidence of complianceand resources

16. Worth The extent to which a program(s) or activity(ies) is/are essential to a school’s, district’s, agency’s, or individual’s mission. Worth is an indication of the program’s or activity’s perceived value to constituents or to a single individual. • Do participants value this program?

17. Merit The value of the program is judged by comparing its performance against established standards of excellence in the profession. • Did the program achieve its goals and meet its standard(s) of success?

18. Means goals vs. End goals

19. Programs Demonstrating Merit May or may not assess the impact of the program on student achievement results. • Will — if the goal is to increase student achievement (results) • Will not — if the goal is to provide service

20. Evaluation to Assess Impact • Requires program goals to be about increasing student achievement results. • Requires a theory of change to explain how the program will produce intended results.

21. Program vs. Event Evaluation A program is a set of purposeful, planned actions and the support system necessary to achieve the identified goals. Effective professional learning programs are ongoing, coherent, and linked to student achievement.

22. Example • A workshop on Common Core standards. • A workshop on Common Core standards coupled with planned planned classroom-support, ongoing coaching and feedback, demonstrations, and observations in other classrooms incorporated into a comprehensive system, ongoing plan designed to develop educator knowledge, attitudes, skills, aspirations, and behaviors.

23. 3 Types of Evaluation • Planning – before program design to provide information on conditions or needs to address • Formative – during implementation to provide information on whether the program is working as designed • Summative – after completion to provide information on outcomes or overall impact

24. Note to Self • How are key components, purposes, and types essential to understanding evaluation? • What about evaluating professional learning interests your most?

25. ? Input Output Black Box Evaluations A simplistic approach to professional learning evaluation that fail to amplify the underlying theory and operation of the professional learning program. Glass Box Evaluations Results Actions A comprehensive approach to professional learning evaluation that illuminates how professional learning program components interact to produce results.

26. Black Box Focus on outputs rather than what occurs in the program or what is presumed to be causing those outcomes and why. ? Professional Learning Action Student Achievement Results

27. Black Box Focus on inputs and fail to shed light on HOW a program’s activities and resources interact to produce results. Curriculum Development ? Student Achievement Results Professional Learning Nonacademic factors

28. Glass Box Focus on what occurs and how it occurs within the program. Student Achievement Results Professional Learning Actions

29. Glass Box Focus on illuminating factors contributing to transformation process. Coaching/Follow-up Instructional Resources Implementation Monitoring Student Assessment Student Achievement Results Professional Learning

30. Black Box vs. Glass Box What is the difference?

31. BREAK See you back in 15 minutes.

32. Your Evaluation Process Write 4-5 sentences that describe the evaluation process/steps you currently use as a leader member responsible for assisting others with evaluating professional learning. • Be specific. • Be frank. • Write in complete sentences.

33. 8 Smooth Evaluation Steps • Reporting • 7. Disseminate and Use Findings • Evaluate the Evaluation • Planning • Assess Evaluability • Formulate Evaluation Questions • Construct Evaluation Framework • Conducting • 4. Collect Data • 5. Organize, Analyze, & Display Data • Interpret Data

34. Deconstructing the Steps • Distribute evaluation step cards at your table. • Independently, read about your assigned step. • Regroup with others with the same step. • Describe how the step contributes to evaluation. • Identify concepts embedded in the step. • Recommend strategies for addressing this step to increase the effectiveness of professional learning evaluations.

35. Group Discussion Protocol • Return to your original table group. • In round robin fashion, beginning with step one, share key points, evidence, and strategies. • Consider how steps are independently and collectively important. • Be prepared to discuss insights gained with the whole group.

36. Note to Self • Revisit your process. • Based on what you now know about the eight evaluation steps, which steps were present? Missing? • What is one actions you will take to address missing steps in your current process?

37. Summer Institute Academy Vision Maria Pitre Martin Lynne Johnson

38. LUNCH See you back in 45 minutes.

39. Assumptions about Change • Independently, read the article by Michael Fullan. • With a partner, respond to one or more of the questions below. • Jot down notes, and identify common themes that emerged. • What aligned with your thinking? Share why. • What challenged your thinking? Share why. • Describe something you want to explore. Be specific. • What was the most important takeaway for you?

40. Change is Learning Concerns-Based Adoption Model : Developed by Bill Rutherford, Gene Hall, Shirley Hord, and Susan Loucks-Horsley 4 Components: • Stage of Concern – 7 stages of responses • Levels of Use – eight ranges of intervention use • Innovation Configuration – described actions • Changefacilitators – leaders of learning

41. Change Learning Exchange • Distribute numbered cards (#1-4) at your table. • Read and the corresponding article on change. • Prepare a two-minute talk about your article. Use any of the following to prepare. • Why your focus area is important. • Implications of the change process. • Ways to facilitate your area of change. • Challenges you anticipate when helping others understand this area of change.

42. Change Spotlight • Find a partner that read a different article. • Take two minutes each, discuss your article. Focus on any of the following: • Why your focus area is important. • Implications of the change process. • Ways to facilitate your area of change. • Challenges you anticipate when helping others understand this area of change. • Listen for the timer to repeat the process.

43. Note to Self • What was one benefit of revisiting the change process? • What new insights did you gain as a result of your reading and discussion with others?

44. Change Self-Assessment • Use the self-assessment to determine the level of implementation in your system. • Note the evidence that informs your score. • Identify your area of strength and area for improvement. • Be prepared to meet with others about your results, and evidence.

45. Strengths • Determine which area is your strongest. • Meet with colleagues who share that same area as a strength. • Share the evidence of your strength, i.e., what is it you are doing in relationship to this area? • Record strategies you receive.

46. Areas for Improvement • Determine which area is your greatest area for improvement. • Meet with colleagues who share that same area for improvement. • Identify what happens when this area is a gap. • Use strengths collected from others to identify possible strategies you might consider to address this gap.

47. Group Reflection • What does this mean for your work? • What do you need to share and with whom? • How will you continue to develop strengths? • What is one strategy you will take away to develop an area for improvement?

48. BREAK See you back in 15 minutes.

49. Evaluation of ill-conceived professional learning programs waste limited human and fiscal resources. In these cases, evaluations sanction programs with gaps in their conceptual frameworks by appearing to condone mediocrity. Wasted evaluations rob students and educators of powerful opportunities to learn at high levels. And when evaluations attempt to assess programs with fuzzy goals, faulty conceptual frameworks, or illogical, poorly structured activities, the evaluation may be difficult to design and of questionable integrity. -Killion, J. (2008). Assessing Impact. p. 33.

50. Thorough planning facilitates sound evaluation.