ASSESSING IMPACT Evaluating Staff Development Second Edition

1. ASSESSINGIMPACTEvaluating Staff DevelopmentSecond Edition JoellenKillion Corwin Press, 2008 A joint publication with the National Staff Development Council

2. As you get settled . . . • Write any questions you have about evaluating staff development on index cards. • One question per card, please. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

3. Intended outcome: think like evaluators and build staff development evaluation into the planning process for the TSSRRP grant. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

4. Describe the eight-step evaluation process and the components of each step. • Define evaluation in relationship to staff development. • Understand how a staff development program’s theory of change and logic model influence the success of a staff development program and its evaluation. • Construct an evaluation framework for a staff development program. • Acquire strategies, tools, and resources to assist in evaluating staff development. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

5. Norms • Share experiences to enrich others. • Ask questions. • Learn by doing. • Set aside any preconceived notions about evaluating staff development. • Apply to your own work. THINK S – A – L – S – A ! Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

6. Evaluation is: A systematic, purposeful process of studying, reviewing, and analyzing data gathered from multiple sources in order to make informed decisions about a program. IN THE BOOK Page 8 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

7. Key Elements IN THE BOOK Pages 8-9 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

8. Scientifically based research identifies interventions that have been “proven” to be successful under conditions that allow generalizations that these interventions will produce similar “positive” results in other situations. Evaluation determines whether the selected interventions (based on scientific research or not) have merit(achieved its intended results),worth(perceived as valuable), and/orimpactunder these conditions. Scientifically Based Research vs. Evaluation Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

9. Purposes of Evaluation • Determine merit and/or worth • Assess impact • Identify improvements • Provide accountability Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

10. Provide training Increase student achievement Means goals vs. End goals Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

11. Evaluation to Assess Impact • Requires the program’s goal to be about increasing student achievement. • Requires a theory of change to explain how the program will produce intended results. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

12. Program vs. Event • A program is a set of purposeful, planned actions and the support system necessary to achieve the identified goals. Effective staff development programs are ongoing, coherent, and linked to student achievement. IN THE BOOK Page 11-13 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

13. Staff Development vs. Staff Development Program Talk it over. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

14. Evaluating Staff Development • Planning • Assess Evaluability • Formulate Evaluation Questions • Construct Evaluation Framework • Conducting • 4. Collect Data • 5. Organize, Analyze, & Display Data • Interpret Data • Reporting • 7. Disseminate and Use Findings • Evaluate the Evaluation Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

15. Self-Assessment +understand — not sure ?don’t understand Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

16. Types of Evaluation • Planning • Formative • Summative IN THE BOOK Pages 13-17 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

17. ? Input Output Black Box Evaluations Glass Box Evaluations Results Actions IN THE BOOK Pages 26-28 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

18. Black Box ? Professional Development Action Student Achievement Results Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

19. Black Box Curriculum Development ? Student Achievement Professional Development Nonacademic factors Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

20. Glass Box Student Achievement Results Professional Development Actions Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

21. Glass Box Coaching/Follow-up Instructional Resources Implementation Monitoring Student Assessment Student Achievement Professional Development Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

22. Black Box vs. Glass Box What is the difference? Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

23. Assumptions • The staff development program is data-driven, research-based, and well-defined (p. 21). • The school, district, or regional agency has the capacity, including fiscal and human capital, to implement both the program and evaluation with fidelity to their designs (p. 22). • Key stakeholders in the school, district, or agency intend to use the evaluation results to make decisions about the program (p. 23). Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

24. Step 1:Assess Evaluability

25. Step 1: Assess EvaluabilityPlanning Phase The degree to which a proposed plan for staff development clearly articulates what the staff development program is and how it is expected to produce change. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

26. Read and Talk • Turn to p. 33. • Read last paragraph • Turn to p. 34 • Read first 2 paragraphs • Can you think of an example? • What was the issue? Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

27. Four Components of Assessing Evaluability • Examine program goals, objectives, standardsfor success • Examine program’s theory of change • Examine program’s logic model • Adjust program based on assessment Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

28. Clarifying Your ProgramWhat are the program’s goals? • Goal is desired state. • Improve physical well-being. • Objectives describe the changes expected. • Lose 15 pounds in six months. • Activities explain how to achieve the goals and objectives. • Walk daily. • Eat healthful foods. IN THE BOOK Pages 35-36 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

29. Goal, Objective, or Activity? G • Develop responsible citizens. • Increase student proficiency in applying the scientific process in a laboratory setting as measured by a performance task. • Provide opportunities for professional learning. • Establish professional learning communities. • Increase student performance 15% on state assessment in mathematical problem solving. G A A O Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

30. SMART Goals • Specific change • Measurable change • Attainable, realistic outcome • Results driven (student achievement) • Timebound —Garmston, JSD, Summer 1997, pp. 64-65 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

31. Sample SMART Goals • By 2010, student achievement in mathematics at all grade levels 3–8 will increase by 15% as measured on the state assessment. • By 2010, 90% of the 10th-grade students will demonstrate proficiency in writing on the state writing assessment. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

32. Results What results occur from professional learning for individuals, schools, communities … Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

33. Setting Standards for Acceptable Performance • Specifies how good is good enough • Specifies “success” in advance • Provides a benchmark/baseline for comparison before and after staff development IN THE BOOK Page 39 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

34. Sample Standards for Success • Increased student achievement • 8% overall increase on state assessment • 7% more students in the advanced category and 20% fewer students in the not-proficient category • 25% more underrepresented students in the proficient category Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

35. Types of Change • Knowledge • Attitude • Skill • Aspiration • Behavior IN THE BOOK Page 38 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

36. KASAB IN THE BOOK Page 38 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

37. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

38. Theory of Change • Identifies the components/actions of a program (what the program does) • Specifies the relationship among the components to explain how the change occurs (sequence of actions) • Delineates the underlying assumptions upon which the program is based IN THE BOOK Pages 40-46 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

39. Everyday Theory of Change Headache Headache gone; return to work; contribute to economic well-being of community Get pills Take pills Source: University of Wisconsin Extension Service Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

40. Theory of Change • Components/Actions of the program • Sequence of components to show how results will occur • Assumptions underlying the program Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

41. Student performance increases. Teachers implement new learning in their instruction. Teachers participate in collaborative learning experiences. IN THE BOOK Page 41 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

42. Activity • Form groups of two to three people. • In each group, make five cards. Asp K Att S B Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

43. Possible Theories of Change K + S B +Att+ Asp Asp +Att K B Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

44. Activity • Using the index cards you made, create two to three possible theories of change. You might begin with a familiar one. • For each theory of change you create, identify its underlying assumptions. • For each theory of change, also identify the potential implications for key decisions about professional development. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

45. Sample Theory of Change • Review the theory of change on page 42. Start at 12 o’clock. • Read the assumptions on which this program is based on the first paragraph on page 43. • State in your own words what a theory of change is. IN THE BOOK Page 42 Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

46. Write the goals, objectives, standards of success, and indicators of success for your case. • Create a theory of change for the staff development program you want to evaluate. When you are satisfied, map it on the large chart paper. Write your assumptions below the diagram. • Post your chart. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

47. Organization Central Office Principals Coaches Teachers Students Layering Theories of Change for Systemic Reform Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

48. What is the value of having a theory of change for a staff development program? Talk it over. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

49. Logic Model • Specific kind of action plan of the program specifying the flow of changes (results) that will occur • Inputs/resources • Activities/processes • Initial outcomes • Intermediate outcomes • Results IN THE BOOK Pages 46-50 Inputs Activities Intermediate Outcomes Initial Outcomes Results (Goals) Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

50. Homework Assignment • Read pages 46-51 • Create a logic model for at least 2 actions or components of your staff development program’s theory of change. (Worksheet #4) • Assess your progress using the Evaluability Assessment Rubric on page 51. Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council