Program Evaluation Webinar Series Part 2:

1. Program Evaluation Webinar Series Part 2: “Getting Started and Engaging Your Stakeholders” Presented by: Leslie Fierro and CarlynOrians

2. Getting Started and Engaging Your Stakeholders • Leslie A. Fierro, MPH • TKCIS Contractor NCEH/CDC • Let6@cdc.gov CarlynOrians, MA, Battelle Centers for Public Health Research and Evaluation orians@battelle.org Presented November 20, 2008

3. Agenda • Definition of evaluation. • Difference between evaluation & research. • CDC’s Evaluation Framework. • The who, why, when & how of stakeholders. • Tangible examples of engaging stakeholders.

4. What is the Purpose of Evaluation?

5. Evaluation: Definition 1 Evaluation is the systematic investigation of the merit, worth, or significance of an object. -- Michael Scriven

6. Evaluation: Definition 2 Evaluation is the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program or policy. -- Carol Weiss

7. Evaluation: Definition 3 Evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. -- Michael Patton

8. The Common Element Note that all 3 of these statements share a common element. .. …they all define evaluation as a systematic and formalized endeavor.

9. Summative Evaluations • Summative evaluations seek to judge a program by asking, “Should this project be…” • cancelled? • continued? • expanded?

10. Formative Evaluations • Formative evaluations seek to use the evaluation findings. • They ask the question: • “Is the program being conducted as planned?” • “Is the program doing well?” Formative evaluations seek to improve programs or policies.

11. The CDC Definition • Evaluation is the systematic collection of information about the activities, characteristics and outcomes of the program to make judgments about the program, improve program effectiveness and/or inform decisions about future program development.

12. Research vs. Evaluation Research and evaluation share methodologies but ask different questions. Specific to Research Specific to Evaluation Commonalities

13. The Research Context

14. Research Findings Research is conducted to: test and improve theories. develop generalizable theories. theories apply across different settings, people, and times.

15. The Evaluation Context

16. Evaluation Findings Evaluation results: are not usually generalizable. focus on a specific situation. evaluation of a single program in a particular context.

17. Research vs. Evaluation • Evaluation asks: • “Is this program working?” • Research asks: • “Will this program work across multiple settings?”

18. The Role of Researchers Researchers: Play a single role as content experts.

19. The Role of Researchers Researchers: Play a single role as content experts. Identify gaps in current knowledge.

20. The Role of Researchers Researchers: Play a single role as content experts. Identify gaps in current knowledge. Derive their own questions.

21. The Role of Researchers Researchers: Play a single role as content experts. Identify gaps in current knowledge. Derive their own questions. Perform their own research.

22. The Role of Evaluators Evaluators: Play multiple roles—facilitator, educator, scientific expert, etc.

23. The Role of Evaluators Evaluators: Play multiple roles—facilitator, educator, scientific expert, etc. Involve stakeholders.

24. The Role of Evaluators Evaluators: Play multiple roles—facilitator, educator, scientific expert, etc. Involve stakeholders. Collaborate to identify and prioritize questions.

25. “Research seeks to prove, evaluation seeks to improve.” M.Q. Patton In a Nutshell…

26. The Findings Must be Useful • To improve a program, the findings must be useful! • How? • The stakeholders must be involved. • The questions must be relevant to the program. • The findings must be credible to key stakeholders.

27. Getting Started and Engaging Your Stakeholders • The Who, When, Why, and How • of Stakeholder Involvement • in CDC’s Evaluation Framework

28. “Who” are the Stakeholders? • Stakeholders are: • people and/or organizations that are • interested in the program, are • interested in the results of the evaluation and/or • have a stake in what will be done with the results of the evaluation.

29. Examples of Stakeholders

30. Each Program is Different • Develop a list of stakeholders at the start of any evaluation activity.

31. Which Stakeholders Matter Most? • Review your list of stakeholders and think strategically about these questions: • “Who do we need to…”

32. Which Stakeholders Matter Most? • Review your list of stakeholders and think strategically aboutthese questions: • “Who do we need to…” • enhance credibility?

33. Which Stakeholders Matter Most? • Review your list of stakeholders and think strategically about these questions: • “Who do we need to…” • enhance credibility? • implement program changes?

34. Which Stakeholders Matter Most? • Review your list of stakeholders and think strategically about these questions: • “Who do we need to…” • enhance credibility? • implement program changes? • advocate for changes?

35. Which Stakeholders Matter Most? • Review your list of stakeholders and think strategically about these questions: • “Who do we need to…” • enhance credibility? • implement program changes? • advocate for changes? • fund, authorize, or expand the program?

36. “Why” Engage Stakeholders? • Engaging stakeholders is an important part of the CDC Evaluation Framework. • Involving stakeholders may be a requirement of your program.

37. “Why” Engage Stakeholders? • Engaging stakeholders is an important part of the CDC Evaluation Framework • Stakeholders will add credibility. • If you want the results to be acted upon, they must be credible.

38. “Why” Engage Stakeholders? • Engaging stakeholders is an important part of the CDC Evaluation Framework. • Stakeholders will add credibility. • Stakeholders may have resources to help. • Stakeholders may be able to contribute data, skills, analytical skills, etc.

39. “Why” Engage Stakeholders? • Engaging stakeholders is an important part of the CDC Evaluation Framework. • Stakeholders will add credibility. • Stakeholders may have resources to help. • Stakeholders may be critical to implementing or advocating for action based on the results.

40. “Why” Engage Stakeholders? • Engaging stakeholders is an important part of the CDC Evaluation Framework. • Stakeholders will add credibility. • Stakeholders may have resources to help. • Stakeholders may be critical to implementing or advocating for action based on the results. • You will build trust and understanding among program constituents. • Involving stakeholders helps to reduce fear of the evaluation process.

41. “Why” Would Stakeholders Want to be Involved? • Get answers to their questions.

42. “Why” Would Stakeholders Want to be Involved? • Get answers to their questions. • Learn about evaluation.

43. “Why” Would Stakeholders Want to be Involved? • Get answers to their questions. • Learn about evaluation. • Influence the design and methods.

44. “Why” Would Stakeholders Want to be Involved? • Get answers to their questions. • Learn about evaluation. • Influence the design and methods. • Protect their constituents.

45. “Why” Would Stakeholders Want to be Involved? • Get answers to their questions. • Learn about evaluation. • Influence the design and methods. • Protect their constituents. • Motivated to help program succeed.

46. CDC’s Evaluation Framework STEPS Engage stakeholders Ensure use and share lessons learned Describe the program Standards Utility Feasibility Propriety Accuracy Focus the evaluation design Justify conclusions Gather credible evidence

47. “When” and “How” Can You Engage Stakeholders? • You can use the CDC Evaluation Framework to engage stakeholders in: • identifying and prioritizing evaluation questions, • selecting credible sources, and • developing a plan for action based on evaluation results.

48. “When” and “How” Can You Engage Stakeholders? • You can engage stakeholders in every step of the evaluation process. • But… you don’t have to engage stakeholders • in all these ways in every evaluation.

49. “How” Depends on… • Evaluator preference.

50. “How” Depends on… • Evaluator preference. • Stakeholder preference.