Resources for State Oral Health Program Evaluators January 30, 2013

1. Resources for State Oral Health Program Evaluators January 30, 2013 • B. J. Tatro, PhD • ASTDD, bj@bjtatro.com Cassandra Martin Frazier, MPH, CHES CDC, bkx9@cdc.gov

2. General Reminders • This webinar will be recorded and archived on the ASTDD website • Questions will be addressed after the speakers are finished. Please type your question into the “chatbox” that will appear at the end of the webinar and then click on the bubble to the right of where you type your question to send it to the moderator • Please respond to the polling questions at the conclusion of the webinar

3. This presentation was supported by Cooperative Agreement 5U58DP001695 from CDC, Division of Oral Health. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC

4. Desired Results • Increase awareness of the importance of evaluation and evaluation capacity within state oral health programs • Increase familiarity with the scope of public oral health evaluation topics • Increase familiarity with and access to a variety of practical and useful evaluation resources relevant to public oral health

5. PART 1 Importance of Evaluation and Evaluation Capacity in Public Oral Health

6. Evaluation: A Key Component of State Oral Health Program Infrastructure

7. Status of Evaluation in State Oral Health Programs • In 2012, ASTDD completed a report on the status and factors impacting state oral health infrastructure. The capacity to conduct evaluation was one of these. Findings included the following: • According to CDC, the number of funded states reporting using evaluation to improve programs had increased from 3 to 11 of the 12 by the end of their second state grant funding cycle • In a 2009 ASTDD survey, most states indicated that they had access to evaluation expertise, but 13 did not • Currently there is no way to assess and track if and how non-CDC funded states are evaluating their programs, partnerships, or policies

8. Key Factors Associated with Program Success/Improvements in Oral Health “Evaluation helps build infrastructure and enhance sustainability when results are used to improve programs, increase program visibility and demonstrate program achievements. Strong programs have evidence-based goals, conduct routine evaluation and alter their programs based on evaluation results.” (ASTDD, State Oral Health Infrastructure and Capacity: Reflecting on Progress and Charting the Future, 2012)

9. Evaluation Capacity Evaluation helps build infrastructure but the capacity to do evaluation must be present Evaluation requires: • Sufficient and appropriate staffing • Ongoing training • Access to data • Access to decision makers • Financial resources

10. Importance of Evaluation in Public Oral Health In the ASTDD Guidelines for State and Territorial Oral Health Programs, there are 10 Essential Public Health Services to Promote Oral Health in the US, one of which is: Evaluate effectiveness, accessibility, and quality of personal and population-based oral health promotion activities and oral health services

11. State Roles in Public Oral Health Evaluation • Create a comprehensive evaluation plan • Evaluate availability, utilization, appropriateness, and effectiveness of activities and services, with partners • Provide evaluation consultation to partners • Validate and share results with community members, stakeholders, and other partners • Use results in oral health improvement and planning • Leverage and allocate sufficient resources for evaluation

12. Essential Public Health Services to Promote Oral Health in the U.S.

13. Evaluation of Other Essential Public Oral Health Services Virtually all of the other nine Essential Services have an evaluation component!

14. Evaluation of Other Essential Public Oral Health Services

15. Funder Expectations for Evaluation • Federal Government:“A central pillar of good government is a culture where answering such [outcome and impact] questions is a fundamental part of program design and where agencies have the capacity to use evidence to invest more in what works and less in what does not.” (Program Evaluation, in Budget of the United States Government, Fiscal Year 2011: Analytical Perspectives, 2010, 90–92. Washington, DC: U.S. Government Printing Office) • Foundations: “The field of philanthropy is undergoing a fundamental transition toward more performance-centered and forward-looking evaluation approaches that provide foundations and their grantees with timely information and actionable insights.” (Mark Kramer, From Insight to Action: New Directions in Foundation Evaluation, April 2007)

16. CDC Structure for Evaluation • Describe the program • Develop logic models • Develop a comprehensive evaluation plan and objectives • Document what happens in the program • Maintain accurate and organized records on programs • Designate a person responsible for evaluation • Create success stories • Improve the program • Use evaluation findings to plan program improvements • CDC’s Framework for Program Evaluation in Public Health

17. CDC Structure for Evaluation • Incorporate evaluation into the following required activities for CDC-funded states: • Leadership • The surveillance system • The state oral health plan • Partnerships and state oral health coalitions • Sealant programs • Policy development

18. Competencies for Public Oral Health Evaluators CDC has identified competencies for State oral health evaluators

19. Competencies for Public Oral Health Evaluators Competencies based on: • Guiding Principles for Evaluators, American Evaluation Association • CDC, Framework for Program Evaluation • Stevahn L, King J, Ghere G, Minnema J. Establishing Essential Competencies for Program Evaluators.American Journal of Evaluation,2005; 26 (1) (43-59) • Yarbrough, D, Shulha, L, Hopson, R, and Caruthers, F (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users (3rd ed.). Thousand Oaks, CA: Sage • ASTDD Competencies for State Oral Health Programs

20. PART 2 Resources for Public Oral Health Evaluators

21. Evaluation Planning and Management

22. Evaluation Planning and Management

23. Evaluation Planning and Management

24. Evaluation of Prevention and Promotion Programs

25. Evaluation of Prevention and Promotion Programs

26. Evaluation of Prevention and Promotion Programs

27. Evaluation of Policy Work

28. State Plan Evaluation

29. Evaluation of Surveillance

30. Evaluation of Leadership and Leadership Development

31. Evaluation of Partnerships and Collaboration

32. Where to Find New Resources • ASTDD website (slides from this webinar will also be posted on the ASTDD website) • CDC evaluation website • Join the American Evaluation Association (AEA) or a local affiliate • AEA Linked-In • University of Wisconsin Extension Centerevaluation website • Western Michigan University, the Evaluation Center • Online Evaluation Resource Library – National Science Foundation website • Bureau of Justice Assistance evaluation website • Conferences, including the AEA Annual Conference, National Oral Health Conference and preconference workshops, and CDC/AEA Summer Institute • Online and live trainings, including AEA, CDC, and colleges/universities

33. Next Steps • Are you interested in: • Future evaluation webinars? • NOHC evaluation roundtables? • NOHC evaluation presentations? • NOHC evaluation pre-sessions? • A evaluation community of practice? • An evaluation blog? • Are you willing to post evaluation articles, data sources, tools, or other resources to a shared site, if it were available?