Partnerships for Evaluating Policy Change

1. Partnerships for Evaluating Policy Change Panel Discussion June 13,2006

2. Why Policy and Environmental Change • Essential Part of Behavior Change

3. Social-Ecological Model Policy, Systems, Environment Community Institutional/Organizational Interpersonal Individual

4. Why Policy and Environmental Change • Essential Part of Behavior Change • Bigger Bang for the Buck

5. Multilevel Pyramid Model of Stepped Care Interventions Reach Intensity / Cost Highest Lowest Specialty Primary Care Health Systems Community and Neighborhood Media Policies Highest Lowest

6. Why Policy and Environmental Change • Essential Part of Behavior Change • Bigger Bang for the Buck • Policy and Environmental Changes Stay

7. Panelists • Amy Roussel, PhD - RTI • Vic Colman, JD – Washington State Department of Health • Shelly Curtis, MPH, RD – The Children’s Alliance • Donna Johnson, PhD, RD – University of Washington • Marilyn Sitaker, MPH - Washington State Department of Health

8. Evaluation and Policy Change Presented byAmy Roussel, PhD RTI International Presented atAnnual Meeting of the Association of State & Territorial Public Health Nutrition Directors Seattle, WA June 13, 2006 RTI International ■ 3040 Cornwallis Road ■ Research Triangle Park, NC 27709Phone: 919-990-8324 ■ Fax: 919-485-5589 ■ roussel@rti.org ■ www.rti.org RTI International is a trade name of Research Triangle Institute

9. Session Overview • Define our terms • Demystify evaluation • Discuss types of evaluation • Introduce CDC Framework for Evaluation

10. What Do We Mean by Evaluation? The systematic investigation of the merit, worth, or significance of program activities or outcomes • Evaluation also explores how and why those activities are occurring • Evaluation builds on the program monitoring activities that your programs currently conduct

11. What Do We Mean by Policy? A principle, plan, or course of action, as pursued by a government, organization, group, etc. • General term: includes policy, legislation, environmental change, systems change • Occurs at all levels of the socioecologic model

12. The Socioecologic Model

13. Some Policy Change Examples • Imposing a 1% sales tax on items listed as “sweet” or “snack” in the USDA National Nutrient Database for Standard Reference • A community building a permanent structure to house a farmers’ market • School systems eliminating deep-fat fryers from all cafeteria kitchens • Worksites subsidizing healthy food choices • Families instituting sit-down dinner policies (& no TV!)

14. What Have You Done?

15. More on Evaluation Evaluation helps you • build capacity • manage programs effectively • assess and improve existing efforts • understand reasons for performance • plan and implement new policies or programs • demonstrate the value of your efforts • ensure accountability

16. Evaluation Types and Questions • Formative evaluation: To develop or improve a policy What kind of policy changes will people respond to? • Process evaluation: To assess process or implementation How [or how well] was the policy implemented? • Outcomes evaluation: To assess outcomes or impact What difference did the policy make?

17. Start Where You Are • Consider developmental trajectory of policy change • Development – formative evaluation • Implementation – process evaluation • Institutionalization – outcomes evaluation

18. Consider the Evidence Base • Evidence-based practice • Policy changes known to be effective • Practice-based evidence • Your evaluation will help build the evidence base • Plan your evaluation accordingly

19. Specifying Goals and Questions • What do you want to know? • What do stakeholders want to know? • What do you intend to do with what you learn?

20. CDC Framework for Evaluation

21. 6-Step Model • Engage stakeholders • Describe the program • Focus the design • Gather credible evidence • Justify conclusions • Ensure use and share lessons learned

22. Step 1: Engage Stakeholders Stakeholders include: • Those interested in the program or policy • Those affected by the policy • Those who are primary users of evaluation

23. What Stakeholders Can Do Key stakeholder evaluation functions include: • Describing the policy and intended impacts • Selecting evaluation questions and methods • Serving as data sources • Interpreting findings • Disseminating information • Implementing results

24. Step 2: Describe the Program • Summarize the policy change being evaluated • Establish common definitions and terms • Delineate intended outcomes and other factors that may help or hinder • Describe how the policy fits into the larger picture

25. Program Descriptions • Often involve logic models – but not necessarily! • Ask yourself: What are we trying to accomplish? • And then what happens? “If you don’t know where you’re going, you’ll end up somewhere else.” -Yogi Berra

26. Step 3: Focus the Evaluation • Consider the purpose • What questions are you and stakeholders asking? • How will evaluation information be used? • Reality checks • What’s the developmental stage? • What’s the evidence base? • What resources are available for evaluation? • Evaluation questions + policy stage  design

27. Step 4: Gather Credible Evidence • Qualitative or quantitative? • Primary (new) or secondary (existing) data? • Clear protocols help to ensure rigor Use multiple data sources whenever possible

28. Step 5: Justify Conclusions • Structured analysis - reproducible • Assess findings against standards • What would constitute success? • Seek stakeholders’ inputs on this early • Interpret the data – what does this tell you?

29. Step 6: Ensure Use & Share Lessons Learned • Present findings • Discuss findings • Develop action plan • Act on findings Evaluation without action? What’s the point?

30. Sharing Lessons Learned • Choose your medium – or media • Match method to audience • Allow room for stories, accounts, testimonies • The report is not the end of the evaluation • Solicit feedback from partners, staff, stakeholders • Open dialogue is essential • Share results and brainstorm next steps • Focus on improvement • Let the past inform the future

31. Acting on Findings • Evaluation ought to be useful and actionable • Participatory planning facilitates action • What needs to be done? • Who will do it? • Who will help? • Turn the plan into action • Keep goal of improvement in mind

32. Review • Policy – an inclusive term • Evaluation – not so scary any more? • The CDC Framework – a handy tool • Putting it Together – see handout • Key items learned • Action items • SMART objectives

33. Any Questions?

34. Victor Colman, JD Senior Policy Advisor Washington State Department of Health Making Policy Choices: Importance of Evaluation June 2006

35. Learning Objectives • Learn about a state’s approach to improving nutrition and physical activity via the entire spectrum of prevention • Highlight challenges in making policy choices – with a particular focus on the role of evaluation

36. Nutrition and Physical Activity: The Washington State Story Rare Opportunity Now in Place to Create a “Healthier Washington” • part of Governor’s Health Care agenda • convergence of public, media, public health & policymaker interest in chronic disease prevention, most notably nutrition and physical activity

37. ENVIRONMENT AGENT HOST PUBLIC HEALTH MODEL

38. The Spectrum of Prevention

39. Our Vision:

40. Nutrition and Physical Activity: The Washington State Story Coalitions and Partnerships • Washington’s Action For Healthy Kids Team • Access to Healthy Foods Coalition • Washington Coalition for Promoting Physical Activity • Transportation Choices Coalition • Bicycle Alliance of Washington • Breastfeeding Coalition of Washington • Anti-Hunger Nutrition Coalition • WA Association of Local WIC Agencies

41. Nutrition and Physical Activity: The Washington State Story Importance of Evidence-Based Policy Research • In early 2005, DOH released a “Policy Resource Guide” as a tool for local, state and private interests

43. Nutrition and Physical Activity: The Washington State Story Importance of Collective Policy Development • Emerging Role of the Nutrition and Physical Activity Policy Leadership Group • Ongoing development of a broad policy platform

44. Policy Development: Conceptual Framework

45. Nutrition and Physical Activity: The Washington State Story Importance of Policy Implementation – Recent Bills • SB 5436 – school policy • SB 5186 – physical activity and planning

46. Nutrition and Physical Activity: The Washington State Story Lessons Learned • Too much focus on change at state capital level • Policy change is, on balance, easier to enact at local and institutional levels • Determining adequate enforcement capacity can be a critical factor in successful policy implementation

47. Nutrition and Physical Activity: The Washington State Story Challenges • Heightened accountability with government funds • Prevention still held to a higher standard that other kinds of policy choices • Policymakers have short attention spans – prevention-based solutions take time • Government employees can often be confused about their role in policy development

48. Nutrition and Physical Activity: The Washington State Story Take Home Message When making policy choices, the feasibility of and the need for evaluation should NOT be an afterthought

49. Nutrition and Physical Activity: The Washington State Story Victor Colman, JD Senior Policy Advisor Division of Community and Family Health, Washington State Department of Health Tel. # 360.236.3721 victor.colman@doh.wa.gov