780 likes | 803 Views
This presentation discusses the opportunities for growing the field of evaluating policy and environmental changes. It highlights the different team members and partners involved in this field, as well as the importance of evaluation in informing local decision-making and obtaining more funding. The presentation also touches on the need for reliable and valid quantitative tools and measures, as well as study design and execution to ensure confidence in the findings.
E N D
Tools to evaluate policy and environmental changes: Presented by: Laura K. Brennan, PhD, MPH Opportunities for growing the field
Our Team & Partners • Robert Wood Johnson Foundation • Tracy Orleans, Laura Leviton (and Punam Ohri-Vachaspati) • Active Living By Design • Sarah Strunk, Phil Bors, Rich Bell, Fay Gibson, Joanne Lee, Mary Beth Powell, Tim Schwantes, Risa Wilkerson • Community Partnerships (25 ALbD, 50 Healthy Kids, Healthy Communities) • Washington University Institute for Public Health (St. Louis) • Ross Brownson, Cheryl Carnoske, Christy Hoehner, Peter Hovmand, Timothy Hower • Saint Louis University School of Public Health • Elizabeth Baker, Cheryl Kelly • Transtria LLC • Tammy Behlmann, Sarah Castro, Julie Claus, Peter Holtgrave, Courtney Jones, Allison Kemner, Laura Runnels (many part-time staff and interns)
Healthy Kids, Healthy Communities (50 Grantees) HKHC Leading Site Communities Seattle/King County, WA Portland/Multnomah County, OR Houghton, MI Benton County, OR Fitchburg, MA Rochester, NY Milwaukee, WI Buffalo, NY Somerville, MA Kingston, NY Flint, MI Kane County, IL Philadelphia,PA Chicago, IL Omaha, NE Oakland, CA Hamilton County, OH Washington, DC Denver, CO Watsonville/Parajo Valley, CA Kansas City, MO Charleston, WV Central Valley, CA Columbia, MO Louisville, KY Nash/Edgecombe Counties, NC Cuba, NM Baldwin Park, CA Knoxville, TN Moore/Montgomery Counties, NC Rancho Cucamonga, CA Chattanooga, TN Boone/Newton Counties, AR Greenville, SC San Felipe Pueblo, NM Spartanburg, SC Jefferson County, AL Desoto/Marshall/ Tate Counties, MS Phoenix, AZ Milledgeville, GA Grant County, NM Jackson, MS Cook County, GA El Paso, TX New Orleans, LA San Antonio, TX Duval County, FL Houston, TX Lake Worth/Greenacres/ Palm Springs, FL Caguas, PR Healthy Kids, Healthy Communities
National Policy/Practice Advisory Group • Don Bishop • Elaine Borton • Leah Ersoylu • Steve Farrar • Harold Goldstein • Dean Grandin • James Krieger • Elizabeth Majestic • Jacqueline Martinez • MalisaMcCreedy • Leslie Mikkelsen • Joyal Mulheron • Thomas Schmid • Marion Standish • Ian Thomas • Mildred Thompson • National Research Advisory Group • Elizabeth Baker • Rachel Ballard-Barbash • Frank Chaloupka • William Dietz • Lawrence Green • Terry Huang • Shiriki Kumanyika • Marc Manley • Robin McKinnon • Shawna Mercer • Meredith Reynolds • Barbara Riley • Eduardo Sanchez • Loel Solomon Our Advisors • National Working • Group • Karen Glanz • Debra Haire-Joshu • Laura Kettel-Khan • Maya Rockeymoore • James Sallis • Janice Sommers • Mary Story • Sarah Strunk • Antronette Yancey
Connecting Projects • Bridge research/evaluation and policy/practice • Evaluate system, policy and environment change impacts and outcomes • Assess reach, adoption, implementation and sustainability of policy and environment interventions • Accelerate translation of replicable, evidence-based policy and environment interventions • Model the complex pathways from community contextual factors to behaviors and health
Why Evaluate? Green LW. Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence? Family Practice 2008; 1-5.
Why Evaluate? Building the scientific evidence
Why Evaluate? Shaping, creating policies and practices for the field
Why Evaluate? Chances of sustainability are higher for projects that have been evaluated StevensB, Peikes D. When the funding stops: Do grantees of the Local Initiative Funding Partners Program sustain themselves? Evaluation and Program Planning2006;29 (2): 153-161.
Why Evaluate? • To determine the effectiveness of local policy, environment, and system changes • To identify the changes with the greatest impact, relevance, feasibility and sustainability • To inform local decision-making, document successes & obtain more funding THESE ARE NOT MUTUALLY EXCLUSIVE…
Why Evaluate? • To determine the effectiveness of local policy, environment, and system changes • Short-, intermediate- & long-term health behavior or obesity outcomes • Reliable & valid quantitative tools & measures • Study design and execution to ensure confidence in the findings
Why Evaluate? • To determine the effectiveness of local policy, environment, and system changes • Short-, intermediate- & long-term health behavior or obesity outcomes • Reliable & valid quantitative tools & measures • Study design and execution to ensure confidence in the findings OPPORTUNITY: What is a meaningful decrease in BMI? Increase in activity? Decrease in sugar consumption?
Why Evaluate? • To determine the effectiveness of local policy, environment, and system changes • Short-, intermediate- & long-term health behavior or obesity outcomes • Reliable & valid quantitative tools & measures • Study design and execution to ensure confidence in the findings OPPORTUNITY: Should we use height/weight (BMI)? skinfold thickness? waist circumference? accelerometers? 3-day food recall interviews? surveys?
Why Evaluate? • To determine the effectiveness of local policy, environment, and system changes • Short-, intermediate- & long-term health behavior or obesity outcomes • Reliable & valid quantitative tools & measures • Study design and execution to ensure confidence in the findings OPPORTUNITY: Prospective cohort designs? Complex time series designs? Natural experiments? What is the intervention population? What is a representative evaluation population?
Why Evaluate? • To identify approaches with the greatest impact, relevance, feasibility and sustainability • What works, where it works, when it works, how it works & why it works (or why not) • Multi-method quantitative & qualitative measures • Local representation and participation to ensure confidence in the findings
Why Evaluate? • To identify approaches with the greatest impact, relevance, feasibility and sustainability • What works, where it works, when it works, how it works & why it works (or why not) • Multi-method quantitative & qualitative measures • Local representation and participation to ensure confidence in the findings OPPORTUNITY: Partnership or coalition formed? New decision-making body or position created? Policy developed? Policy adopted? Funds appropriated? New/improved structures, facilities or conditions? Policy compliance? Policy enforcement? Use of new facilities? Maintenance of new facilities?
Why Evaluate? • To identify approaches with the greatest impact, relevance, feasibility and sustainability • What works, where it works, when it works, how it works & why it works (or why not) • Multi-method quantitative & qualitative measures • Local representation and participation to ensure confidence in the findings OPPORTUNITY: Should we use policy assessment? environmental audits? direct observation? key informant interviews? focus groups? surveys? photovoice or digital storytelling? GIS mapping? web-based tracking systems?
Why Evaluate? • To identify approaches with the greatest impact, relevance, feasibility and sustainability • What works, where it works, when it works, how it works & why it works (or why not) • Multi-method quantitative & qualitative measures • Local representation and participation to ensure confidence in the findings OPPORTUNITY: (Perspective of community members) Does the policy or environmental change increase access to healthy foods or opportunities for physical activity? Did community members participate in planning, implementation and evaluation? Can the community sustain the changes over time?
Why Evaluate? • To inform local decision-making, document successes & obtain more funding • Track unintended results & practical considerations (resources, costs, assets & challenges) • Simple, quick measures serving multiple purposes (advocacy, marketing, cost savings) • Findings respond to the interests of local audiences (decision-makers, business owners)
Why Evaluate? • To inform local decision-making, document successes & obtain more funding • Track unintended results & practical considerations (resources, costs, assets & challenges) • Simple, quick measures serving multiple purposes (advocacy, marketing, cost savings) • Findings respond to the interests of local audiences (decision-makers, business owners) OPPORTUNITY: Who did the work? For how long? Who provided resources? How much did it cost? What funding was provided/leveraged? What were unanticipated benefits/challenges?
Why Evaluate? • To inform local decision-making, document successes & obtain more funding • Track unintended results & practical considerations (resources, costs, assets & challenges) • Simple, quick measures serving multiple purposes (advocacy, marketing, cost savings) • Findings respond to the interests of local audiences (decision-makers, business owners) OPPORTUNITY: What policy assessment tools inform policy development? What audit tools can be used to increase awareness? Can digital storytelling be used for advocacy? What surveys help to estimate costs?
Why Evaluate? • To inform local decision-making, document successes & obtain more funding • Track unintended results & practical considerations (resources, costs, assets & challenges) • Simple, quick measures serving multiple purposes (advocacy, marketing, cost savings) • Findings respond to the interests of local audiences (decision-makers, businesses, schools, residents) OPPORTUNITY: What are the costs? What are the impacts on economic development, academic performance, community safety or air quality? Are changes implemented and enforced equitably throughout the community?
Evidence With expanded definitions of evidence, answers to these questions help to bridge the gap between research/evaluation and policy/practice efforts…
Connect the intervention and evaluation Work plan and logic model Determine the evaluation approach Participatory, # communities, # strategies Use multi-method quantitative & qualitative measures Reliability, validity, feasibility Seek partners for data collection & data analysis Existing data, new data Translate and disseminate findings Audience, content, medium Evaluation: Beginning to End
Connect the intervention and evaluation Work plan and logic model Determine the evaluation approach Participatory, # communities, # strategies Use multi-method quantitative & qualitative measures Reliability, validity, feasibility Seek partners for data collection & data analysis Existing data, new data Translate and disseminate findings Audience, content, medium Evaluation: Beginning to End
Work Plan Planning Goals Objectives Activities Implementation Events Actions Evaluation Outcomes Impacts Processes
Work Plan Planning Goals Objectives Activities Implementation Events Actions Evaluation Outcomes Impacts Processes
Work Plan Planning Goals Objectives Activities Implementation Events Actions Evaluation Outcomes Impacts Processes
Logic Models • What is a logic model? • Description of how activities to be carried out during a project are related to the expected outcomes • Five core components: • Inputs: resources, contributions, investments • Activities: actions, events • Outputs: immediate products (trained staff, meeting attendees) • Outcomes: changes related to your objectives • Impacts: changes related to your goals • Other considerations: • Assumptions, External or Contextual Factors: beliefs about the people involved, interactions and influence of the environment, political context, social determinants of health
Connect the intervention and evaluation Work plan and logic model Determine the evaluation approach Participatory, # communities, # strategies Use multi-method quantitative & qualitative measures Reliability, validity, feasibility Seek partners for data collection & data analysis Existing data, new data Translate and disseminate findings Audience, content, medium Evaluation: Beginning to End
Evaluation Approaches • Understanding system, policy and environment impacts • Similar settings, approaches and strategies • Common tools and/or measures
Connect the intervention and evaluation Work plan and logic model Determine the evaluation approach Participatory, # communities, # strategies Use multi-method quantitative & qualitative measures Reliability, validity, feasibility Seek partners for data collection & data analysis Existing data, new data Translate and disseminate findings Audience, content, medium Evaluation: Beginning to End
Direct Observation Strengths (+) Pre/post comparison (+) Evaluates the impact of physical changes or improvements on behavior Challenges (-) May depend on external factors (e.g., weather, special events) (-) Requires many observations (times of day, days of week)
Environmental Audits Strengths (+) Validated tools (+) Pre/post comparison (+) Impact of policies or physical projects on environmental conditions Challenges (-) May not compare across different communities or physical projects (-) May not have facilities or environments to audit at baseline
Photos & Videos Strengths (+) Provides visual representation of project impacts (+) Conveys project impacts to diverse audiences Challenges (-) May be expensive depending on equipment and production (-) Requires consent for photo release
Policy Analysis CDC Community Health Assessment aNd Group Evaluation (CHANGE) Tool Strengths (+) Identifies policies that hinder or support healthy eating or active living (+) Tools provide potential policies for planning/ implementation W. K. Kellogg Food & Fitness Planning Guide http://www.cdc.gov/healthycommunitiesprogram/tools/change.htm YMCA’s Community Healthy Living Index (CHLI) Challenges (-) May not capture practices (informal policies) (-) May not track policy development, implementation or enforcement http://ww2.wkkf.org/DesktopModules/WKF.00_DmaSupport/ViewDoc.aspx?LanguageID=0&CID=6&ListID=28&ItemID=5000564&fld=PDFFile http://www.ymca.net/communityhealthylivingindex/tools.html