American Public Health Association November 19, 2003 San Francisco, CA

1. Can a Very Large-Scale, Diverse Nutrition InitiativeBe Evaluated at Multiple Levels of Impact?B Foerster1, H Chipman2, J Guthrie3, C Olander4, Eileen Stommes3, S Nitzke5, J Voichick51 California Dept. of Health Services;2 South Dakota State University and CSREES, USDA; 3Economic Research Service, USDA; 4Food and Nutrition Service, USDA;5Cooperative Extension, University of Wisconsin American Public Health Association November 19, 2003 San Francisco, CA

2. Objectives • Introduce issues in evaluation of state Food Stamp Nutrition Education • Describe evolution of state and national evaluation efforts • Synthesize the issues as a public health nutrition practice challenge, from a state perspective

3. What Is Food Stamp Nutrition Ed (FSNE)? • Optional administrative activity for Food Stamp state agencies • Nut Ed = “any combination of activities leading to the voluntary adoption of healthy eating behaviors” • Cooperative Extension started in early 1980’s; social marketing in mid-1990’s • 50:50 Federal Financial Participation with USDA; in-kind OK • Grew from 0 to 49 states; <$1 Million to $250 Million by FFY 2003 • Federal oversight: Annual Guidance from HQ; state plans and semi-annual reports approved by FNS Regional Offices

4. Association of State Nutrition Network Administrators(Formal Social Marketing Nutrition Networks) • Alabama Missouri • Arizona Nevada • California New Jersey • Colorado North Carolina • Georgia Oklahoma • Iowa Oregon • Kansas Pennsylvania • Maine South Dakota • Michigan Washington • Minnesota Wisconsin

5. Three USDA Agencies Participating + Other OversightFrom a State Perspective • Food and Nutrition Service—Food Stamp Program, the Office of Analysis, Nutrition and Evaluation; 8 Regional Offices, Internal Audits • Economic Research Service, Food Assistance and Nutrition Research Program • CSREES • Office of Management and Budget • Government Accounting Office • Office of the Inspector General? • State agencies—Food Stamp + health department/university

6. Benchmark: Recommended Priorities w/ Next Steps, Mid-late ‘90’sWhite Papers Published in JNE Supplement • Diet quality • Food insecurity • Food safety • Food resource management • Systems and environmental change Papers Soon To Be Available on ERS Website www.ers.usda.gov Papers Soon To Be Available on ERS Website www.ers.usda.gov

7. Benchmark: ERS Workshop, January 2003 • Small group format (USDA agencies, other feds, academics, state agency) • Identify research needs to support evaluation • Foci—further define outcomes; develop info systems, process measures, strategies for collaboration; maintain ongoing dialog and evaluation research processes • ERS FSNE ExtraNet, FNE Conference and Web Page w/ data links

8. Benchmark: SNE/ERS Conference, July 2003 Objectives • Identify potential outcome measures for nutrition education w/ low income consumers • Explore automated, web-based systems for evaluation and linking process strategies • Identify how program reporting systems can be supported and process strategies linked • www.ers.usda.gov (topics include diet and health, food security, links, data base of projects)

9. SNE/ERS Highlights: Challenges in Outcome Measurement(Isobel Contento, PhD) • Stakeholders, partners, communities; accountability, effectiveness • Deciding on objectives, core elements; diversity and creativity of programs; general planning framework (What to put in the ‘impacts’ box?) • Measures for individual factors (validity and reliability of tools) • Defining measures for environmental factors (organizational; community; social structure, policy and systems) • Measuring impacts of environmental change (institutional, community, state levels)

10. SNE/ERS Highlights: Recommendations for Outcome Measurement(Isobel Contento, PhD) • Strengthen w/ state and national evidence (beyond local); decide on core behaviors and impacts for all programs • Match intervention objectives w/ measurement tools; make appropriate for duration and intensity • Use control/comparison groups • Monitor environmental noise and mediating variables • Use appropriate theories to select measures • Track over time; find creative ways of capturing data, e.g., on-line reporting, partnering

11. SNE/ERS Highlights: Existing State FSNE Reporting SystemsResults of a Survey, Spring ’03 (n = 42 states) • Both quantitative and qualitative data are collected by virtually all S/A on 4 core areas; of those, food security is least reported • There is a reporting ‘pyramid’ by Level of Influence, e.g. 46-88% of S/A collect data @ Individual/Household level v. 7-17% @ Social Structure/Policy Change • At the Community level, formal partnerships were the most common measures • At the Social and Policy levels, fewer than 4 programs reported on public opinion or public policy • These data should be mined further

12. SNE/ERS Highlights: Issues in Automated Data Systems • Other agencies are doing it—education, substance abuse, social science • Why? To link activity w/ performance, connect groups, minimize reporting burden • Many technical considerations in designing systems • Example: California Communities of Excellence in Tobacco Control • OTIS Features—contracts, progress reports, electronic approvals, cost reports • Statewide reporting of Core Indicators—ETS, availability, countering pro-tobacco influences, assets, cessation (% s/w activity, costs) • Reports—coalition activities, challenges/barriers, staffing, progress, evaluation plan, budget—s/w evaluation comes from s/w surveys • Many lessons learned

13. SNE/ERS Highlights: FNS ‘EARS’Formation of an Education & Administrative Reporting System • Objectives: Develop standard state reports and uniform data elements; paint picture of state activities • Uses: Inform management decisions, support policy initiatives; document for legislative, budget and other requests; support planning functions • Evaluation niche: Describe target groups, gaps, channels, resources • Progress to date: Public notice, appoint Working Group, review S/A reports, establish subcontracts for facilitator and field testing • To do: Draft report, synthesize comments, field test, get OMB clearance implement by FFY ‘05

14. SNE/ERS Highlights: CSREES Logic Modelwww.csrees-fsnep • Overview—InputsOutputsOutcomes/Impact (short, medium, long-term) • Core Elements—Dietary quality, food security, food safety, shopping behavior and resource management indicators • Inputs—Financial, planning, materials, people • Outputs—Activities, participation • Three levels—Individual and households (individuals, agencies, policy makers); Communities and institutions; Social structures, policies, and practices

15. SNE/ERS Highlights: CDC Perspectivewww.cdc.gov • CDC is asked by Congress about FSNE issues! • Model of Behavioral Epidemiology: Guidelines, recommendations, training materials, fundingpolicies, programs, service delivery, provision of benefitsrelevant knowledge, attitudes, skillshealth behaviorshealth, social outcomes, well-being, quality of life • Cycle: SurveillanceDefine problemEstablish prioritiesDevelop program plan and objectives ImplementMeasure change and strengthen program • Logic Framework: Modifiable determinants (food supply, environment, consumer demand)Population food intakeCommunity health outcomes • Available surveillance systems: BRFSS, NHANES, YRBSS, PedNSS, PNSS, SHPPS

16. SNE/ERS Highlights: Build on Social Sciences ResearchExamples from the Southern Rural Development Center, MSU • Multiple obesity and poverty, food assistance studies • Retail globalization and food access in the South • Relationships of food insufficiency to disease risk and outcomes • Contextual determinants of food security in Hispanic and African-American neighborhoods; assessment of faith-based food assistance programs • Charitable agency profiles and directors’ perceptions of needs and opportunities • Studies on EBT, micro- and macro-level determinants of Food Stamp utilization

17. SNE/ERS Highlights: Observations of an Evaluation SpecialistEllen Taylor-Powell, PhD Top 10 Requirements • Strong leadership • Common language • Engaged stakeholders—funders, partners, staff, clients • Uses must be showcased for learning and internal value • Start with the “what”, then go to the “how”

18. SNE/ERS Highlights: Observations of an Evaluation SpecialistEllen Taylor-Powell, PhD Top 10 Requirements • Create an evaluation culture—it’s everyone’s responsibility • Build-in incentive and reward systems • Streamlined data collection system • Excellent training and TA • Investment—research studies, time, analytical expertise, collaboration

19. Proceedings of ERS January Workshop and SNE/ERS Workshop Now Available at http://www.csrees-fsnep.org/confproc.cfm

20. Benchmarks: Who’s Doing What Next?A Work in Progress • USDA—Interagency Working Group, web info, CDC participation? • FNS—EARS + contracted report on qualitative profile of S/A FSNE • CSREES—Continued work on Community Nutrition Logic Model, state reports • ERS—Developing research project to improve outcome measures; developing FSNE page on ERS website with interactive data features for use in FSNE planning and evaluation • ASNNA—Evaluation Committee, individual states share systems at Winter meeting, development of indicators in ‘outer spheres’? • GAO—Report scheduled for April ‘04 • Congress, state agencies and legislatures??? • SNE—social marketing division and scientific programs