Evaluation Methods for School-based Health/Wellness Programs

1. Evaluation Methods for School-based Health/Wellness Programs Scott Burg scott@rockman.com Frances Montell frances@rockman.com Rockman ET AL www.rockman.com

2. Rockman ET AL • Research and evaluation organization • Projects focus on formal and informal education • Plan and conduct formative and summative evaluation • Implement research models, and conduct strategic planning • Clients include: • Oakland Unified School District • California Academy of Sciences • West Virginia State Department of Education • Intel • Pacific Gas and Electric • Offices in San Francisco and Bloomington, Indiana www.rockman.com

3. Cover both product and process Cite relevant literature Tell who will conduct the evaluation, and how/why they were chosen Define variables and data gathering methods Explain test instruments, questionnaires, etc, and their validity and reliability Describe data analysis procedures Relate the evaluation to a plan for improving the project Describe evaluation reports to be produced A good evaluation section should…

4. Who is your audience? funders / potential funders clients partners potential participants What kind of information will be persuasive to this audience? numbers stories and anecdotes (most audiences need both) Most of all, a good evaluation section should tell a story

5. Formative AND summative assessments Mixed methods Determine evaluation budget ranges Some fixed/other variable Links to sustainability What happens when funding ends Important to tie to evaluation plan Collaborations Diversity and breadth Plans for applying/leveraging findings elsewhere Elements for Evaluation:Private Foundations

6. Use of internal vs. external evaluators Reporting Formats Schedule Private Foundations

7. Havenscourt Salad Bar Evaluation • Primary partners • Havenscourt Middle School • Oakland Unified School District (OUSD) • Oakland Small Schools Foundation (OSSF) • Rockman et al • SD Bechtel Jr. Foundation

8. OUSD • 6th largest school district in CA (42K students) • 41% African American, 34% Latino, 15% Asian, 3% Caucasian • 68.9% qualify for Free and Reduced Meals • Shared control with state • 125 total schools • 23 with salad bars • 15 planned for 2008/9

9. Havenscourt Middle School Complex • Three school complex, part of small schools reorganization • Roots Academy; 360 students, Grades 6-8 • Coliseum Prep; 420 students, Grades 6-12 • Located in high poverty West Oakland neighborhood • Primarily African-American/Latino • 48% ELL • Free/reduced around 86% • Shared cafeteria

10. SD Bechtel Jr. Foundation • Family foundation of Bechtel Corporation • Primary program areas • Science, technology, engineering, math (STEM) • Environment • Funding primarily in SF Bay Area • Support OUSD Expect Success and Small Schools programs

11. Bechtel Interest in Program • Linkage of health programs to formal science curriculum • Connection of improved health status to higher test scores • Concern in making additional investments in OUSD programs • Request to work with 3rd parties

12. OUSD Salad Bar Model

13. Proposed program outcomes • Students will increase fruit/vegetable consumption • More students will participate in school lunch program • Students will receive information and education about healthy nutrition and diet • Complimentary formal and informal health and education programs in curricular areas such as science will be developed • Parents will become more involved in school nutrition and wellness programs • Havenscourt salad bar will provide a model for other schools in OUSD

14. Assessment methods • School lunch data analysis • Student surveys • Lunchroom observations • Key stakeholder interviews • Students • Teachers • Administrators • Food service staff

15. Sustainability elements • Guarantee that salad bar program has adequate and consistent volunteers to manage the program • Manage labor costs and maintain quality of salad bar without ‘cutting corners.’ • Increase community awareness of salad bar program • Introduce and implement methods to ensure further increases in student participation in lunch program • Incorporate more formal and informal nutrition and health curriculum during school and out-of-school time • No need for additional financial support from outside agencies

16. Timetable • April 2007 • Conducted pre-implementation student surveys and observations • Food service staff and students trained by OUSD consultant • Salad bar implemented • Conducted additional observations

17. Timetable • November/December 2007 • Salad bar in place for a full semester • Conducted interviews with project stakeholders • School principals • PE/Nutrition Education teacher • Food service/cafeteria manager • Students • OUSD Nutrition Services Director • Analyzed data • Submitted final report to SD Bechtel Foundation

18. Findings • General • After initial period of ‘adjustment’, salad bar embraced and accepted by students and school community • Students availing themselves of salad bar options in increasing numbers • For many students, salad bar represents the first time they have eaten diversity of basic fruits and vegetables • For other students, lunch/salad bar is their only meal of the day

19. Findings: Acceptance by Students • The kids look forward to this (salad bar). Their first questions when they came back to school were: Where’s the salad bar? How come the salad bar isn’t set up? (principal) • If there’s a hundred kids that come in here I would guarantee that at least ninety kids stop by the salad bar. Some of them just go in and go straight to the salad bar (cafeteria manager) • I feel like the kids are now thinking of the salad bar as a food option (principal) • Now I eat from the salad bar everyday (student)

20. Findings:Implementation and Training • Two week training of food service staff well received • Training was instrumental in • Creating trust with cafeteria staff • Staff members committed to educating students about benefits of salad bar and healthy eating • I think they (other cafeteria managers) will have the same apprehensions that I had initially, and I think the same thing will happen with them that happened with me. It’s going to be smooth, and it’s not going to be as bad as they thought it was. (cafeteria manager)

21. Findings: Access to and Knowledge of Healthy Food • Students knowledge of and access to healthy food was limited, but as a result of salad bar access and knowledge improved • Some students appeared to understand that fruits and vegetables could improve health, but could not articulate why • Indication that some students reduced intake of ‘unhealthy’ foods because of salad bar experience

22. Findings: Access to and Knowledge of Healthy Food • I’ve got boys, particularly, who take fresh fruit all the time who wouldn’t touch anything fresh when it came from behind the counter and kids who will eat vegetables on the salad bar that they won’t touch if they come in the tray, because they can self-select (principal) • Interviewer to student: Have you stopped eating food you think isn’t healthy, since you’ve been eating the salad • I stopped eating chips (student) • I eat less candy (student)

23. Findings: Access to and knowledge of healthy food • In deciding which salad bar items to select/taste, peer acceptance and food appearance were the primary criteria. • Jicama, which most students had never heard of, ended up as one of the most popular salad bar items. • Latino students ‘educated’ the African-American and Asian students about jicama • Introduction of new items such as jicama helped improve attitudes and knowledge of food service staff • Increase in student acceptance of and familiarity with salad bar items was due in large part to peer-peer ‘education’

24. Findings: Access to and knowledge of healthy food • There have been a few times when there was melon that’s not cantaloupe. They have to wait to see if somebody else takes it and likes it before they’ll take a big slice (cafeteria manager) • Last year there was this big jicama issue and it was a bunch of Latino boys who were promoting it to everyone else….now everybody eats the jicama. (cafeteria manager) • The Latino kids are educating the African American kids, but the African American kids are teaching the Latinos as well. As long as they’re talking and being educated, that’s a good thing. (principal)

25. Salad bar staffing • Initial plan was to have students staff salad bar • Students did not show up to work • Difficult to manage • Students had difficulty managing other students and portion size • Parent volunteers recruited to staff salad bar • Parents have been successful in: • Educating and encouraging students to try different items • Managing portion size • Improving perception and morale of food service staff • Improving student perception of parents role in school activities

26. Salad bar staffing • Process of recruiting parents provided informal, but effective, method for informing them of health/wellness activities at school • Parents become informed and involved in school health/wellness programs • Parents helped break down stereotypes of how school food service operations are managed

27. PE and Nutrition Education • No ‘formal’ linkages from salad bar to core content subject areas (math, science, etc) • Two days a week all grade levels are provided Nutrition Ed as part of PE • Not an OUSD model; specific to Havenscourt • PE teacher used curricular material downloaded from National Dairy Council website • Informal integration of ‘lessons’ from cafeteria and salad bar • Discuss importance of eating healthy. Link to student’s performance in class • Kept food diaries outside of class

28. PE and Nutrition Ed • I think through that (addition of salad bar) and some of the curriculum that we have changed student behaviors and knowledge about healthy eating. It's broadened their perspective. It's not just, "Okay, I know what a tomato is." And so they can talk about now I believe different types of tomatoes. Which is good. Which is something they couldn't do before. Most of them can tell you what food group it fits into and how many servings they should have per day which is something that prior to the curriculum and the salad bar they couldn't do before. (PE teacher)

29. Staff Wellness • Increasing numbers of faculty and staff are eating in the school cafeteria since the salad bar was introduced • Faculty/staff beginning to model healthy eating and other behaviors for their students

30. Parents • Not easy to involve parents in school or after school health and nutrition activities • Outreach to parents has included: • Soliciting for snack bar volunteers • Advisory sessions with school faculty • Student work material from PE/Nutrition class

31. Going Forward • Evaluation resulted in funding for neighborhood produce stand, serving Havenscourt and 3 additional OUSD sites • Funder provided support for salad bars at two additional OUSD school sites • Proposal submitted for broader evaluation of OUSD salad bar sites and models • Linkage of these studies to school garden activities • Involvement of local CBOs and national organizations and funders

32. Salad Bar/School Garden Study • Diversity of grade level and demographics • Focus on process and implementation components • Other areas: • Parent/community involvement in staffing and maintaining salad bar/school garden • Teacher and parent attitudes • Methods for linking salad bar and school garden programs • Formal and informal • Role of other partners (city, CBOs, individuals)

33. Designing an Evaluation Create a Logic Model Develop Research Questions Choose Methods

34. What is a Logic Model? • graphic depiction of the conceptual framework of your program • sometimes called a “Theory of Change” • describes components of program, outcomes, and impacts, and the links between them • shows the logic behind the theory that the program will lead to the anticipated changes • Why is it necessary? Because without an explicit model, you will fall back on…

35. Unconscious Evaluation Model

36. Other Logic Model Examples • Nutrition Education Project • CDC programs for reducing exposure to tobacco see Evaluation Springboard: http://www.evaluationspringboard.org/ for these and other examples

37. Template

38. Activity: Create a Logic Model of Your Program • Start with Goals (fundamental changes that the program will achieve) • Inputs/Context/Resources = opportunities and constraints • Program Activities = what you will do • Outputs = tangible results of program activities • Outcomes/Impacts = specific consequences of the Activities and Outputs

39. OUSD Salad Bar Model

40. Research Questions: Formative(Salad Bar and Garden Projects) • Implementation (formative or process evaluation) • What was involved in getting the salad bar/school garden up & running? What issues came up and how were these dealt with? • What was the level of parent/community participation in staffing and maintaining salad bar and school garden? • How often was the salad bar available to students? • How effective and informative were the marketing and signage? What strategies were most effective?

41. Research Questions: Summative(Salad Bar Projects) • Consumption (outputs) • Was there an increase in student participation in the lunch program? • What was the level of student consumption of salad bar food?

42. Research Questions: Summative(Salad Bar & Garden Projects) • Attitudes (outcomes/impact):Did the project affect student attitudes? • What were students' attitudes about "salad" and food before the project? What were their attitudes about the environment? • Were there cultural differences in these attitudes? • What misconceptions did students have about food and nutrition? • How did students feel about the salad bar? What did they like/dislike? • Did student attitudes about food and "salad" change after their experience with the salad bar?

43. Research Questions: Summative(Salad Bar and Garden Projects) • Knowledge (impact) • Did the program increase student knowledge of nutrition, health, or science? • Did the program increase students’ understanding of key life sciences concepts and science inquiry skills? • Behavior (impact) • Did the project increase students’ willingness to try new foods? • Did students make healthier snack choices? • Do students eat more fruits & vegetables? • Did students increase their physical activity level?

44. Methods • Data Collected by School, District, others (school lunch records, salad bar purchasing other surveys?) • school lunch participation • salad bar consumption • other info?

45. Methods • Interviews with School Staff (administrators, cafeteria staff, teachers) • Implementation Issues, how dealt with • Analysis/effectiveness of marketing, signage • Other impacts on school, students

46. Methods • Individual Interviews with Students • Student Focus Group Interviews • Student Surveys • Parent Surveys • Student Feedback on Salad Bar (general likes and dislikes, amount of participation/ consumption, Factors influencing food choices, suggestions for salad bar menu)

47. Methods • (Data from Students & Parents, contd.) • Students' Attitudes (about "salad," "salad bars," food, nutrition and health; cultural background; factors influencing food choices) • Students' Knowledge (of nutrition, health, science) • Students' Out-of-School Food Choice/Behavior (mealtimes, snacks, usual diet)

50. Closing Thoughts • Evaluation vs. Research • You need to pilot test instruments before you use them with your participants! (You cannot know if you are getting good info until you try to analyze it.) • Other resources • Evaluation Springboard: Online evaluation resource developed by Rockman to provide information on evaluation methods and strategies. • http://www.evaluationspringboard.org/ • Rockman et al • http://www.rockman.com/