Process and Outcome Evaluations

Process and Outcome Evaluations ZhaBlongXiong, Ph.D. Associate Professor Department of Family Social Science University of Minnesota

Process Evaluation Design • Data for the process evaluation were based on a cross-sectional design using a mixed method approach. Specifically, observation, survey, face-to-face interview, and personal journal were used to collect data from the advisory team and staff to determine STEEP’s strengths and weaknesses.

The Dandelion Effect Approach • The dandelion effect approach is based on the idea that to prevent tobacco use in the Southeast Asian community it is essential to: • build the capacity of the community by providing on-going partnership opportunities, mentoring, and support for members of the community or volunteers to deliver the tobacco education; • use innovative but culturally-tailored, multi-approached education to bring awareness to the people and agencies in the SEA community; and • engage the community to bring change to socio-cultural norms and practices.

Strengths • Members of the Collaborative • have participated in various local and national conferences, training, and workshops on tobacco control, coalition building, program sustainability, and financial accountability. • are passionate about the collaboration and continue to maintain a strong commitment to serve the community. • are willing to look beyond their differences to work for a common goal. • willing to share resources and sacrifice for the community.

Strengths • Staff • Have four full-time Southeast Asian staff whose work focuses exclusively on tobacco education and engagement. They are passionate about the work, motivated to change community norms and policies, and committed to help each others. • Data show that the staff’s confidence level has increased from a “moderate” level to a “high” level. • Their ability to teach about tobacco issues has increased from a score of “low” to a score of “moderate” to “high.” • On average, they evaluated the effectiveness of the training program as “quite effective” in regards to tobacco knowledge and presentations.

Strengths • Feedback data from program participants found the educators to be knowledgeable about the subject matter (mean = 9.02; SD = 1.10). • Program participants also reported that the educators responded very well to the audience’s questions (mean = 9.26; SD = .92). • The majority of the program participants rated the training favorably as to how it was presented (mean = 9.20; SD = .90) and how valuable the training was for their work (mean = 8.85; SD = 1.39).

Limitations and Challenges • Collaborative Board: • Not everyone has a chance to participate in all of the conferences, training, and workshops. • Concept of time is viewed differently by different collaborative members. • Showed up late at meetings • Financial accountability and reporting • Management practices – reallocating STEEP money to cover other projects for sustainability reasons. • Accounting practices – the practice of simplifying the accounting system without complying to the fiscal agent’s requirements.

Limitations and Challenges • Collaborative Board: • Different levels of commitment from the collaborative members. • Hurried life, limited time. • Unequal staff size and experience between organizations. • Lack of funding for smaller agencies. • “A new problem we are facing now is that each partnering organization has faced its own financial problems. Not [only the] tobacco budget, but also budget from our funders. All of us have to cut something, and we have to work harder. If you have that kind of problem, you tend to have less commitment to STEEP. For agencies with fewer staff, EDs have to work a lot harder in their own agencies.”

Limitations and Challenges • Staff • The challenge of training and retaining volunteers without any monetary incentives. • The challenge of record keeping and documenting information to demonstrate the levels of penetration and impact in the community. • The challenge of seeking alternative funding sources to help sustain the collaborative beyond the current funding stream.

Outcome Evaluation Design • Based on a cross-sectional evaluation study using a retrospective design to follow up with volunteers, program participants, and partners who have been involved with STEEP in the past. • A sample of 190 (53.7% male, 44.7% female) where 28% were Cambodian, 26% were Hmong, 32% were Lao, 6% were Vietnamese, and the remaining 8% were from other ethnic groups. • Assessed five areas: • tobacco knowledge, • beliefs about tobacco use, • norms around tobacco use, • visibility of STEEP in the community, and • effect s STEEP has on the participants.

How much have we been penetrating the SEA communities? • STEEP’s staff have attended 65 events where Asians were present (or where there were a high concentration of Asians in attendance). • The educators have delivered their messages to 40,000 people in three major target Southeast Asian locations: Twins Cities, Rochester, and St. Cloud. • The educators have recruited and trained 120 volunteers. • Of the 120 volunteers, 20 have done some education in their respective community.

How much have we made a difference in the SEA communities? • All collaborative agencies have adopted a policy that all agency sponsored activities be tobacco-free. • All collaborative agencies have adopted a policy not to accept money from tobacco companies. • Worked closely with St. Paul Parks to adopt a policy not to smoke in parks if youths under the age of 18 are present. • As a result of STEEP’s involvement, a new policy has been signed into law in Minneapolis prohibiting people from smoking at play grounds, swimming areas, and other designated areas. • The Planning Committee of the World Refugee Day has verbally agreed for two years (2009 – 2010) to have a smoke-free event. • The Dragon Festival Planning Committee has verbally agreed for 2 years (2009-2010) to have a smoke-free event.

How much do the people in the community know about STEEP?

How much have we made a difference in the SEA communities?

How much have we made a difference to the people we educate?

Ethnicity • The Cambodian group indicated greater knowledge gained than did the Hmong group. • The Lao group indicated greater change in their beliefs compared to the Hmong group. • When asked the question “Do you think smoking should be used at all times, some of the times, or not at all during family gathering events and celebrations?” three of the four ethnic groups (Hmong, Lao, and Vietnamese) tended to think smoking should not beused at all during family gatherings, whereas the Cambodian group had more respondents who still believed smoking should be used. • When asked, “In the past two years have you seen any changes in your community about smoking?” the Hmong group tended to see fewer changes in their community compared to Cambodian, Lao, Vietnamese, and other Asians groups.

Involvement • “In the past two years have you seen any changes in your community about smoking?” • Both the partner and volunteer groups saw more changes in the last two years than did the participant group. • Furthermore, the partner group tended to report seeing more changes than all the other groups.

Age • “In the past two years have you seen changes in your community about smoking?” • The middle-aged group (age 46 – 54) reported seeing the most changes in their community about smoking in the last two years, while the young group (age 12 -25) reported seeing the least amount of change. • “Do you think smoking should be used at all times, some of the times, or not at all during family gathering events and celebrations?” • The young group (age 12 – 25) overwhelmingly felt that smoking should not be used at family gatherings. Whereas the middle-aged group (age 46 – 54) continued to hold some beliefs that smoking should be used at family gathering events and celebrations.

Do you have any suggestions to help STEEP do a better job of educating more people in the community about the dangers of tobacco use? (n=129) • Continue and increase STEEP involvement. Comments such as “Keep sharing over time, the hope is some people will change” and “They won’t come to you, tell them about the harm” indicate the urgency respondents feel to the issue of continuing the educational process within their communities. • Involve and empower ethnic communities. A very strong theme of continuing STEEP education in the SEA communities and the value of empowering culturally specific methods was urgent as seen in the following comments, “I want STEEP staff to continue working on the issue so one day people can stop using it”, “Keep educating the community, if STEEP stops the project – who will do it?” • Strategies for outreach to target areas. Respondents have many, many ideas for how to outreach into target areas. Outreach includes geographic locations; not yet reached such as rural areas. “Reach wider”, go to events, and a very specific direction “Go to the funeral home in South St. Paul.” These are just a few comments that can guide future outreach in the next phase of STEEP projects.

Recommendations • Target new places of community congregation (i.e., funeral homes, multi-unit housing). • Target youths both as volunteer participants. • Target the middle-aged group (age 46 – 54) since respondents continued to hold some beliefs that smoking should be used at family gathering events and celebrations and. • Target supporting the existing pool of volunteers.

Process and Outcome Evaluations