Analyzing your Survey Data: The Impact of the Campaign

1. Analyzing your Survey Data: The Impact of the Campaign

2. KAP Data analysis Part 2: BR, BC, TR and CR

3. By the end of this lesson you will be able to: • Analyze your data according to your analytical plan (BR, BC, TR, CR) • Analyze your data based on your reporting needs • Complete Sections 3.4 B – D of your Campaign Learning Report that identify the impact of your campaign along the Theory of Change

4. Structure for the Day • Check in 9:30 – 10:00 AM • Individual Work time 10:00 – 12:00 • Check in 1:30 – 2:00 PM • Individual Work time 2:00 – 3:00 • Check in/Assignment 3:00 – 3:30

5. Assessing Impact • Provides an overview of the results of the campaign along the: • BR SMART Objectives • BC SMART Objectives and • TR SMART Objectives • CR SMART Objectives

6. Flow/Guidelines • Review of your KAP Monitoring Plan (BR, BC, TR, CR) and Hypothesis • Complete the tables in your section 3.4 per Target Audience • Write a short narrative after each table describing how well your campaign achieved its SMART Objectives • Assess the impact of your campaign along TOC

7. Analyze Data: Statistics clarify truth Show critical data in body of report (SMART Objectives) • Tables • Charts Show all your data in tables in Appendix: • Honest thing to do • May show you where your campaign failed so you can fix it in future!

8. Analyze Data: Comparability of Surveys • Best: • All Chi-Square tests < 95% • OK: • A few Chi-Square tests ≥ 95% • Differences in frequencies small (5 to 10 percentage points) • So-so: • Several Chi-Square tests ≥ 95% • Differences in frequencies large (10 to 20 percentage points) • Unacceptable: • Many Chi-Square tests ≥ 95% • Differences in frequencies large (> 20 percentage points)

9. Worse Case Scenario • Assume you have large differences in gender: • Baseline = 35% male • Post-campaign = 56% male • Difference of 21 percentage points • Problem: If a dependent variable increases by 15 pp, it could be due to 2 things: • Pride campaign impact • More men in 2nd sample • Solution is to “control” for gender using filters: • Filter for men, run analysis • Filter for women, run analysis

10. Reminder # 1 • For BR, BC, TR and CR, KAP SMART objectives are based on self-reported data. • Triangulate results with non-KAP metrics

11. Analyzing BR Example: BR KAP Question • (36) I am going to read you a number of statements about the management of the local no-take area. For each statement, I would like you to tell me if you strongly agree, agree, disagree, or strongly disagree with it. • (E) There is enough money and other resources to fully manage and enforce the rules of the no-take area • [ ] SA • [ ] A • [ ] D • [ ] SD • [ ] NS/DK • (G) The rules of the no-take area are unclear and local fishers don't understand them • [ ] SA • [ ] A • [ ] D • [ ] SD • [ ] NS/DK BR: Non-KAP Measure?

12. Analyzing BC Example: BC Question • During the past 6 months, would you say that you have been regularly involved, occasionally involved, or not involved with the creation and/or the management of a no-take fishing area in your local area • [ ] Regularly involved • [ ] Occasionally involved • [ ] Never involved • [ ] Don't know / not applicable BC: Non-KAP Measure?

13. Analyzing TR Example: TR Question • I am going to read you a list of different types of fishers, and for each one, I would like you to tell me whether you remember seeing someone like that fishing in this area in the past 6 months (show the NTZ on a map of the area but don't mention whether it is NTZ or not) • (A) Subsistence fishers from your village • [ ] Seen • [ ] Not seen • [ ] Not sure / Don't remember • (B) Subsistence fishers from nearby villages • [ ] Seen • [ ] Not seen • [ ] Not sure / Don't remember TR: Non-KAP Measure?

14. Analyzing CR Example: CR Question • Has your catch increased, decreased or stayed the same as a result of the Lola Marine Sanctuary? (If the person does not fish or glean mark as NA) • [ ] Decreased • [ ] Increased • [ ] Stayed the Same • [ ] N/A CR: Non-KAP Measure?

15. Structured Time:KAP monitoring Plan data survey template • Proceed to section 3.4A in your campaign learning report

17. Reminder # 2 • GENERAL COMMUNITY (NON-HUNTERS)

18. Interpersonal Communication Our goal for interpersonal communication was simple. We wanted people to talk to each other about wildland fires, about the causes, the effects, the billboards, etc. We achieved 190% of the objective for the general community, though the data for the hunters was not statistically significant and cannot be assessed. This may be attributed to a very small sample size of this audience in the pre campaign survey. Respondents were also asked if they had heard that there were wildland fires in Guam’s watersheds. While cannot be linked directly to interpersonal communication, it is an interesting question to look at to see if there is an increase in people hearing about fires since the pre campaign survey. The general results of all of the respondents showed an increase from 22% to 45% of people who had heard about wildland fires, indicating that there is indeed more information about fires being communicated. While many of the campaign images focused on the impacts of wildland fires, such as the campaign poster and billboards, they did not directly state that fires were caused by people and that Guam had a very low occurrence of natural fires. The campaign display game addressed this as did the community and school presentations, but it may have been advantageous to include this messaging into more campaign materials. It may also be a good idea to do some supplemental surveys with more direct questions about these important knowledge concepts to get a more accurate representation of what people know.

19. Reminder # 2 • Use the exact SMART objectives language Ex. Strongly agree is not the same with agree SurveyPro result

20. Scenario • There is an increase in reported BC but the SMART Objective target are not met. • WHY?

21. Could it be how you set your SMART target? • Historic data • Baseline - Diffusion of innovation • Type of audience • Increase/Decrease • Maintain

22. The potential for change for different types of objectives across the Theory of Change Historic Data Pride campaign average results summary till 2010

23. Baseline and diffusion of innovation Laggards – 16% The Late Majority – 34% Early Adopters – 13.5% The Early Majority – 34% Innovators – 2.5% Source: Everett Rogers, graph from Wikipedia.org According to Diffusions of Innovation the Rate of Change Depends on the Starting Point

24. The potential of change for different types of audiences • Selective perception(Hassinger) – people who don’t “want” to know don’t seem to learn. • The critical mass phenomenon / social norms

25. 3. Using right words for accuracy The SMART objective should use the same words as the question & as the answer option used

26. Reminder # 3 • Check if you use the right filter. The latest appears in the dropdown menu. Take note of the code of the filters you use.

27. Stages of behavior question Can it be used to identify the current stage of behavior?

28. Reminder # 5 • Try other figures beside using tables • When you are done with your analysis, do a publish report

29. Reminder # 6 • When you are done with your analysis, do a publish report