1 / 39

Evaluation of campaigns

Evaluation of campaigns. Dr. L.R. Pol Tabula Rasa / Erasmus University Rotterdam. Tabula Rasa?. Consultancy for evidence based and research driven advice … … in communication strategy and influence. Agenda. a. Influencing Behaviour. Evaluation Practice. Background. Background. a. a.

bary
Download Presentation

Evaluation of campaigns

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation of campaigns Dr. L.R. Pol Tabula Rasa / Erasmus University Rotterdam

  2. Tabula Rasa? • Consultancy for evidence based and research driven advice … • … in communication strategy and influence

  3. Agenda a Influencing Behaviour Evaluation Practice Background

  4. Background a a Influencing Behaviour Influencing Behaviour Evaluation Practice Evaluation Practice Background Background Governm. Practice Kinds of campaigns

  5. What kind of campaigns? • Isolated mass media campaigns: • Ads, RTV-spots, etc

  6. What kind of campaigns?(2) • Integrated campaigns: intervention programs which include for example: • Interpersonal communication: information, counselling • Teaching material • Tailor-made counselling via internet • Communication with intermediaries such as teachers, parents, physicians • Ads, RTV-spots, outdoor, website • Free publicity

  7. Governmental campaign practice in Holland • Increasing number of integrated campaigns (governmental campaigns) • Why: effects of isolated mass media spots and ads on behaviour are quite small • That’s why ads and spots are more and more part of an intervention program

  8. Governmentalcampaignpractice in Holland (2) • Isolated campaigns: 1 or 2 % change in behaviour (meta-analysis Derzon and Lipsey 2002) • Nevertheless: can be very worthwhile in the case of (for instance) public health • Especially when cost effectiveness is OK • Besides: new insights in behavioural change can lead to increasing effects

  9. Influencing Behaviour Influencing Behaviour Evaluation Practice Background Influencing Automatic Behaviour Planned behaviour Good Practice

  10. Crucial question in the case of influencing • Which kind of behaviour is involved? • Planned behaviour • Automatic behaviour

  11. Planned behaviourTheory of Planned Behaviour (Ajzen) (Beliefs ) Attitude  Social norm  intention  behaviour Self-efficacy 

  12. Automatic behaviour • However: at least 95% of our behaviour is not planned behaviour, but … • … automatic behaviour • We do it just because we always do it, not because we are considering it

  13. Automatic behaviour (2) • Knowledge and attitudes are not relevant determinants of automatic behaviour • Arguments aren’t even noticed! • Habits are the key determinants of automatic behaviour • But … automatic behaviour is changeable! • Only not via arguments, but via social influence, heuristics, primes • A lot of recent research on the topic

  14. Influencing automatic behaviour • Quite some campaigns still try to influence automatic behaviour by influencing attitudes • That’s useless and a waist of time and money • What needs to be done, is either break through routines… • … or instead use the fact that behaviour is automatic • In both words play an important role. And other stimuli: images, sounds or perfumes • They influence us very often, while we don’t even notice we were influenced

  15. Influencing automatic behaviour: some techniques • Priming • Use (or: abuse) heuristics (rules of the thumb) • Make use of the inclination to consistency • Using the inclination to reciprocity • Using authority • The liking principle: similarity, flattery • Social validation • The right example

  16. Voorbeeld: Probleemfocus

  17. Influencing automatic behaviour: alternative techniques • Alternative: break through routines: • Social network method • Using a negative approach • But not fear appeals!

  18. What depends good practice in governmental communication on? • Analyses of the behaviour is a condition for success: automatic or planned? • Empirical social scientific research offers crucial assistance for analyses and interventions • In case of influencing behaviour: trusting in hunches and creativity is dangerous: often counterproductive

  19. There’s no harm in just following hunches ? • A sloppy approach can result in the opposite of what you want to reach: • The wrong images or words (aggression) • Activation of undesirable exemplary behaviour (alcohol abuse) • Activation of undesirable processes (voting)

  20. Evaluation Influencing Behaviour Evaluation Practice Background What do you want to know What do you need to know Issues and Best practice

  21. Why evaluate? • At least three good reasons: • Find out if you didn’t cause a counterproductive effect • Learning effect: improve the campaign under construction during the process and use insights in new campaigns • Account for the use of government money

  22. Evaluation: what do you want to know? • What do you want to know? • Was the campaign effective? • What is effective? • What the target group knows? • What the target group thinks (attitude)? • What the target group does (behaviour)?

  23. Evaluation: what do you need to know? • When objective is to change knowledge or attitudes: • Direction of the change: positive or negative • Size of the change • Why did the attitudes change: • Change of the beliefs that constitute the attitude and relative weight of the beliefs

  24. Evaluation: what do you need to know? (2) • Planned behaviour: • Effect size of the total change in behaviour • Which determinants caused the change: • Change in the attitude towards the behaviour? • Change in vulnerability to what others think? • Change in the personal belief in the possibility of performing the desired behaviour

  25. Evaluation: what do you need to know? (3) • Automatic behaviour: • Change in the critical habits • Effect size of the change

  26. Subjects of evaluation • Effects • Effect size • Why did the effect occur (effects on determinants of behaviour and attitudes) • Process • Was the target group really confronted with the materials? If not: why not? • Did the target group understand the message • How did they evaluate the likeability of the message • Did they use the information? Did they do that in the intended way? • The same questions for the intermediaries (physicians, teachers, etc.) • Cost effectiveness

  27. Evaluation practice • Everyday practice: great diversity: • No need to know the real effects • Not want to know the effects • Deficiencies in the evaluation because of: • Lack of theoretical knowledge: asking questions about attitudes and intentions in the case of automatic behaviour; confusing intentions and the real behaviour

  28. Evaluation practice (2) • Lack of necessary methodological knowledge • Lack of time and money for a genuine evaluation • Naïveté towards research bureaus • And happily also good practice

  29. Best practice • Gather the parameters you want to evaluate from adequate theory: • for instance planned behaviour and current insights into automatic behaviour • Pre-test the campaign materials (in a sound way) and adjust them if and where necessary • Evaluate the campaign materials in randomized and controlled experiments (lab experiments)

  30. Best practice (2) • Evaluate the effect and the process in a field experiment

  31. Serious problems in evaluation • No time and / money for: • Lab experiments: therefore no sound insight in the question if the campaign materials reallycause the aimed effects (on behaviour, attitudes, knowledge) • Field experiments: therefore no possibility for generalization and also no learning effect

  32. I’d suggest the next solution • Don’t evaluate every individual campaign according to the standard of the best practiceif there’s no money to do it as it should be done • Evaluate extensively one specimen of a special kind of campaigns • Evaluate individual campaigns much less extensive: • Effect size • Process evaluation

  33. More serious problems in evaluation • Methodological problems: • Control groups in the case of field experiments not always possible • In Holland never possible in the case of mass media campaigns • As a consequence no possibility for reliable attribution of observed effects on behaviour etc. to campaign • Alternative: interrupted time series design

  34. Serious problems in evaluation (3) • Pre and post tests of materials often clumsy • Artificial: ‘what’s your opinion about this ad?’ Results in test bias • Materials should be shown in natural surrounding • Respondent shouldn’t know about which specific item he is supposed to give his opinion • Use of focus groups: group processes obstruct a clear sight on what individuals think

  35. Summarizing • Good evaluation presupposes good knowledge of the theory of influencing behaviour: • Otherwise there will be no learning effect • There’s a gap between evaluation of campaigns as it should be done and how it’s often done. If time and money is the cause: • Don’t evaluate every single campaign extensively • Instead evaluate campaigns that are typical of a kind of campaigns

  36. Questions? • Ask me now • Mail me: bert.pol@tabularasa.nl • Call me: (+31) 70 3600 949 / (+31) 65323 1764 • Visit me: • Tabula Rasa • Anna Paulownastraat 107 • 2518 Den Haag • Holland

More Related