Einstein on Evaluation: Spencer on Stats

1. Einstein on Evaluation: Spencer on Stats 2nd November 2011

2. “Not everything that can be counted counts, and not everything that counts can be counted”

3. “Not everything that can be counted counts ...”

4. What can be counted? Placement Quote Clip/story counts Photo Audience impressions Message Tone / sentiment Cost per impression Advertising Value Equivalent (AVE) Weighted Media Costs (WMC)

5. What can be counted? Who thinks what? How many think x? Opinions? How many do x? How many have seen x?

6. So, what counts? What counts is success

7. What is success? Success Pronunciation: / sək’sɛs / noun [mass noun] 1 the accomplishment of an aim or purpose: the mission had some success in restoring confidence in the country 2archaic the good or bad outcome of an undertaking: the good or ill success of their peacekeeping mission Origin: mid 16th century: from Latin successus, from the verb succedere 'come close after' (see succeed)

8. And how can we count success? By using smart objectives S pecific Measurable A chievable Realistic Time

9. The ladder of success Did an attitude or behaviour change? Did the message resonate and stick? Did the audience actually see it? Did the target audience have the opportunity to see it? Was the message placed in the right medium? Was the messages changed or distorted? Did the message appear? Was the message compelling to the medium or intermediary?

10. The laddering approach to measurement Measures of effectiveness (MoE) Measures of performance (MoP)

11. Measures, metrics & methods Measures Metrics Methods Qual. & Quant. research Attitudes & behaviours Pre/post surveys Outcome Pre-testing/ evaluation Message pick-up/take out Out-take Events/ ‘contacts’ Audience penetration (direct) Media analysis (readership/listenership/viewership) Reach of audience (indirect) Output Media evaluation (traditional & new social media) Message placement Media coverage/ interviews Media uptake

12. Methods – Primary Research

13. Quantitative approaches Data collection methods: • Face to Face research • Telephone research • Online research • Postal research (rare)

14. Face to Face research • Perfect for:- • Subject sensitive studies • Where other methods (phone, emails) impractical • Pen & paper or CAPI* where available • Street (max 5 mins.) or in home (20 to 40 mins.) Benefits: Very thorough, can interview people in situ and comfort of their own home, rich information Drawbacks: Expensive fieldwork, time consuming, some areas are “no go” areas *Note -Computer Aided Personal Interviewing

15. Telephone research • Ideal for geographically dispersed populations • Pre-provided telephone numbers or “direct dial” • Trained interviewers calling from a telephone facility • 10-30 mins is the norm Benefits: Potential to reach broad population Drawbacks: Expensive NB mobile phone interviewing.

16. Online research • Fastest growing data collection tool • But only suitable where internet penetration high • DIY option • Using survey software • Outsource data processing to DP House • Write own questionnaire, script and web-host • Online Panel option Benefits: inexpensive, quick & easy & good for showing stimulus material Drawbacks: not suitable for hard to reach or low literacy populations

17. Designing sample Sample Size Answers accurate + or - %* (using a statistical measure of 95% confidence limit) 400 4.9% 500 4.6% 600 4.0% 800 3.5% 1000 3.1% 1600 2.3% * For example if your base size is 1600, and the results show that 50% of the sample have voted in the recent election, you can be confident that 47.7%-52.3% of the total population have voted. Samples • Consider the population and the different subgroups you will need to analyse • dictates overall sample size and design. • Random sampling techniques produce representative results • Census data is a challenge Why is it important to consider base sizes? • The larger the base size the more confidence we have in the results being a true reflection of the study results • We cannot act on the results if the sample size is not robust and the results therefore reliable

18. Principles of good research practice (applies to all methods) • Pilot where you can – test out the survey if at all possible with the appropriate population • Check the flow, the language, the layout, do not ask for feats of memory from the respondent • Make the questionnaire as inviting to complete and as interesting as possible • But do not make it so complex it confuses them • Include some open ended questions if possible so people can express their own views in their own way • Give time for analysis at the end – it is the output that is important not the research process in itself

19. Recommendations • Face-to-Face methodologies should be considered first • Do not rely upon government statistics; NGOs often have good reports on social movements and situations • Contact NGOs who have completed research in the area. Find out about the work they have done and with who • Be aware of social mores when assigning interviewers • Timelines must be flexible, and projects on a best efforts basis • Monitor political and social situation closely.

20. Qualitative approaches Data collection methods: • Focus Groups • Mini groups of 4 to 5 (or full groups of 6 to 9 attending) • In-depth interviews • Face to face or telephone

21. In depth interviews work when: People cannot travel easily and it is easier for you to go to them (disability, ill health, lack of mobility, suspicious, too busy..) People you want to speak to are geographically dispersed The subject is highly sensitive and people would not want to discuss openly with others there There is time for the executives to do individual phoning or to travel to interviews Advantages: Much richness in the encounters; puts most onus on the researcher; reaches “difficult to reach” groups Disadvantages:Time consuming, costly, more difficult to cover a wider sample Focus Groups work when: People can travel to the groups and are prepared to do so The subject is one that people are happy to discuss in the open and share their thoughts with others There is enough sample to cluster people so that you can run a group (ie not too geographically dispersed) Advantages: Richness in the exchange and thrust and parry of the interchange; takes people out of their usual environment; more effective in terms of researcher time, more cost effective to reach a wider sample (than depths) Disadvantages:Logistics (viewing facilities etc) can be costly; people do not always turn up; analysis more intense & what people says more difficult to disaggregate Qualitative Approaches – some rules of thumb

22. Methods – Secondary Research

23. Media evaluation What counts? The principles of effective communication count... Reach of audience: did you reach them sufficient times for message to stick? Delivery of message: did the message even appear, and how did it appear (credibly, negatively?)? AVE/WMC/OTS/GIs – what purpose do they serve?

24. Monitoring the online environment Uses & Benefits:- • Proxy for public opinions • Will tell you how people feel about issues and how online discussion is enhancing or undermining a position • Identify how current events shape content and discourse about organisations or social issues in online spaces. Considerations:- It provides a deep dive into online discourse that is taking place but is not intended to provide a representative study of people living in a certain community Supplements, rather than replaces, other sources of information that are available to the organisation, whether gathered via primary research studies or informally from stakeholders and ‘allies’ It is a channel for communications and like any comms. needs to be measured!

25. What counts is the message Around half of the content analysed had a key strategic message embedded in it – primarily support for pro-democracy initiatives Base: 820 items of Online Content (Jan 2008 to Oct 2009)

26. Methodology • Content analysed sourced from: • Websites and online medias • Blogs and microblogging sites • Forums/ Discussion groups • Social Network sites • File sharing sites (e.g. YouTube and Flickr) • Wikis/ Answer sites/ Podcasts • Software tools used to uncover content posted: • Alterian SM2 • Free search tools e.g. Google

27. Deciding what counts Rear view Forward looking Stop doing Start doing Keep doing How did the campaign perform in terms of:- • Media • Return How effective was the campaign in terms of:- • Messaging • Attitude/behavioural shift

28. Decisions that count Shoot the messenger? Shoot the message?

29. “Not everything that counts is counted...”

30. Accounting for flat-lining of measures

31. And accounting for casuality issues

32. How they already act & feel about voting Attitude Behaviour

33. Combined effect of communication - vote at some elections Q14: Please indicate how you feel about voting in terms of the following aspects.. Setting an example for younger people (very important, important, not very important, not at all important)

34. Quantifying campaign effect

35. Calculating cost benefit

36. When you can’t count, important to triangulate the data Primary survey data Ethnographics Secondary (media) data

37. Thank you and questionsClaire Spencer FCIPR, i to i research limited DD: +44 (0) 203 178 2162Mob: +44 (0) 7786 543 506email: claire.spencer@itoiresearch.com@claireyscherubs