Evaluating behaviour change programs

1. Evaluating behaviour change programs Liz Ampt Concepts of Change

2. What are we measuring? • Whether people are doing things differently WasteMinz Roundup 2014was WasteMINZ Roundup 2014

3. How can we measure it? Measuring changes in levels of reducing/ reusing/ recycling Observing/recording behaviours Asking about change Each is a form of survey or data collection exercise

4. The survey process Richardson, Ampt, Meyburg 1995

5. Each is a form of survey

6. Key elements of survey design Preliminary planning Selection of method Sample design Survey instrument design Pilot Survey implementation Expansion/weighting Analysis – over to you..

7. 1. Preliminary planning • Define survey objectives • Very specific: what, by whom, over what period, where • Review of existing information • Useful methodologies from elsewhere; use of stated preference? • Define terms • From your objectives and from respondent’s perspective • Survey content • Dot points

8. Exercise • Think of 2 terms you would want to use in a survey related to your work • Write down clear definitions for both • Ask another person (not from your organisation) to do the same with your terms • Compare

9. 2. Selection of a Survey Method for Measurement • Observation surveys • Intercept surveys • Self-administered surveys • Telephone surveys • Personal interview surveys • Internet/online surveys

10. Observation methods • Chosen when • Possible to count accurately • Possible to count all or select a representative sample • Can be manual, automatic, video

11. Intercept Surveys • Intercepting people • At an activity centre (e.g. workplace, transfer station, shopping centre) • Possible methods • distribution - mail-back/on-line • personal interview • collect phone no. • Always needs a total classification count

12. Intercept Surveys • Advantages • Able to reach specific populations • Can combine with observational counts • Can use multiple survey methods • Disadvantages • Generally low response rates (20-30% for self-completion) • Hurried conditions • Must allow for non-random sampling • No follow-up possible in most cases

13. Self-administered Surveys • Possible targets • households • activity centres/workplaces/transfer stations • Method of Distribution • mail-out vs. hand delivered • Method of Collection • mail-back vs. hand collection

14. Self-administered Surveys • Advantages • Can get extensive geographical coverage • No interviewer effects • Can obtain considered responses • Hand-collection to good response rate • Disadvantages • Layout and wording must be clear - hard to design • No probing possible • Answers not independent • Response rates lower than face to face

15. Telephone Interviews • Advantages • Wide geographic coverage • Intermediate costs • Good supervision - CATI • Multilingual capabilities • Computerised

16. Telephone Interviews • Disadvantages • Sample usually weak • Low phone ownership for some groups • Answer-phones, mobile phones, screening devices • Hard to know how it represents the population • Credibility of interviewer (confusion with telemarketing) • Low response rate • No follow-up for refusals

17. Personal Interviews • Can be paper or computer • Advantages • Generally higher response rates (60-90%) • Flexibility of information • Presence of interviewer • Maintain interest • Spontaneous answers

18. Personal Interview • Disadvantages • High costs • Interviewer influence • personal characteristics • interrupt household/work routine • opinions of interviewers • interpretation of vague answers • Considered response difficult

19. On-line Surveys • Advantages • Low costs • Can use elaborate visual effects • Can use adaptive techniques (can give different scenarios for different responses) • Good for workplaces if sufficient follow-up • Disadvantages • Usually very biased sample • Low response rate • Hard to get all people in household if needed

20. Exercise • Think of a behaviour you would like to measure • Discuss with a partner • Best method of collection • Strengths • Weaknesses

21. 3. Sample Design in theSurvey Process Sampling Methods

22. Preliminary Concepts Sampling Methods • What is a sample? • a collection of things which is some part of a larger population and which is selected so as to be representative of some or all of that population • Target Population • who are we trying to survey? • Sampling Units • what are we going to sample? • Sampling Frame • where are we going to get a list of these things?

23. Sampling Frame Sampling Methods • a base list to identify the sampling units • should contain all the sampling units • examples, • all households on a street (e.g. Council records) • telephone directories • mailing lists • maps • electoral rolls • blocklists

24. Sampling Frame Problems Sampling Methods • inaccuracy • incompleteness • duplication • inadequacy • out-of-date • Must check the reason for which the list was originally compiled to understand likely deficiencies.

25. Sampling Error & Sampling Bias Sampling Methods • Sampling Error • due to the simple fact that we are taking a sample, and not the population. • can minimise error by taking larger sample. • Sampling Bias • due to systematic omission of some elements from our final sample. • cannot minimise error by taking larger sample.

26. Random Sampling Sampling Methods • Each unit is selected independentlyand each unit in the population has an equal probability of being selected. • Must use random sampling to avoid sampling bias.

27. Random Sampling Methods Sampling Methods • Simple Random Sampling • Stratified Random Sampling • Variable Fraction Stratified Random Sampling • Multi-stage Sampling • Cluster Sampling • Systematic Sampling • Note quotas not on list

28. SampleSize Sampling Methods • How much data do we need? • Too much data >>> too expensive • Not enough data >>> not able to draw conclusions • Somewhere in the middle is a sample size which enables us to draw sufficient conclusions at a reasonable cost Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.

29. Exercise • Hand out random sampling sheet • Explain • Questions?

30. 4. Survey design in the survey process

31. Instrument design for reliable measurement • Question content • Question types • Physical design - also for observation/counting

32. Question Content • Reliability • repeatable • easy to answer • Accuracy • no question bias • measures what we want • Relevance • must appear relevant to respondent

33. Question Types • Factual • “What did you do?” • Classification (e.g. socio-demographic) • for comparing with secondary data • Opinion and attitudequestions • “What do you think about ……?” • Stated Response Questions • “What would you do if ……?”

34. Physical Design of Forms/Apps • Observational surveys • Ergonomic • Size/format – not too big or small • Weather-proof • Need log forms • Test under actual conditions

35. Physical Design of Forms • Self-administered forms • Layout vital • Minimal writing should be required • No coding aids should appear • Instructions very clear • Professional appearance • Include ID number

36. 5. Pilots in the survey process

37. Pilot Surveys • Why not do a pilot survey? • too expensive • not enough time • Why do a pilot survey? • too expensive to omit it • not enough time to omit it

38. Pilot Surveys • pilot survey is a test of ALL aspects of design • scope for experimental design • saves expensive mistakes

39. Uses of the Pilot Survey • "Skirmishing" of wording • Adequacy of questionnaire • definitions clear? • too many "don't knows“? • too long? • open to closed questions • Efficiency of interview/surveyer training • Non-response rate • Analysis • Cost and duration

40. 6. Survey implementation

41. Conducting the Surveys • Need high response rate for validity • Consider • Announcement letter/message >> higher response • Follow-up regime >> higher response rate

42. Exercise – dot points for a survey you need • What method? • How to get a sampling frame? • What questions? Need any for weighting? • Other issues/questions? • Richardson, Ampt, Meyburg (1995) • http://www.geog.ucsb.edu/~deutsch/geog111_211a/code_books/Survey_Methods_For_Transport_Planning.pdf

43. 7. Weighting (correction)/Expansion

44. Weighting & Expansion of Data • Getting the sample data to represent the population from which it was drawn, as nearly as possible • Why? – systematic errors • Non-response >> weighting certain type of respondents higher • Missing Data >> can make assumptions – or note • Inaccurate Reporting >> e.g. social desirability bias

45. Example of weighting • Your response is 50/95 – what about the 45? Say 30 males (67%) 15 females (33%) • Your secondary data (e.g. counts, other data) males 50% females (50%) • missing responses from females • Responding females are therefore ‘weighted’ with a slightly higher value (1.5) males (.75) Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.

46. Summary • To measure behaviour change it is essential to understand the data collection and survey process • In particular need to understand: • Survey method • Sampling principles • Importance of a pilot • Implementation options • Need for weighting • Vital for future funding as well as for sharing methodologies

47. lizampt@conceptsofchange.com.au