SURVEYs AS A METHOD OF DATA COLLECTION

SURVEYs AS A METHOD OF DATA COLLECTION This file contains hyperlinks to take you to the various linked sections BEGIN

WHAT WILL BE COVERED Outline: • Introduction: - What are surveys and when would you want to use them? • Examples of real surveys: - Some common Australian surveys • Conducting surveys: - How do you design and carry out a survey? • Developing survey questions: - Eliciting useful data • Sampling: - How do you sample the population of interest? • Errors and bias: - Making sure your data is good quality • Considerations collecting data: - Some key things to be aware of • Analysis and reporting: - Processing and presenting your data • Further reading: - Other good sources of information NEXT

1. INTRODUCTION:What are surveys and when would you want to use them? What are surveys? Surveys are a way of gathering primary data from a human population of interest (e.g. individuals, households, farms, businesses) by asking a standard set of questions to a sample of the population “Surveys provide a means of measuring a population’s characteristics, self-reported and observed behaviour, awareness of programs, attitudes or opinions, and needs. Repeating surveys at regular intervals can assist in the measurement of changes over time. These types of information are invaluable in planning and evaluating government policies and programs.” (http://www.qgso.qld.gov.au/about-statistics/survey-methods/) Hence key issues involved include: • Who is the population and how can you obtain the data you need? • Asking clear and specific questions, • Robust sampling methods to enable statistically strong study conclusions. ... We will explore these types of issues in the presentation. NEXT Return to contents

1. INTRODUCTION:What are surveys and when would you want to use them? What are surveys? Difference between a survey and a census: • A survey samples a subset of a population of interest, whereas • A census seeks to sample the entire population of interest For a probability survey, the survey sample is chosen objectively so that it statistically represents the wider population (i.e. no bias) – “each member of the population will have a known non-zero chance of selection” (http://www.qgso.qld.gov.au/about-statistics/survey-methods/) ... This then allows you to generalise the survey findings to the wider population (the ultimate goal) NEXT Good sample Poor sample Total population Return to contents

1. INTRODUCTION:What are surveys and when would you want to use them? Purpose of surveys There are many possible purposes of surveys, including: • Collecting baseline data, or following-up for a time series, • Comparing two groups (these can be such things as clinical trials), • Evaluating the effect of interventions (eg education, policies or programs), • Identifying needs for funding or resource allocation, or • Tracking of population characteristics in society (e.g. employment, awareness of topics, internet access, etc). NEXT Return to contents

1. INTRODUCTION:What are surveys and when would you want to use them? When to use a survey? Surveys are resource and time-intensive (including design, identifying and distributing to potential respondents, effort for respondents to complete survey, time to wait for responses). Hence need to consider whether a survey is the best strategy for gathering the data you need e.g. does relevant data already exist in other published reports, on the ABS website, etc? However, often other people’s data may not meet your specific research needs, so a survey may be required. ... If so, then there’s severalthings you need to consider at the outset. NEXT Return to contents

1. INTRODUCTION:What are surveys and when would you want to use them? Things to consider for doing a survey*: Practicality • Can the information be collected cost effectively and accurately via a survey? • How complex and how sensitive is the topic? • Do respondents have access to the required information? Will they be willing to supply it? • Will their responses to the questions be valid? Timing • When is the information required? • Is enough time available to ensure that data of sufficient quality can be collected and analysed? • When is the best time to conduct the survey? (eg growing seasons, holiday periods). Survey requirements • What will the survey findings be used for (eg to target program improvements)? Ensure to collect data that will enable you to report with sufficient statistical accuracy, and focus on sub-groups of interest (eg geographic areas, age groups, sex, industry, size of business) *Ref: http://www.qgso.qld.gov.au/about-statistics/survey-methods/ NEXT Return to contents

1. INTRODUCTION:What are surveys and when would you want to use them? Things to consider for doing a survey*: Accuracy • What level of error is acceptable? (depends what survey results will be used for) Frequency • Will the survey be repeated? How often? Resources • Are the necessary financial, staff, computer or other resources available? Legislative powers • Does your organisation (eg university, government department/agency) have authority to collect the information through a survey? This is linked to the final issue: Ethical considerations • All research involving human subjects must follow strict ethics procedures (explained more later in the presentation). *Ref: http://www.qgso.qld.gov.au/about-statistics/survey-methods/ NEXT Return to contents

2. Examples of real surveys:Some common Australian surveys Monthly population survey - Australian Bureau of Statistics (ABS): This surveys 35,000 households every month on issues including employment, education, the environment, conditions of employment, and child care arrangements. The main component is the Labour Force Survey, which is where the national monthly employment figures come from! Note that this is a survey of households NEXT See: http://www.abs.gov.au/websitedbs/d3310114.nsf/home/current+household+surveys Return to contents

2. Examples of real surveys:Some common Australian surveys Business surveys - Australian Bureau of Statistics (ABS): Cover all manner of topics Note that this is a survey of businesses See: http://www.abs.gov.au/websitedbs/d3310114.nsf/home/survey+participant+information NEXT Return to contents

2. Examples of real surveys:Some common Australian surveys Business surveys - Australian Bureau of Statistics (ABS): ... Zooming in on agriculture-related surveys: NEXT See: http://www.abs.gov.au/websitedbs/d3310114.nsf/home/survey+participant+information Return to contents

3. Conducting surveys:How do you design and carry out a survey? Steps in the survey process: 5. Data analysis and reporting 1. Design and plan survey 2. Design questions 3. Pilot test and adjust survey 4. Conduct survey Repeat steps if needed NEXT Adapted from: http://www.qgso.qld.gov.au/about-statistics/survey-methods/ Return to contents

3. Conducting surveys:How do you design and carry out a survey? Step 1. Design and plan survey: Key things to do at the first step: • Define purpose and objectives of study • Eg to describe characteristics of a population (individuals, households, businesses, ...), to compare different groups (eg the target group for a program vs a control group), or to gain feedback from a group of people about their views • Identify the output required (eg what types of data, charts/tables do you want to produce at the end?) • Define target population and sampling strategy • Decide data collection method (eg paper, online, telephone or in-person interview) • Develop survey procedure • Obtain ethics approvals (from appropriate university or government department ethics committee) NEXT Return to flowchart Return to contents

3. Conducting surveys:How do you design and carry out a survey? Step 1. Design and plan survey: Target population: The target population needs to be defined carefully and explicitly eg what demographic characteristics define the target population? In a rural population, what is meant by ‘rural’? What is the geographical area of interest? Data collection methods: http://www.qgso.qld.gov.au/about-statistics/survey-methods NEXT Return to flowchart Return to contents

3. Conducting surveys:How do you design and carry out a survey? Step 2. Design questions: Key considerations: • Survey questions need to measure things accurately, and it takes a lot of work and careful design to write questions that can do this – this is not trivial! • For example, questions need to be clear to all potential respondents, not ‘leading’ questions (i.e. worded in a way that may bias the response), and allow for robust data to be collected • Questions MUST be designed in such a way that they have reliability and validity ... Question design is covered in more detail in Section 4 NEXT Return to flowchart Return to contents

3. Conducting surveys:How do you design and carry out a survey? Step 3. Pilot test and adjust survey: What needs to be done: • Pilot test the entire survey with a small number of respondents from the target population (i.e. ‘real’ respondents, although you may not be able to actually use this data) • You want to check how respondents find the questions and the survey as a whole • eg are their any problems with the layout or structure? Do the questions all make sense? Is the length of the survey manageable? • Find out afterwards what the pilot respondents though of the survey • Modify survey as appropriate (eg procedures, questions, structure), and repeat the first step of design and planning if needed – this is not a bad thing, it’s extremely positive because you will get better data and have more robust findings! NEXT Return to flowchart Return to contents

3. Conducting surveys:How do you design and carry out a survey? Step 4. Conduct survey: What’s involved: • Finalise survey and procedures to be conducted in the field • Finalise sampling strategy and ensure recruitment strategy clearly identified (e.g. contact lists, places survey will be advertised) • Train interviewers (if survey to be conducted in person using research assistants) • Distribute survey through data collection methods decided in the first step of the process (eg mail out, online, in person) • Wait for results to arrive, meanwhile monitor process – eg are there any problems in the survey completion? Have any respondents contacted you with questions or concerns either before or after completing the survey? How will their concerns be addressed – both in terms of the data collection process as well as in terms of any ethics issues raised? NEXT Return to flowchart Return to contents

3. Conducting surveys:How do you design and carry out a survey? Step 5. Data analysis and reporting: How do you go about this? • Prepare tables and files to enter data into (eg MS Excel spreadsheets) • Code and edit data (eg fill gaps, check data quality, convert qualitative responses to numerical scales if required) • Analyse data eg count responses, calculate statistics (eg %’s, averages, standard errors) • Prepare tables and charts to present data • Prepare reports and/or papers to report on survey results • Evaluate and reflect on the survey process – What worked well? What didn’t work so well? What would you do differently next time? Can your survey be used again in the future (by yourself or others) eg to develop a time series of the same parameters? NEXT Return to flowchart Return to contents

4. Developing survey questions:Eliciting useful data Why does this matter? A survey is used to measure something, just like if you were using a thermometer or scales in an experiment in a lab – so it needs to be accurate otherwise you’ll get poor readings and your data won’t make sense! More specifically, we need to make sure our survey has reliability and validity • Reliability means that you get the same result if you repeatedly measure the same property – i.e. that a survey question means the same thing to all respondents. • Validity means that the survey question accurately measures what it intends to measure Both reliability and validity are essential – a classic example of why this is important is the case of faulty bathroom scales. The scales might consistently read 70kg (it is reliable) but they may be badly calibrated, and your real weight is 60kg (hence the reading is not valid). NEXT Return to contents

4. Developing survey questions:Eliciting useful data How do you develop good questions? This is a skill and takes a lot of time to develop, edit and refine the questions Some general tips are: • Focus clearly on the goal of the survey – critically reflect on how each question will help you address that goal • Avoid asking unnecessary questions – a good technique is to draft the charts and tables that you want to make from the data to help clarify which questions are most important and which ones are not • See if you can find previous surveys as examples to help you build yours – this can be very useful to build on and to get a sense of how to write good questions • Try to maintain a parallel structure for all questions – this significantly improves the ability of respondents to follow and effectively respond to the survey. NEXT Return to contents

4. Developing survey questions:Eliciting useful data Practical tips: Some practical tips are: • Avoid leading questions – i.e. ensure it doesn’t seem like you’re expecting a certain response, but give a full range of options for respondents. For example, instead of asking respondents to agree or disagree with a statement, provide a full range of possible responses such as: not at all, a little, somewhat, quite a bit, extremely • Keep questions simple • Avoid questions that should be two questions • Avoid using complex words, technical terms, acronyms or jargon. Instead, use language that is commonly used by the respondents. • If you ask about a past experience then give a time boundary / reference period. Also, note that recent recall will be sharper eg ‘Have you been to the movies in the last month?’ Rather than: ‘How many times did you go to the movies in the last year?’ oqi.wisc.edu/resourcelibrary/uploads/resources/Survey_Guide.pdf NEXT Return to contents

4. Developing survey questions:Eliciting useful data Practical tips: More practical tips: • Avoid using complex words, technical terms, jargon, and phrases that are difficult to understand. Instead, use language that is commonly used by the respondents. • For example*: NEXT *Sourced from: oqi.wisc.edu/resourcelibrary/uploads/resources/Survey_Guide.pdf Return to contents

4. Developing survey questions:Eliciting useful data Response format: Questions may be open-ended in allowing respondents to answer a question in their own words • This has advantages such as: of gaining richer answers and insights into the respondents views, and the possibility of unanticipated answers • However, a disadvantage is that it is more difficult to interpret and analyse this data Questions may also be closed, in that they provide a pre-defined set of categories and the respondent chooses the answer that best represents their view (egvery dissatisfied, somewhat dissatisfied, neither satisfied nor dissatisfied, somewhat satisfied, very satisfied) • This has advantages that: respondents may find it easier and less time-consuming to choose a response on a scale, and it is more straightforward to analyse this type of data • However, a disadvantage is that it confines the range of responses to pre-defined categories which must be chosen very carefully to be appropriate and intuitive to respondents NEXT Return to contents

5. SAMPLING:How do you sample the population of interest? Definitions: • Target population: The group of people or organisations whose activities, beliefs or attitudes are being studied • Sample: The subset of the target population that forms the set of respondents to the survey • Sampling frame: This is essentially the boundaries that you draw around the target population to define who the possible set of sample subjects may be i.e. By what criteria do you ‘bound’ the population? For example, this could constitute a telephone directory, or a list of business or organisations or households. Need to consider: is this list available, and how current is it? MUST avoid sending to deceased people or non-existent organisations (ethical practice, plus will affect calculated response rate) • Respondent: An individual or organisation providing answers to survey questions NEXT Return to contents

5. SAMPLING:How do you sample the population of interest? Methods of survey sampling: This refers to the way that respondents within the sample are selected from the target population There are two broad types of sampling strategies: • Probability sampling – each member of the population has a known (non-zero) probability of being selected in the sample (e.g. choosing names randomly from a phonebook) • Non-probability sampling – the probability of each member of the population is not known and/or we do not know whether each member has a non-zero chance of being selected Probability sampling has the advantage that survey statistics for the population can be easily and reliably calculated. However, it may not be possible to build knowledge of the entire population (e.g. all businesses in the country), or it may just be prohibitive in terms of the work required to do so. Non-probability sampling has advantages of cost and convenience. However, the main drawback is that sample statistics are likely to differ from the characteristics of the actual broader population, and it is hard to know by how much. NEXT Return to contents

5. SAMPLING:How do you sample the population of interest? Probability sampling: In probability sampling, every observation in the population from which the sample is drawn has a known probability of being selected into the sample When that probability is the same for every observation in the population, the sample is an equal probability sample. These samples have certain desirable properties; for example, the simple formulas for computing means, standard deviations, and so on can be applied to estimate the respective parameters in the population. Different types of probability sampling are: • Simple random sampling • Stratified sampling • Cluster sampling • Multistage sampling • Systematic random sampling NEXT Return to contents

5. SAMPLING:How do you sample the population of interest? Types of probability sampling: Simple random sampling: • This occurs if: • For a population (N) and a sample (n), • All possible samples (n) of the population (N) are equally likely to occur • For example, a random sample from a telephone book is an example of this. Stratified sampling: • This is done when the population involves groups with differing characteristics that of relevance to the study, and you are interested in having sufficient resolution in your data to analyse survey data for different types of respondents (e.g. respondents in different geographical locations such as neighbourhoods within a city; by age; level of education; ...) • The population is divided into groups or ‘strata’, and then respondents are sampled within these strata NEXT Reference: http://www.qgso.qld.gov.au/about-statistics/survey-methods Return to contents

5. SAMPLING:How do you sample the population of interest? Types of probability sampling: Cluster sampling: • This is a technique where each member of the population is assigned to a group (a ‘cluster’), and a set of clusters are chosen, from which individuals are then surveyed. • Clusters basically represent ‘small scale’ versions of the population i.e. each cluster contains heterogeneous individuals similar to the population (although the mean of the clusters should be the same). This differs to stratified sampling where separation by strata aims to tease apart heterogeneity in the population in key ways that are likely to affect the results. • Cluster sampling can have less precision than random sampling, but it can allow you to greatly increase the sampling size for the same effort/budget. If you really need a larger sample size but don’t have the budget to do a comprehensive simple random sample for a sample that size (and you don’t mind the loss in precision) then cluster sample may be useful NEXT Reference: http://www.qgso.qld.gov.au/about-statistics/survey-methods Return to contents

5. SAMPLING:How do you sample the population of interest? Types of probability sampling: Multistage sampling: • This applies multiple stages of cluster sampling in successive stages • For example, if a large household survey is to be conducted (e.g. across QLD) then you could divide the population into districts (e.g. ABS districts) then choose a sample of those districts. Within the chosen districts, you may then sample a set of neighbourhoods. Finally, within each neighbourhood, you then sample individual households. • The advantage of this method is that it can reduce the costs of a large-scale survey, and means you only need to develop the sampling frame at each stage as required, rather than for the whole population at the outset. Systematic random sampling: • First you need to define the sample size and create a list of the entire population. You then randomly select the first sample element, and then take the kth next element on the list, and continue until you reach your desired sample size. • As long as the list does not contain hidden order, this is very much like simple random sampling, but is just a bit simpler NEXT Reference: http://www.qgso.qld.gov.au/about-statistics/survey-methods Return to contents

5. SAMPLING:How do you sample the population of interest? Non-probability sampling: Non-probability sampling is non-random sampling. It is used largely out of convenience and often either as an exploratory method or for looking at relationships between variables. The main drawbacks are that the relationships between the survey sample and the total population is not measureable and it is not possible to determine bias. Some methods are: Voluntary sample: • Respondents self-select into the survey (often people with a strong interest in the topic) • E.g. Responding to an online poll Convenience sample: • Respondents are people that are easy to reach • E.g. university students sampled for a study conducted at their university Purposive sampling: • Respondents are purposively selected because the belong to a particular category of interest to the researcher (e.g. a particular demographic) NEXT Return to contents

5. SAMPLING:How do you sample the population of interest? Sample statistics: The purpose of doing a survey is to measure some attributes of a population by calculating these values for a sample of the population. This is done by calculating sample statistics that ideally reflect (or closely approximate) the wider population. • Population parameter – A population parameter is the true value of a population attribute. • Sample statistic – A sample statistic is an estimate of a population parameter, based on the survey sample For example, you may want to find out the percentage of dairy farmers in Australia that support a new government policy on milk. You decide to do a survey of the 7,000 dairy farms in Australia. The actual percentage is a population parameter (i.e. the actual percentage of the 7,000 farms if you were to ask every one of them). The estimate of that parameter gained through a survey is a sample statistic (calculated through your survey of a smaller sample of the total population). It is clear that “the quality of a sample statistic (i.e., accuracy, precision, represent- ativeness) is strongly affected by the way that sample observations are chosen; that is, by the sampling method.” * http://www.qgso.qld.gov.au/about-statistics/survey-methods NEXT Return to contents

6. Errors and bias:Making sure your data is good quality Sources of error : Errors with surveys can be divided into two types: • Sampling error – this arises because a survey samples a subset of the population • Non-sampling error – all other errors that can occur during a survey Estimates of sampling error (e.g. standard error) can be calculated, and will vary with: • Sample size (a greater size will reduce the sampling error) • Variability in the population (sampling error will increase if the responses of interest vary greatly in the population) • Sample design (this affects whether or not standard errors can be calculated – i.e. whether or not probability sampling was done) NEXT Return to contents

6. Errors and bias:Making sure your data is good quality Non-sampling errors: These cannot be measured, but they are just as important as sampling errors: NEXT Source: http://www.qgso.qld.gov.au/about-statistics/survey-methods Return to contents

6. Errors and bias:Making sure your data is good quality Non-sampling errors (continued): These cannot be measured, but they are just as important as sampling errors: NEXT Source: http://www.qgso.qld.gov.au/about-statistics/survey-methods Return to contents

6. Errors and bias:Making sure your data is good quality Sources of bias: Non-response to survey: • Non-response occurs in almost all surveys, and may be due to factors such as refusal, non-contact or language difficulties • It is a source of bias if the characteristics of non-respondents differ to respondents • It is good practice to try to minimise non-responses if possible (e.g. making call-backs to non-contacts or following up on refusals) • Response rate is expressed as a % Sampling frames: • Bias can be introduced through imperfect sampling frames (which is common) • E.g. telephone listings exclude those with ‘silent’ numbers, or households without landlines which may lead to under coverage of younger demographics NEXT Return to contents

6. Errors and bias:Making sure your data is good quality Sources of bias: Measurement error: Poor measurement process can lead to bias and error e.g. if the survey is conducted in an unsuitable environment, the questions are asked poorly, or the survey respondents are rushed For example: • Leading questions – this is where survey questions systematically lead respondents to answer in a certain way by unduly favouring a particular response • E.g. If the respondent is asked whether they are: ‘satisfied, dis-satisfied, or very dis-satisfied’ then this will be likely to lead respondents towards dis-satisfied responses because they are given two options for this • Social desirability – this refers to the way that people like to portray themself in a favourable light, so answers may be biased towards what is considered to be acceptable NEXT Return to contents

7. Considerations COLLECTING DATA :Some key things to be aware of Response rates: Survey response rates can be very variable, and depend on many factors such as: the topic of the survey, whether the survey is conducted in a convenient way for respondents, the demand on respondents (e.g. once-off for 20 mins vs 1 hour per session repeated every month for 6 months), and whether respondents see that the survey is worthwhile and in their interests to participate. Response rates can be improved by considering the needs of respondents carefully eg: • Contacting respondents and conducting the survey in a way that is convenient to the respondents (e.g. by telephone, online, paper, in-person as appropriate) • Not expecting too much from respondents (eg in terms of time, effort, confidentiality) • Following-up on non-responses • Explaining how the survey is relevant to the respondents and how they stand to benefit from the study, and/or what are the broader benefits to society of the study • Explaining how the survey follows ethical research procedures (eg how their data and identities will be protected, how the data will be used, who they can contact if they have any grievances or further questions) NEXT Return to contents

7. Considerations COLLECTING DATA :Some key things to be aware of Ethics considerations: Ethics approval from the relevant university or government ethics committee is required when conducting surveys This is because you are engaging with members of the public to collect data, and it is vital that the public is protected in the way that the data is collected and used – at it’s core, this is about human rights to safety, security, privacy, etc This requires preparing the survey questions and procedure for consideration by the ethics committee as to whether you meet the ethical requirements to conduct the study NEXT Return to contents

7. Considerations COLLECTING DATA :Some key things to be aware of Ethics considerations: At the beginning of the survey you need to provide some basic but essential ethics information such as: • Information on the project, why it is being conducted, who it is being conducted by, and what the results will be used for • Whether the survey is voluntary or compulsory • Information on who the respondent can contact for further information or grievances • A statement that the survey has received ethical clearance from the relevant ethics committee • A question about whether the person gives permission for use of the data in other studies, or to be re-contacted NEXT Return to contents

8. ANALYSIS AND REPORTING:Processing and presenting your data Simple analysis: Survey statistics may be relatively simple to calculate in when you finally have the data collected, but the main trick is everything up to this point – i.e. collecting data in a way that is robust and methodologically sound Descriptive statistics are often the first thing to calculate eg: • totals, means, median, proportion of households • plots of responses for a rating scale e.g. For questions with a numbered scale (such as a Likert rating scale which is widely used) or using a qualitative response scale (such as: very dis-satisfied, dis-satisfied, neutral, satisfied, very satisfied) Each statistic (mean, total, etc) should be reported with its margin of error. This can be called a ‘relative standard error’ (given as a percent) Analysis involving more than one variable: You may do more complicated statistical analysis such as: chi squared tests of independence, analysis of correlations, logistic regression, etc NEXT Return to contents

9. FURTHER READING:Other good sources of information Useful websites: http://www.qgso.qld.gov.au/about-statistics/survey-methods/index.php http://stattrek.com/statistics/data-collection-methods.aspx?Tutorial=AP http://www.nss.gov.au/nss/home.nsf/pages/NSS+Data+Quality+resources+landing+page http://betterevaluation.org/ http://www.statcan.gc.ca/edu/power-pouvoir/toc-tdm/5214718-eng.htm https://www.statpac.com/surveys/ http://www.abs.gov.au/ausstats/abs@.nsf/mf/1125.0 (Statistical skills for statisticians) Key texts: Schaeffer, N.C. and Presser, S. 2003. The science of asking questions. Annual Review of Sociology 29:65-88. NEXT Return to contents

SURVEYs AS A METHOD OF DATA COLLECTION