1 / 14

Statistics Networking Day Question design Jacki Schirmer Senior Research Fellow

Learn how to design effective survey questions that produce valid, reliable, and accurate data. Avoid common pitfalls and ensure your questions align with your survey population's language, response options, literacy and numeracy levels, and sensory needs.

grenier
Download Presentation

Statistics Networking Day Question design Jacki Schirmer Senior Research Fellow

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Statistics Networking Day Question design Jacki Schirmer Senior Research Fellow Health Research Institute & Institute for Applied Ecology jacki.schirmer@canberra.edu.au

  2. Designing survey questionsWhether you are using existing measures, adapting existing measures, or designing new measures, you need to ensure your questions produce valid, reliable, accurate data

  3. Ideal survey question design process • Conceptualise • Clearly define end uses of data • Design topics that will achieve objectives • ‘Straw man’ draft • Draft initial measures – but expect them to change • Qualitative testing • Focus group with survey users • They attempt the survey and then discuss • What was confusing, hard, offensive, unclear • Where does wording need to change • You revise the questions • Expert review • Get a statistician or survey expert to review questions • Advise on response options, direction of question • Pilot test • Sample of participants complete survey under exact conditions in which it will be delivered • Analyse results from pilot sample to identify problem areas • Revise • Deliver your survey

  4. Key considerations (common pitfalls) • What language/terminology does your target survey population use? • What response option formats are best? • Literacy and numeracy level of target survey population • Eyesight, deafness prevalence

  5. Statistics Networking Day Reweighting Jacki Schirmer Senior Research Fellow Health Research Institute & Institute for Applied Ecology jacki.schirmer@canberra.edu.au

  6. Why reweight? • Your sampling technique was not based on a specific sample frame • Online surveys promoted via social media • Self-select surveys • Your sample is not representative of the population you sampled • Due to deliberate over-sampling of some groups • Due to bias in who responded to your survey

  7. Responses by region – 2014 Regional Wellbeing Survey

  8. How should you analyse representativeness? • Common approach: analyse demographics • Gender • Age • Education • Occupation • Assume that if you have similar distribution to population, survey is OK • But bias is often related to salience, not demographics • Are you biased to people with a greater interest in your topic (who may have different views to those with less interest) • Is there a general dataset you can compare to which asks questions that let you assess this?

  9. How do you tell if your sample is representative? • Explore for bias

  10. Comparison of survey response to population benchmarks • All surveys have biased samples • Ours is no exception • In 2014, we deliberately oversampled farmers, Victorians • We also have a bias to older, female respondents (common across many surveys) • Our large sample enables us to weight data so we can make statements about the population as a whole ABS GregWT procedure used to weight data, using Census data as benchmarks. This ensures we use national best practice in analysis to correct for biases in our sample

  11. If it’s not representative, how do you weight? • Simple multiplicative approach • I have 15% young people in my survey compared to 30% in the population. Every young person will be assigned a weight of ‘2’ to ensure they are representative • OK if weighting by simple demographics and not many of them • Problematic when you weight by multiple attributes, as requires all cells to be populated, and simple approach does not represent interactions well • Regression and other statistical modelling approaches • Use regression modelling to assign weighting to each respondent • E.g. GREGWT used by ABS • See Tanton et al. (2014) discussion http://microsimulation.org/IJM/V7_1/4_IJM_7_1_Tanton_Williamson_Harding.pdf

  12. Challenges • Cell sizes – when they become small, there is greater potential to ‘weight the bias’ and get an extreme result • Number of benchmark variables used

  13. Scale (population) weights vs proportional weights • Scale weights scale to population size • Proportional weights preserve your n • Following examples are from file://stafffiles.win.canberra.edu.au/Homes$/s427944/Windows%20Profile/Desktop/Creating%20Weights%20to%20Improve%20Estimates.pdf

  14. Problems with weighting • Problematic to calculate standard error, confidence intervals for weighted data • Other statistical tests may be non-robust • Some programs assign population n, rather than sample n, to weighted data – with resulting problems with statistical analysis (SPSS in particular) • Weighting does not correct all types of survey error • Weighting is only as good as your benchmarks – pick the wrong benchmarks and it has limited utility • Non-response effects – weighting does not correct for these

More Related