1 / 17

A model for Cross-national Questionnaire Design and Pretesting

This presentation discusses the state-of-the-art in cross-national questionnaire design and pretesting, with a focus on analyzing the process using the Logical Framework Approach (LFA). It explores the goals, measurable indicators, means of verification, and important assumptions in this process.

almap
Download Presentation

A model for Cross-national Questionnaire Design and Pretesting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A model for Cross-national Questionnaire Design and Pretesting NTTS Conference Diana Zavala-Rojas Rory Fitzgerald May 13-17, 2017 Brussels, Belgium

  2. Outline • State-of-the-art in cross-national questionnaire design and pretesting • Analysing the Cross-national Questionnaire Design and Pretesting process in the ESS using the Logical Framework Approach (LFA), (USAID, Rosenberg L., 1969-1972) • Overall goals, purpose & outputs • Measurable indicators • Means of verification • Important assumptions • Conclusions and Q&A NTTS Conference | 16.03.17

  3. Cross-national questionnaire design and pretesting • Goals • Minimizing instrument and respondent components of total survey error (Smith 2011) • Producing questions that result in equivalent measures.(Jowell et al. 2007) • Know-how • Ask the Same Question (ASQ) and Translate approach vs. adaptation (Harkness 2003) • Sequential, parallel or simultaneous design of source and target versions • Questionnaire design as an iterative process • Conceptual and measurement validity (Billiet 2016) NTTS Conference | 16.03.17

  4. Thechallenges of crossnationalquestionnairedesign in the ESS Thesourcequestionnaireshouldwork in the British context and at thesame time, aftertranslation, it, shoudworkequallywell in 25+ languages NTTS Conference | 16.03.17

  5. ESS questionnaire • Core module: 128 questions • Human values scale: 21 questions • Experimental questions: 6 • Two rotating modules: 30 questions each Europeans and Democracy | European Parliament | 04.03.15

  6. Cross-national questionnaire design and pretesting • Where are we? • State-of-the-art literature includes comprehensive inventory of options and current practices to design comparative survey questions. • Guidelines describe approaches used in a variety of social, political, public health, public policy and psychological survey research • Useful as an inventory for potential strategies • What is missing? • An analytical evaluation of the approaches listed • A blueprint of how to best organise the questionnaire design and pre-testing process overall, considering its iterative nature NTTS Conference | 16.03.17

  7. In this chapter we…. • Introduce the European Social Survey (ESS) Questionnaire Design and pretesting model • Evaluate the extent to which the 17 stage model ensures the effective design of the source instrument • Use the Logical Framework Approach (LFA) methodology to analyse the components of the process in a clear, concise, logical and systematic way. • Overall goals, purpose & outputs • Specific goals, purpose & outputs • Measurable indicators • Means of verification • Important assumptions NTTS Conference | 16.03.17

  8. ESS Model for Cross-national Questionnaire Design and Pretesting 1 2 3 4 5 6 Proposalsfrom QDT Expert review of questions Use of Survey Quality Predictor Program Revised proposals from QDT and CST Consultation with ESS National Coordinators Steps 1, 2 and 4 repeated . Omnibus Testing & Cognitive Interviewing 12 11 10 9 8 7 Expert review & revised proposals from QDT and CST Pilot analysis & results of advance translation Large-scale two-nation quantitative pilot & Advance Translation Split ballot MTMM experiments developed Revised proposals from QDT and CST Omnibus Analysis, CI Analysis 13 14 15 16 17 Consultation with ESS National Coordinators Final source Q’aire produced Translation & Translation Verification SQP coding Country pre-tests Questionnaire ‘alerts’ (onlywhennecessary) Mainstagefieldwork NTTS Conference | 16.03.17

  9. How to evaluate the questionnaire design and pretesting in the ESS using the LFA? • General goals • Minimizing instrument and respondent components of total survey error • Producing questions that result in equivalent measures. • General purpose • Producing an effective questionnaire allowing comparable measurement across countries • Overall outputs • Theoretical, conceptual and items specification finalised in template and updated when necessary • Questions finalised for cognitive interviewing, omnibus survey(s), pilot study • Optimal source questionnaire released NTTS Conference | 16.03.17

  10. Are we producing an effective questionnaire? • The overall goals can be verified with respect to the overall purpose after questionnaire design, once data is collected, by defining measurable indicators and means of verification NTTS Conference | 16.03.17

  11. Are we producing an effective questionnaire? • Specific goals and outputs are assessed with respect to the overall purpose (how they contribute to design better questions?) • Defining measurable indicators and means of verification NTTS Conference | 16.03.17

  12. An example: draft item in Round 6 • “How difficult or easy do you think it is for immigrants to get the right to vote in national elections in [country]? • 0 = far too difficult; 10 = far too easy” • Concept specification: • Evaluation: measure how inclusive the political system is: whether it opens immigrants possibilities to participate in politics, facilitate their integration in the system and broadentheirrights. • Goals • Minimizing instrument and respondent components of total survey error • Producing questions that result in equivalent measures. • General purpose • Producing an effective questionnaire allowing comparable measurement across countries NTTS Conference | 16.03.17

  13. An example: drafting an item in Round 6 • “How difficult or easy do you think it is for immigrants to get the right to vote in national elections in [country]? (0 = far too difficult; 10 = far too easy)” • Expert review pointed out this was an issue which respondents in general simply did not know about. • Omnibus testing showed high item non-response • Pilot study showed a substantial proportion of ‘don’t know’ or mid-point response in both the UK and Russia. • Cognitive interviewing. Some respondents interpreted this as a ‘knowledge’ question: lack of knowledge about the relevant legal situation for immigrants or lack of relevant experience, they were not immigrants, they could not judge the difficulty of getting the ‘right to vote. • Decision after quantitative and qualitative evidence suggested that there was a ‘source question problem’ (CNEST) (Fitzgerald et al, 2011). NTTS Conference | 16.03.17

  14. Important assumptions • The success of the questionnaire design process relies upon important assumptions • Assessmentmethods are correcttoolsforassessingmeasurementquality and equivalence • Resources are availableforpretesting and scientistsparticipating in theprocess • Nationalteamsengagefully in theprocess • Triangulation of findingsisperformedrigorously • QDT and CST seekchanges and improvementsderivedfromevidence NTTS Conference | 16.03.17

  15. Conclusions • The ESS Cross-national Questionnaire Design and Pretesting model provides a structured, multi-layered and culturally sensitive framework for cross-national projects • It produces a questionnaire that minimises instrument and respondent components of TSE and yields equivalent measures • Core elements are specification, documentation, multi-disciplinary input, qualitative and quantitative testing, advance translation and detailed data analysis • Questions are developed by triangulating findings from an array of qualitative and quantitative methods • Rigorous environment for developing a source questionnaire translatable into target languages across Europe NTTS Conference | 16.03.17

  16. Conclusions • Current challenges include: • The long time period for development (24 months) • Limited pretesting of the target questionnaires • Largeresourcesrequired to fundthework NTTS Conference | 16.03.17

  17. R.Fitzgerald@city.ac.uk, diana.zavala@upf.edu , www.europeansocialsurvey.org @ESS_Survey • Fitzgerald, R., & Zavala-Rojas, D., (forthcoming), “A model for Cross-national Questionnaire Design and Pretesting” in Paul Beatty, Amanda Wilmot & Debbie Collins (eds.) Questionnaire Design Evaluation and Testing, Wiley & Sons, Inc.

More Related