320 likes | 331 Views
Managing Multi Mode Collection Instruments in the 2011 UK Census. Frank Nolan, Heather Wagstaff, Ruth Wallis Office for National Statistics UK. Contents. What I plan to cover Electronic Questionnaires Challenges Outcomes. Context. Traditional Census High Profile Involves ALL citizens
E N D
Managing Multi Mode Collection Instruments in the 2011 UK Census Frank Nolan, Heather Wagstaff, Ruth Wallis Office for National Statistics UK
Contents What I plan to cover • Electronic Questionnaires • Challenges • Outcomes IAOS2008
Context • Traditional Census • High Profile • Involves ALL citizens • Enumerators – face-to-face • Increased Diversity • Lower response rates • Modern Government / NSI - eforms • Options for Multi Mode IAOS2008
1. Electronic Questionnaires IAOS2008
Cost / Benefits overview • Intuitively we might think that there were both costs and benefits • Two dimensions • Respondent Burden • Quality • Graphically IAOS2008
Burden Electronic X Paper Interventions
Respondent Burden • Reductions • Burden of logging in – security, .. • Skipping not applicable questions • Increases • Repeated entry for edit failures IAOS2008
Quality Electronic X Paper Interventions
Quality • Increases • Key entry – no recognition issues • Handwriting / Within box marks / .. • Prompt for non response / …… • Reductions • Constraints on response • Repeated entry • False response (to get around edits) IAOS2008
Improved Timeliness (of response) • Internet completions can arrive prior to Census day (quality?) • More dynamic survey taking? IAOS2008
Reduction in Cost • Key entry completed by respondent • So no scanning / recognition • IF QUALITY IMPROVES • Less back office correction • Still cost of internet software set up etc. IAOS2008
2. Challenges IAOS2008
Interventions • How can we improve the electronic questionnaire – balance mode effects • Introduce interventions / edits • Similar to Interviewer controlled, but less variability (machine not human driver?) IAOS2008
Spectrum of Interventions Spectrum from • Low impact / sensible / simple / expected • ……. • ……. • High Impact / burdensome / complex IAOS2008
Spectrum of changes • Repeated data automatically entered • Skip sections not applicable (filter) • Constrain when only ONE response required • Range check • Highlight item non response • Within and between record checks • Voluntary or Mandatory check (hard / soft) IAOS2008
Model • Theoretical models developed • To be followed with field work • Current draft of questionnaire • Simplified with • Working adults (35 questions) • Children (20 questions) • Measurements • Time to complete = Burden • Non response = Quality (Inverse) IAOS2008
Model - Burden T = KS + B + L + I + C • KS - Key Strokes • B – Tick Boxes • L – Lines for write in questions • I – Instruction lines • C – Cognition levels – understanding and recall IAOS2008
Model – Non Response NR = ∑ NRq • A simple score function • NRq is the 2001 question non response rates • NRqi is applied for each intervention i IAOS2008
3. Outcomes IAOS2008
Response (Time) v Interventions IAOS2008
Time to Complete • Time reduced with (small effect) • Repeated questions • Skipping • One response (radio button) • Time Increased (larger effect) • Range check • Non response - soft edit • Non response – hard edit IAOS2008
Time to Complete • More effect for working adults than children • Resulting from more questions • Is there an effect for perceived saving in addition to the actual saving? IAOS2008
Non Response v Interventions IAOS2008
Non Response • Only small effect for first five interventions • One repeated question • Few skips • Radio button (does not effect non response) • Gains • Improve item non response • see Williams 2006 Australian Census IAOS2008
Where is the Balance? • YES to • Repeated responses • Filters to skip • Radio Buttons (one tick) • Positive for response and burden IAOS2008
Where is the Balance? • BUT • Range checks • Item non response soft check • Positive for improved response and • Negative for response burden IAOS2008
Where is the Balance? • NO to • Item non response hard check • Negative for response and burden IAOS2008
4. Conclusion IAOS2008
Drivers • Meet user needs • Multi Mode: Duality of delivering / collecting data in a range of formats • Tension between consistency and relevance • Mode Effect: Same questionnaire versus advantages from electronic questionnaires IAOS2008
Internet Offers • Significant cost advantages of high Internet take up for Census • High field and processing costs • BUT internet not universal • AND Internet respondents are not representative of population • Younger, larger household, mobile, studying and in labour force, higher income, higher qualifications, … [Williams 2007] IAOS2008
Electronic Questionnaires • Improve quality of response and save respondent costs with some checks • Net reduction in quality of response with too many check • Quality and Respondent Burden balance depends on number of applicable questions • Good quality improvements with soft edits for range and non response checks IAOS2008
Thank You IAOS2008