290 likes | 301 Views
A Panel Pilot Study for English Business Survey. Presented by Yi Zhang Department for Business, Innovation and Skills. Overview. Introduction to EBS Why, Who, How Dissemination Panel Pilot Study Pros and Cons of Panel approach Objective Methodology Timescale
E N D
A Panel Pilot Study for English Business Survey Presented by Yi Zhang Department for Business, Innovation and Skills
Overview • Introduction to EBS • Why, Who, How • Dissemination • Panel Pilot Study • Pros and Cons of Panel approach • Objective • Methodology • Timescale • Sample Selection and profile • Fieldwork • Results • Analysis of responses • Mode effects • A propensity weighting approach • Conclusions and Next steps • Comments and Questions
Introduction to EBS EBS - Why? • Need to maintain regular flow of timely sub-national data after close-down of RDA • EBS seen as best way to provide this • Survey intended to complement existing sources, not replace them • In accordance with transparency agenda, data will be freely available for users to conduct own analysis
Introduction to EBS EBS - Who? • Data collected by TNS-BMRB – independent contractor commissioned by BIS • Three-thousand workplaces surveyed every month • Covers all nine English regions • Sample drawn from Inter-Departmental Business Register • Covers all sectors, including public sector
Introduction to EBS EBS- How? • Short telephone-based survey • Designed to be light-touch as possible • Approximately 10-12 minutes to complete • Voluntary • Directional questions - able to complete without detailed material • Currently use a cross-sectional approach • Panel pilot study undertaken to explore the possibility of moving to a panel from year 2 onwards
EBS Dissemination Balance statistics (i.e. higher minus lower) by workplace size) Jan ’12 vs. Oct ‘11 • Monthly Statistical Releases • Planned Quarterly Statistical Releases • Online reporting tool (under development)
Pros of a Panel Approach • Increase analytical capability • Track individual workplaces over time and across samples • Enhance indicators of economic activity and behaviour • Improve quality and reliability • More larger businesses can be interviewed • Less variation in the sample between waves • Cost effectiveness • Cheaper to re-contact the same respondents
Cons of a Panel Approach • Potentially increase respondent burdens - a maximum of four interviews per year for each workplace as opposed to one currently • May cause a break in the series so far – size of break is unknown but could be investigated in early stages of panel • Expected to be small given the questions remain the same and the panel gradually built up
Objectives of the Pilot Study • To estimate the proportion of workplaces to participate in a Panel survey (by size, region and sector) • To estimate the proportion of workplaces to respond online • To explore any mode effects between the online and telephone data
Sample Selection 3081 interviews in Nov main fieldwork 2357 agreed to be re-contacted 1000 selected in Re-contact 1 (957 continued in Re-contact 2) 1,357 remain A1: workplaces with email address Re-contact 1 – 688 Re-contact 2 – 662 A2: workplaces who provided no email - to be contacted by phone Re-contact 1 – 312 Re-contact 2 – 295 B: workplaces with email addresses (contacted to boost online response) Re-contact 1 – 926 Re-contact 2 – 914
Sample Profile by Region and Sector • Similar profiles of selected sample, those who agreed to recontact, and the Nov fieldwork sample within each region and sector
Sample Profile by Workplace Size • Profile of those who agreed to re-contact mirrored very closely to the main Nov sample by workplace size • Relatively higher proportion of 50+ selected for the panel • to encourage more larger workplaces to participate
wk1 wk2 wk3 wk4 A1:Supplied email address & Phone # CAWI CATI & CAWI A2:Supplied Phone # only CATI Panel Fieldwork • Two re-contact panels • Each one calendar month: Feb and Apr 2012 • Overlap of respondents between two panels : ~40% • Same design for each follow up: • two groups: A1: mixed modes ; A2: CATI only • two modes: online (CAWI), telephone (CATI)
Results 1: Conversion rates and online take up • Conversion rates consistently lower in Panel Apr across workplaces of different size, regions and sectors • No consistent pattern - conversion rate higher for the CATI only (A2) compared to the mixed mode (A1) in Panel Feb, but conversion rates for A1 and A2 similar in Panel Apr • Due to a lower conversion rate for large workplaces from A2 group in Panel Apr • Rates for the online mode much lower than CATI for both panels
Results 2: Conversion rates by workplace size and industry • Workplaces with 250+ employment were less likely to respond for both panels • Those in Education, Health Public Admin and defence were more willing to participate in both panels
Results 3: Conversion rates by region • Regional variation for both panels • London and the North West were consistently less likely to participate
Mode Effect – Propensity Weighting Approach • To investigate whether there was a online (treatment) vs. telephone (control) effect • Propensity score weighting • Aim: Control for differences in profile between two groups • Method: • A logistic regression model to produce the propensity weights • Applied weights to the telephone group to match the profile of workplaces that completed the survey online Y: mode of completion X: Workplace characteristics - employment size, region, sector, single/multi sites, Age Weight:
Mode Effect – Propensity Weighting Approach (2) • Only workplace size predictive of being in the online sample • The online sample has a higher proportion of small workplaces compared with the telephone sample • After weighting, workplace sizes in the telephone sample closely matches those in the online panel, while region and industry profiles remain similar
Results: Mode Effect • Difference between two modes but not consistent pattern • small on-line sample • samples not aligned as closely as they need to be after propensity score matching • Significant different outcomes of key questions between online and weighted telephone samples: • Those responding by telephone were more likely to agree to re-contact in future and to have their data linked • Different responses to the key questions, but not consistently negative or positive • E.g. Those responding by telephone were less likely to say • ‘The same’ at level of business activity or volume of output last month compared with 3 months before
Conclusions • Workplaces would engage with a panel survey - conversion rates where higher than predicted. If a panel adopted, ~50% of the interviews come from re-contact sample in the first follow up month • Online is not a popular mode - Only a small proportion of any mixed mode panels would be online respondents • Indications of a possible mode effect - exist but not coherent. More agreements to re-contact and data linkage via telephone mode than online mode
Next Steps • Results point to a telephone only panel approach • Further analysis of pilot results • Put to EBS SG for a decision about whether to change the design for November 2012 fieldwork
Thanks for your attention! Comments or Questions?