A Panel Pilot Study for English Business Survey

1. A Panel Pilot Study for English Business Survey Presented by Yi Zhang Department for Business, Innovation and Skills

2. Overview • Introduction to EBS • Why, Who, How • Dissemination • Panel Pilot Study • Pros and Cons of Panel approach • Objective • Methodology • Timescale • Sample Selection and profile • Fieldwork • Results • Analysis of responses • Mode effects • A propensity weighting approach • Conclusions and Next steps • Comments and Questions

3. Introduction to EBS

4. Introduction to EBS EBS - Why? • Need to maintain regular flow of timely sub-national data after close-down of RDA • EBS seen as best way to provide this • Survey intended to complement existing sources, not replace them • In accordance with transparency agenda, data will be freely available for users to conduct own analysis

5. Introduction to EBS EBS - Who? • Data collected by TNS-BMRB – independent contractor commissioned by BIS • Three-thousand workplaces surveyed every month • Covers all nine English regions • Sample drawn from Inter-Departmental Business Register • Covers all sectors, including public sector

6. Introduction to EBS EBS- How? • Short telephone-based survey • Designed to be light-touch as possible • Approximately 10-12 minutes to complete • Voluntary • Directional questions - able to complete without detailed material • Currently use a cross-sectional approach • Panel pilot study undertaken to explore the possibility of moving to a panel from year 2 onwards

7. EBS Dissemination Balance statistics (i.e. higher minus lower) by workplace size) Jan ’12 vs. Oct ‘11 • Monthly Statistical Releases • Planned Quarterly Statistical Releases • Online reporting tool (under development)

8. Panel Pilot Study

9. Pros of a Panel Approach • Increase analytical capability • Track individual workplaces over time and across samples • Enhance indicators of economic activity and behaviour • Improve quality and reliability • More larger businesses can be interviewed • Less variation in the sample between waves • Cost effectiveness • Cheaper to re-contact the same respondents

10. Cons of a Panel Approach • Potentially increase respondent burdens - a maximum of four interviews per year for each workplace as opposed to one currently • May cause a break in the series so far – size of break is unknown but could be investigated in early stages of panel • Expected to be small given the questions remain the same and the panel gradually built up

11. Panel Pilot Study - Objectives

12. Objectives of the Pilot Study • To estimate the proportion of workplaces to participate in a Panel survey (by size, region and sector) • To estimate the proportion of workplaces to respond online • To explore any mode effects between the online and telephone data

13. Panel Pilot Study – Method

14. Timescale for the Pilot Study

15. Sample Selection 3081 interviews in Nov main fieldwork 2357 agreed to be re-contacted 1000 selected in Re-contact 1 (957 continued in Re-contact 2) 1,357 remain A1: workplaces with email address Re-contact 1 – 688 Re-contact 2 – 662 A2: workplaces who provided no email - to be contacted by phone Re-contact 1 – 312 Re-contact 2 – 295 B: workplaces with email addresses (contacted to boost online response) Re-contact 1 – 926 Re-contact 2 – 914

16. Sample Profile by Region and Sector • Similar profiles of selected sample, those who agreed to recontact, and the Nov fieldwork sample within each region and sector

17. Sample Profile by Workplace Size • Profile of those who agreed to re-contact mirrored very closely to the main Nov sample by workplace size • Relatively higher proportion of 50+ selected for the panel • to encourage more larger workplaces to participate

18. wk1 wk2 wk3 wk4 A1:Supplied email address & Phone # CAWI CATI & CAWI A2:Supplied Phone # only CATI Panel Fieldwork • Two re-contact panels • Each one calendar month: Feb and Apr 2012 • Overlap of respondents between two panels : ~40% • Same design for each follow up: • two groups: A1: mixed modes ; A2: CATI only • two modes: online (CAWI), telephone (CATI)

19. Panel Pilot Study – Results

20. Results 1: Conversion rates and online take up • Conversion rates consistently lower in Panel Apr across workplaces of different size, regions and sectors • No consistent pattern - conversion rate higher for the CATI only (A2) compared to the mixed mode (A1) in Panel Feb, but conversion rates for A1 and A2 similar in Panel Apr • Due to a lower conversion rate for large workplaces from A2 group in Panel Apr • Rates for the online mode much lower than CATI for both panels

21. Results 2: Conversion rates by workplace size and industry • Workplaces with 250+ employment were less likely to respond for both panels • Those in Education, Health Public Admin and defence were more willing to participate in both panels

22. Results 3: Conversion rates by region • Regional variation for both panels • London and the North West were consistently less likely to participate

23. Mode Effect – Propensity Weighting Approach • To investigate whether there was a online (treatment) vs. telephone (control) effect • Propensity score weighting • Aim: Control for differences in profile between two groups • Method: • A logistic regression model to produce the propensity weights • Applied weights to the telephone group to match the profile of workplaces that completed the survey online Y: mode of completion X: Workplace characteristics - employment size, region, sector, single/multi sites, Age Weight:

24. Mode Effect – Propensity Weighting Approach (2) • Only workplace size predictive of being in the online sample • The online sample has a higher proportion of small workplaces compared with the telephone sample • After weighting, workplace sizes in the telephone sample closely matches those in the online panel, while region and industry profiles remain similar

25. Results: Mode Effect • Difference between two modes but not consistent pattern • small on-line sample • samples not aligned as closely as they need to be after propensity score matching • Significant different outcomes of key questions between online and weighted telephone samples: • Those responding by telephone were more likely to agree to re-contact in future and to have their data linked • Different responses to the key questions, but not consistently negative or positive • E.g. Those responding by telephone were less likely to say • ‘The same’ at level of business activity or volume of output last month compared with 3 months before

26. Panel Pilot Study – Conclusions and Next Steps

27. Conclusions • Workplaces would engage with a panel survey - conversion rates where higher than predicted. If a panel adopted, ~50% of the interviews come from re-contact sample in the first follow up month • Online is not a popular mode - Only a small proportion of any mixed mode panels would be online respondents • Indications of a possible mode effect - exist but not coherent. More agreements to re-contact and data linkage via telephone mode than online mode

28. Next Steps • Results point to a telephone only panel approach • Further analysis of pilot results • Put to EBS SG for a decision about whether to change the design for November 2012 fieldwork

29. Thanks for your attention! Comments or Questions?