Advancing Paradata

1. Advancing Paradata Paradataand the UK Poverty and Social Exclusion survey David Gordon & EldinFahmy Townsend Centre for International Poverty Research University of Bristol

2. Poverty in the UK: Advancing Paradata Analysis and Open Access University of Southampton Ros Edwards Institute of Education Ann Phoenix Heather Elliott Michelle Cage University of Bristol David Gordon Eldin Fahmy Karen Bell Viliami Fifita Web Site http://www.poverty.ac.uk Web Site http://www.bristol.ac.uk/poverty

3. The research aims • To improve the measurement of poverty, deprivation, social exclusion and standard of living. • To measure the change in the nature and extent of poverty and social exclusion over the past ten years. • To produce policy-relevant results about the causes and outcomes of poverty and social exclusion.

4. Background Every decade since the late 1960s, UK social scientists have attempted to carry out an independent poverty survey to test out new ideas and incorporate current state of the art methods into UK poverty research. • 1968-69 Poverty in the UK survey (Peter Townsend and colleagues), • 1983 Poor Britain survey(Joanna Mack, Stewart Lansley) • 1990 Breadline Britain survey (Joanna Mack, Stewart Lansley) • 1999 Poverty and Social ExclusionSurvey (Jonathan Bradshaw and colleagues) and its 2002 counterpart in Northern Ireland (Paddy Hillyard and colleagues) • 2012 Poverty and Social Exclusion in the UK

5. Survey Data Omnibus Survey: Necessities of Life – A systematic random achieved sample of adults (16+) of over 1,900 interviews in Britain and Northern Ireland. Main Survey: Poverty & Social Exclusion – A follow-up survey to the 2010/11 Family Resources Survey (circa 47,000 households). The main survey in Britain has an achieved sample of over 4,000 households and 9,500 individuals – with approximately 1,000 households in the ‘ethnic’ strata and 1,000 households in Scotland. In Northern Ireland, the achieved sample was just under be 1,000 households and over 2,000 individuals. The survey was divided into a household questionnaire which was answered by the Household Reference Person (HRP) and an individual questionnaire which was answered by all adult household members (aged 18 and over).

6. Qualitative Data British Impoverishment Study, The qualitative study, Life on a Low Income in Austere times sought to provide insights into the experiences of poverty during the recent recession and ongoing programme of austerity. The project collected 62 video/audio testimonies during 2012-2013, in Birmingham, Glasgow and Gloucestershire. Northern Ireland Family Solidarity Study, a qualitative survey of 50 parents to explore the role of family in coping with poverty. The study had a material focus – examining the extent to which resources are transferred among family members (both nuclear and extended) - and it also explores family cultures and relationships as factors affecting poverty and social exclusion. This helps to understand the extent and limits of family solidarity

7. The Bristol Social Exclusion Matrix (B-SEM) Quality of Life Resources Material & economic resources Health & wellbeing Access to public & private services Living environment Social exclusion Social resources Education & skills Crime, harm & criminalisation Participation Social participation Economic participation Cultural participation Political and civic participation

8. 2012 PSE-UK Question development and testing Survey pre-test • Conventional survey pilot with interviewer de-briefing and analysis of non-response Expert review • Input from team members and UK and International Advisory Groups Survey paradata • Behaviour coding; analysis of question timings and interviewer characteristics Focus groups • Exploring public perceptions of deprivation, living standards and social exclusion in the UK today to inform the NatCen Omnibus and PSE Main Stage surveys Cognitive Interviews • Qualitative pre-testing of selected indicators for potential inclusion in the PSE Main Stage survey

9. Paradatain the Hong Kong Standard of Living Survey 《香港的生活水平研究》中的環境數據 Paradata is a recent and rapidly expanding area of interest amongst social survey researchers. Given the utility of paradata as a means of assessing survey and data quality, it would be very helpful if you can spend few minutes to fill in the following information recorded for each interview and included on the subsequent survey data file. 在社會調查研究學科內，環境數據（paradata）是一個新興和迅速發展的研究議題。環境數據是有效評估調查和數據質素的方法，我們希望閣下能在完成每份問卷調查後花數分鐘時間填寫以下的資料，并並附於該問卷調查的檔案中。 Interviewer ID code訪問員的身份編碼: ______________________ Age of the interviewer訪問員的年齡: ______________________ Gender of the interviewer. 訪問員的性別  Male 男  Female 女 Date of the interview訪問日期: ______________________ Time of the interview訪問時間: ______________________ Duration of the interview訪問時間的長度: ______________________ minutes 分鐘 If the interview was terminated before completion訪問是否中止，未能完成 Yes是 No否

10. The PSEHK survey Interviewers thought that only 60% of respondents provided ‘accurate’ information about their household incomes. This could be problematic in a poverty survey if only income information is used to measure ‘poverty’

11. Why behaviour coding? • No consensus on what is involved in adequate testing survey questionnaires • Pretesting often limited to interviewers perceptions of the response process • Non-response analysis uninformative about potential for response problems not resulting in item non-response • Application of systematic coding framework to survey interview transcripts to identify problems questionnaire administration • Assumes analysis of interview behaviour provides a reliable means of identifying question problems without the need to speak directly to respondents • Assumes that deviations from the paradigmatic, standardised survey interaction are a source of measurement error to be minimised • No consensus on what is coded and how: implications for evaluation of survey questionnaires poorly understood

12. About the PSE-UK Main stage survey design • Target interview duration of 50 mins for single person hhld (+30 mins. for each additional adult. • 23 interviews audio-recorded and transcribed verbatim. Sample selected by the survey organisation from experienced interviewers. Subsequent analysis of 111 question items • Household survey conducted separately in Britain and Northern Ireland by NatCen (Britain) and NISRA. Funded by the ESRC • Sample of respondents in 2010/11 FRS giving permission to be re-contacted. PSE fieldwork conducted Feb-Sept. 2012 using CAPI with all adults within selected households

13. Behaviour coding framework • Observations are Resp*Qitems. In theory 2,533 observations (23*111) • 1,689 valid observations in total. Selected respondents provided codeable responses to between 62 and 84 of the 111 items (mean=73.4; sd=6.3)

14. Interview problem codes: frequency counts (%)

15. Respondent problem codes: frequency counts (%)

16. Respondent problems by interviewer behaviourRow percentages

17. Mean number of question problems by interviewer and respondent characteristics

18. Interviewer and respondent problems by question item: standardised mean scores

19. Concluding thoughts NEXT STEPS… • Comparison of behaviour coding with interaction analysis of textual data • Comparison of PSE-UK behaviour coding with: • Cognitive interview question testing • Survey response and question timing paradata • ‘Traditional’ survey pilot results and expert review • How useful (if at all) are different questionnaire pretesting techniques? • To what extent do they give comparable results? What are the main differences in the results they give? • How should we explain and respond to inconsistent results between methods?