Mario Albisinni Elisa Marzilli Federica Pintaldi

1. The European Conference on Quality in Official Statistics Cognitive Test related to Labour Force Survey ad-Hoc Module on Migrants and their Immediate Descendants Mario Albisinni Elisa Marzilli Federica Pintaldi Rome. 9-11 July 2008

2. The European Conference on Quality in Official Statistics Introduction: Labour Force Survey • The Labour Force Survey (LFS) began on quarterly basis in 1959 • It is the biggest and most complex sample survey - 75.000 households / 170.000 individuals - average sampling rate: 1/300 - statistics for small geographical areas (NUTS II) • The LFS is a survey of households that provides information about people's employment status and conditions • Official source for employment and unemployment • Many Eu Regulations Rome. 9-11 July 2008

3. The European Conference on Quality in Official Statistics Introduction: Labour Force Survey • In order to match EUROSTAT standards. on 2002 ISTAT developed a new Labour Force Survey • It replaced the previous old survey in 2004 • The main changes concern both content and technical points of view: - the survey has become continuous - from a PAPI technique to a mixed technique CAPI-CATI - anew questionnaire -new information • Since2005we have started to diffuse information aboutnon nationals Rome. 9-11 July 2008

4. Ad hoc modules: specific items every year The European Conference on Quality in Official Statistics Introduction: Labour Force Survey 2000 Transition from school to working life 2001 Length and patterns of working time 2002 Employment of disabled people 2003 Lifelong learning 2004 Work organization and working time 2005 Reconciliation between work and family life 2006 Transition from work into retirement 2007 Accidents at work and work-related health problems 2008 Labour market situation of migrants Rome. 9-11 July 2008

5. The European Conference on Quality in Official Statistics LFS: Ad-hoc Module 2008 The aim is: • complete identification of the population of migrants and their immediate descendants • comparable data at European level • comparison between migrants (and their descendants) andnationals about main characteristics of labour market analysis of the factors affecting integration in the labour market Rome. 9-11 July 2008

6. The European Conference on Quality in Official Statistics LFS: Ad-hoc Module 2008 • In order to improve the quality of survey data for migrants by decreasing language barriers we conducted a cognitive test Rome. 9-11 July 2008

7. The European Conference on Quality in Official Statistics Cognitive Test • the cognitive test is a method to reduce non-sample errors and improve the quality of the data • the cognitive interview identifies the aspects of the question that lead to errors in the answers • the cognitive interview decomposes and codifies the thoughts of the interviewees when they answer a questionnaire • the purpose is to stimulate the respondents to suggest the difficulties encountered in answering questions Rome. 9-11 July 2008

8. The European Conference on Quality in Official Statistics Cognitive Test The cognitive process can be drawn up into4 steps: • Comprehension: the respondent tries to understand the question and its terms • Retrieval of relevant information from the memory: the respondent retrieves and selects the information in order to be able to answer the question • Decision-making (or judgment) process: the respondent reflects on what is the correct answer and about personal experience to share • Answer process: the respondent decides on the answer to be given and communicates it to the interviewer Rome. 9-11 July 2008

9. The European Conference on Quality in Official Statistics AM2008: Cognitive Test Methodology • An interviewer accompanied by one observer conducted a face-to-face qualitativeinterview • Both the interviewer and the observer had a trace they had to follow and which allowed them not to go off the point • Each interview was registered in order to be able to listen to it again and transcribe it • A standardized scheme was compiled to evaluate the interviewee’s level of question comprehension Rome. 9-11 July 2008

10. The European Conference on Quality in Official Statistics Cognitive Test Sample • We interviewed a selected group of 30 migrants by sex. citizenship and professional status • The sample has been divided in two: - first 15interviews: test questionnaire some questions were re-formulated because of problems emerged -second 15interviews: test new questions Rome. 9-11 July 2008

11. The European Conference on Quality in Official Statistics Cognitive Test The interview The questionnaire contains 3 kind of questions: • Questions to be tested: read exactly the way they are written on the questionnaire to evaluate whether the terms are correct and comprehensible • Probe questions: asked after the questions to be tested allow to examine more in-depth the meaning of the answers given • Questions for the observer: allow to survey the non-verbal reactions of the communication process and suggest what kind of information should be written on for each question Moreover the probe questions can be integrated with other questions the interviewer may freely elaborate regarding aspects that emerge during the interview Rome. 9-11 July 2008

12. The European Conference on Quality in Official Statistics Cognitive Test The probe questions concern: • problems relative to the question comprehension/ interpretation • re-wording of the question by the interviewer • certainty of the answer given • strategy used for retrieving past information • opinion on a specific aspect discovered during the interview • reason for choosing a specific answer and the difficulties encountered in formulating it Rome. 9-11 July 2008

13. The European Conference on Quality in Official Statistics Cognitive Test The cognitive test questionnaire: example Rome. 9-11 July 2008

14. The European Conference on Quality in Official Statistics Cognitive Test The analysis The first step was to transcribe the dialogues of the interviews and notes of the observer. For each question. we indicated: • answer given • respondent’s difficulties • observer’s comments Rome. 9-11 July 2008

15. The European Conference on Quality in Official Statistics Cognitive Test The analysis We compiled a standardized scheme to evaluate the interviewee’s level of comprehension: • interviewee’s question • answer modality • pertinence of the answer Rome. 9-11 July 2008

16. The European Conference on Quality in Official Statistics Cognitive Test The interviewee’s question comprehension: Rome. 9-11 July 2008

17. The European Conference on Quality in Official Statistics Cognitive Test The interviewee’s question comprehension: Rome. 9-11 July 2008

18. The European Conference on Quality in Official Statistics Cognitive Test The answer modality: Rome. 9-11 July 2008

19. The European Conference on Quality in Official Statistics Cognitive Test The pertinence of the question with the interviewee’s condition : Rome. 9-11 July 2008

20. answers with “positive comprehension” Comprehension Index = *100 total answers The European Conference on Quality in Official Statistics Cognitive Test • for each question. the combination of the information from the content analysis has allowed reconstructing a comprehension index • it was created adding the “positive comprehension” answers and dividing these per total number of answers The index moves from 0 to 100 (max comprehension) Rome. 9-11 July 2008

21. The European Conference on Quality in Official Statistics Cognitive Test In relation to thecomprehension index of wording questionswe considered positive “comprehension answer”: • Did the interviewee understand the question immediately? [Modality: yes] • How many difficulty was for interviewee answering the question? [Modality: not much/not at all] • Did the interviewee understand the time reference? [Modality: yes] • Did some words of the question result not very comprehensible? [Modality: no] • Other question comprehension problems? [Modality: no] Rome. 9-11 July 2008

22. The European Conference on Quality in Official Statistics Cognitive Test In relation to thecomprehension index of modalitiesof answerwe considered positive “comprehension answer”: • Were the answer modalities clear? [Modality: yes] • Were the answer modalities exhaustive? [Modality: yes] • How difficulty was for the interviewee have to choose an answer modality? [Modality: not much/not at all] Rome. 9-11 July 2008

23. The European Conference on Quality in Official Statistics Cognitive Test: results Rome. 9-11 July 2008

24. The European Conference on Quality in Official Statistics Cognitive Test: results The new wordings improved the comprehension levels: the average comprehension level of wording question arises from 71% of the 1st group to 87% of 2nd group Rome. 9-11 July 2008

25. The European Conference on Quality in Official Statistics Cognitive Test: results The main problems that could arise when using cognitive test and how they can be corrected Rome. 9-11 July 2008

26. The European Conference on Quality in Official Statistics Cognitive Test: some examples Which is the main reason for coming to Italy? 1|_|To work 2|_|To study 3|_|For family reasons (reunion. transfer) 4|_|For marriage. affective relationship 5|_|To ask for political asylum 996|_|Other (specify____________) 997|_|Does not know PROBLEMS: The answer modalities were neither clear nor exhaustive. The modality “to work”. produced some uncertainties for those who came to Italy to search for a job. R: to create myself a future. I cannot say for work because I did not have a work contract; I came to Italy to find a job The reason changes in the time R: To work. Well. at first not to stay in Italy. My objective was to go to Spain and continue to study. Then. I had to stay in Italy because I did not have much money anymore; I found a job and stayed… Rome. 9-11 July 2008

27. The European Conference on Quality in Official Statistics Cognitive Test: some examples Which is the main reason for coming to Italy? 1|_|To work 2|_|To study 3|_|For family reasons (reunion. transfer) 4|_|For marriage. affective relationship 5|_|To ask for political asylum 996|_|Other (specify____________) 997|_|Does not know • CORRECTIONS: • the first modality was divided in “look for a job” and “to work” • the verb was put to the past tense to focus the attention on the reason for migrating rather than on the reasons that subsequently came to create • a new modality was added “short stay in Italy to go to another country” to survey when Italy is a transitory destination rather than the reason for migrating Rome. 9-11 July 2008

28. The European Conference on Quality in Official Statistics Cognitive Test: some example of results In the 2 years subsequent to your arrival in Italy. did you use one of the following services supplied by public or private institutions/associations: 1|_| Assistance for searching a job 2|_| Professional training (including the one on the workplace) …… PROBLEMS: the structure of the question resulted to be complex and the time reference was not always understood. I: is it clear in the question that we refer only to first two years? R: No. I understood the whole period • CORRECTIONS: • the wording of the question was simplified During the first 2 years you lived in Italy. did you use one of the following services: Rome. 9-11 July 2008

29. The European Conference on Quality in Official Statistics Cognitive Test: some example of results Do you need to know better Italian to find a job that is more adequate to your skills and/or previous working experiences? 1|_|Yes 2|_|No PROBLEMS: the phrase structure resulted complex: interviewees asked several times to repeat the question. three terms were not very comprehensible: “Italian”. “skills” and “adequate” most of the interviewees answered only the first part of the question (knowledge of the Italian language) without considering the second part of the question (work more adequate to one’s own skills and/or previous working experiences) R: what does “Italian” mean? The Italian language or Italian as person?” R: “adequate” though is difficult …I would say “better”… R: to persons who do not understand well Italian. the question is too long. At first. one is not sure of what it is all about R: Instead of “adequate”. I would say “more right for the things you can do” Rome. 9-11 July 2008

30. The European Conference on Quality in Official Statistics Cognitive Test: some example of results • CORRECTIONS: • the question was divided into two parts • in the first part. respondents are asked if they have an adequate job • if “no” we asked if the difficulty is because of poor knowledge of the Italian language Do you have a job close to your studies and competences? 1|_|Yes 2|_|No To have a job close to your studies and competences. do you need to improve the knowledge of the Italian language? 1|_|Yes 2|_|No Rome. 9-11 July 2008

31. Ad-hoc Module 2008: Cognitive Test • Conclusion • after a second cycle of interviews the questionnaire becomes definitive • the cognitive test has been particularly useful in order to set-up of the ahm 2008 • we reduce the non-sample error that bias the accuracy of data • Cognitive test highlights hidden errors Rome. 9-11 July 2008

32. The European Conference on Quality in Official Statistics THANK YOU A LOT! Rome. 9-11 July 2008