Overview of European Data from Official Statistics

1. Overview of European Data from Official Statistics Roxane Silberman CNRS/Réseau Quetelet and DwB coordinator With the support of Cyril Jayet, Marie Cros, Raphaëlle Fleureux, Alexandre Kych, Benoît Tudoux (CNRS-RQ) DwB WP3 and WP5 - CESSDA PPP WP10 DwB training course: Working with data from Official Statistics particularly the European Labour Force Survey Bucharest, 2nd DwB Training Course, February 13-15, 2013

2. Introduction • Main course on SILC, Eurostat microdata • Yet it is important to have a larger overview of official microdata in Europe • European integrated microdata (Eurostat) come from national microdata that are quite diverse • Raises a lot of methodological issues about comparability: ex. SILC • May be complemented • by related microdata available at national level more detailed, other variables, and sometimes on a longer period • by other microdata available at European and national levels • Provide a short and comprehensive overview of the rich resources of official microdata in Europe • What is available at national and European level • Relations between the different levels • How and where to access these microdata • While helping locating in this landscape the SILC, related national microdata and complementary sources on similar topics

3. Outline • Historical background for national and European microdata and consequences • Overview of official microdata in Europe • Integrated European microdata (Eurostat) • SILC • Other European microdata • National microdatacollected and harmonized in European databases • Related topics to SILC • Other national microdata • Related topics to SILC • How to to find information and metadata ? • How access microdata across borders ? • Eurostat access and other European microdata access • Transnational access to national microdata • Support by DwB • Transnational access calls • CIMES

4. I. Historical backgrounds for national and European microdata and consequences A vast, complex and moving perimeter Historical backgrounds for national microdata Historical backgrounds for European microdata Some consequences for research

5. Official microdata: a vast perimeter • Different words (official, government, national etc..) • In any cases, statistics provided by government bodies • A larger perimeter than NSIs and Eurostat • Microdata provided by : • National level • National Statistical Institute • National Statistical administrations coordinated by the NSI + fiscal data • Central banks • Government agencies particularly those in charge of Social security, health, pensions … • Local authorities Number of government bodies producing and providing data vary according to the organization of the each statistical system and degree of centralization • European level • Eurostat • European Central Bank • European Commission and agencies • Databases held by universities

6. … complex and moving • Perimeter may change according to the decision of governments/NSIs • See France: • Customs statistical department went out and in • Fiscal data recently included as a Statistical department coordinated by INSEE • Cereq microdata on transition from school to work went out … • Household finance and consumption surveys (HFCS) move to central banks coordinated by the BCE • Mixed status for some government agencies (social security, unemployment …) • What about data to be certified by NSIs ? • Under different legal framework (surveys, administrative data, business data, fiscal data, health data, financial data …) with consequences for access • Perimeter does not necessarily cover similar data in the different countries depending on role of other producers (universities …) and historical changes (Eastern countries recently joining the EU)

7. Different types of microdata • Censuses or registers + longitudinal samples from censuses • Individual and households surveys + birth cohorts and panels • Business surveys • Administrative data (frequently longitudinal databases) • Combined datasets • Administrative datasets • More common in countries based on registers • Now increasingly the case in all countries (yet requiring a common identifier) • Administrative data and surveys

8. Historical backgrounds for national microdata • Statistical systems initially built at national level • From diverse sources and bodies, piece by piece, • Toward an increasing coordination role of the NSI • Yet keeping traces from the construction even in centralized statistical system where some bodies still remain apart. • Important differences • Surveys and registers countries • Centralized vs decentralized/coordinated • Political systems • Regional autonomy (Spain) • Federal system (Germany and the landers) • UK and Scotland • France and overseas departments • National and local authorities gaining importance

9. Main topics • Similarities • Households: Demography, migration ,family, employment, education, budget and income • More variety • Housing, transportation, health, social networks, opinions, social mobility … • Business • Wages, work organization, innovation… • Increasing number of administrative data combined to lower the respondent burden (firms more reluctant to answer) • Number of official datasets may vary depending on: • Registers or surveys systems • Degree of centralisation • Political changes: Eastern countries • Role of the universities (France vs Germany) • Examples from first results of DwB WP5 : short list in Eastern countries in line with Eurostat microdata, longer list in France, medium in Germany

10. Historical background European level • Increasing harmonization process at international level since WW2 led by international organisations (UN, OECD…) mostly by encouragement and persuasion • European framework specific as there is a political and legal framework • European Statistical System development started with the CECA during the 50ies, progressively developed during 30 years, new start and developments since the 90ies towards more integration • Harmonization process for variables, classifications, methodology • Yet relying frequently on pre-existing national microdata and on the subsidiarity principle • As a consequence, • European integrated microdata are based on national microdata from various sources (surveys, registers, administrative basis) depending on countries • Methodological problems (see Training course part 3 on SILC) • Dissemination under European bodies yet access to national part under decision of countries • Not all integrated microdata include all countries • Access still burdensome even if progress expected in the future • Level of anonymization high due to differences in countries size and in countries requirements • Yet access to national part may be possible at national level. • May be more detailed and accessible in some countries • May include other questions and variables • May offer a wider historical perspective if pre-existing in the countries (yet not for all countries) • Yet the harmonization process may impact the series and determine breaks in the series at national level

11. Some consequences for working with data • At European level • European microdata may gather data coming from surveys or from registers • Some harmonised databases may include government and non government microdata • At national level • Depending on the degree of centralisation for production and/or dissemination information and access are more or less fragmented and may have move from one body to another one • NSI unique or main provider (Netherlands) vs NSI and Statistical departments and other government bodies (France) • Recently the French Income survey moved from the NSI to the Central Bank • Depending on the legal status access may be more or less accessible for researchers • For comparative research projects requiring access to national microdata • Researchers face “silos” for information and access

12. II. European and national microdata Three subsets from a European perspective European integrated microdata National microdata harmonized at European level Other national microdatafor comparative research

13. A. European integrated microdata Produced and provided at national level and integrated and provided at European level by European government bodies To remind : national part may differ at national level and at European level A part are pre-existing surveys integrating harmonization requirements • Eurostat • European Central Bank, • European Commission and other European government bodies • Others under EU regulations or recommendations of Eurostat

14. A.1 Eurostat microdata • Labour Force Survey (LFS) • European Community Household Panel (ECHP) • Statistics on Income and Living Conditions (SILC) • Adult Education Survey (AES) • Community Innovation Survey (CIS) • Structure of Earnings Survey (SES)

15. Related national microdata for LFS and SILC • LFS : a wider historical perspective in some cases • SILC: different sources • More variables in some cases • Surveys and registers

16. Labour Force Survey (LFS) at national level: a wider historical perspective

17. SILC and the Estonian Social Survey • “ESS is the Estonian branch of a pan-European survey of income and living conditions called the EU-SILC (…). Statistics Estonia, however, has added questions, which are of interest to the domestic consumers of Estonia, to the EU-commissioned survey, and attempts to have the survey be a combination of Estonian and European data requirements.” • In 2004, four modules were added (…). They were all commissioned by Estonian domestic consumers. The topics of the four modules concerned social contacts; family attitudes and political views; crime, violence and feeling of security; and ethnic integration. • In 2005, there were three modules in ESS: one by order of Eurostat and the other two by domestic consumers. The topic of the Eurostat module was “Social origin”(…) Estonian domestic modules were entitled “From school to work” and “Trade unions and collective agreements”.

18. SILC and French EPCV and SRCV • SILC (SRCV) starts in 2004, but… • Living conditions 1978-1979, 86-87; 93-94 • European Community Household Panel (1994-2001) • Permanent Living Conditions Survey EPCV (1996-2004) • The current French SRCV system took over from the former permanent survey of living conditions (EPCV) system in 2004: it reprises some questions from the EPCV to produce social indicators considered indispensable for the specific needs of the French statistical institute but not required within the European framework.

19. A diversity of situations • The Belgian datasets are much more detailed than the delivered files to Eurostat  • SILC datasets from Eurostat do not contain Swiss data (2007 - 2010). • The Great Britain component of the EU-SILC dataset is collected by the Office for National Statistics (ONS) as part of the General Lifestyle Survey (GLF) (held at the Archive under Special Licence access conditions - see GN 33403). • The Northern Ireland component is collected by the Northern Ireland Statistics and Research Agency (NISRA) as part of the Living Conditions Survey (LCS) (not currently held at the Archive).

20. A Danish example of register :THE REGISTER FOR HEALTH AND SOCIAL CONDITIONS • The purpose of the register is to provide Statistics Denmark, the Ministry of Health and the National Board of Health with readily available information from public registers on the state of health of the Danish population and relevant socio-economic background conditions. Another purpose is to provide data for research in social medicine. • The register was established in cooperation between the three authorities • The collection of data for the register began in 1994, starting with data from registers with information available from 1977 onwards: The National Register of Patients, the Register of Population Statistics and the Employment Classification Module. • This was followed in 1995 by the inclusion of data from the following registers: • the Register of Causes of Death, the Register of Health Insurance Statistics and the Register of Coherent Social Statistics. • In 1997 a final agreement was made defining the contents of the register, including data from three more registers: • The Integrated Database for Labour Market Research, the Educational Classification Module and the Register of Building and Dwelling Statistics. • At the time, specification was made for the future inclusion of data on medical births, voluntary abortions and deformations in birth.

21. A.2 European Central Bank microdata • Household Finance and Consumption Survey • Every 3 years • First deliverable in 2013 • No pre-existing national survey in some countries while older waves in others • France: • Every 6 years and oldest waves • Questionnaire 112 pages in France vs 65 pages for the European survey, yet some variables collected at European level not in the French survey • Adaptation of the survey was needed (break in the serie) • Moved from the NSI to the Central Bank (with consequence for researcher access)

22. A.3 Other European government microdata • The European Commission, the Directorate-General for Economic and Financial Affairs (DG ECFIN) • Business and Consumer Surveys • EUROFOUND • Surveys on working conditions

23. The Business and Consumer Surveys • The European Commission, the Directorate-General for Economic and Financial Affairs (DG ECFIN) • manages a network of national institutes to conduct a harmonised EU programme of 6 business and consumer tendency surveys (quarterly or monthly from 1985, 1995 for services). Industry Services Consumers Retail trade Building Investment and some others

24. EUROFOUND and surveys on working conditions The European Foundation for the Improvement of Living and Working Conditions • An autonomous EU agency • Set up by the Council Regulation (EEC) No. 1365/75 of 26 May 1975 • Contributes to the planning and design of better living and working conditions Governing board: • 3 representatives per country, for 28 countries (EU27 and Norway) • The ministry in charge of labour • The employers • The unions • 3 representatives for the European Commission • The DG Employment • The DG Health and consumer protection • The DG Research end innovation • Based in Dublin with a Brussels Liaison Office

25. 3 surveys combining companies and employees surveys The European Working Conditions Survey (EWCS) : 1990, 1995, 2000, 2005, 2010 • Working conditions and the quality of work and employment The European Quality of Life Survey (EQLS) : 2003, 2007, 2010 • A broad range of indicators of quality of life, both objective and subjective The European Company Survey (ECS) : 2004, 2009, 2013 • Workplace practices based on the views of both managers and employee representatives The exemple of the EWCS surveys • Year 1990 1995 2000 2005 2010 • Number of workers 12500 15800 21500 30000 44000 • Number of countries EC12 EU15 EU15+12 EU27+4 EU27+7 • A questionnaire developmentexpert group: Eurofound's Governing Board, other EU bodies (EU Commission, Eurostat, European Agency for Safety and Health at Work), international organisations (OECD, ILO), national statistical institutes and leading European experts in the field • Implemented by INRA Europe then Gallup Europe, with national fieldwork partners • Datasets available for download at the ESDS (UKDA)

26. A. 4 Others microdata under EU regulations or Eurostat recommendations for harmonization • Household budget surveys (HBS) • Eurostat recommendations about methodology and harmonization • Information and Communication technologies surveys (TIC) • Regulation n°808/2004 and updated regulation 1006/2009 linked with the European roadmap for the TIC

27. B. National microdata harmonized at European level • IECM (IPUMs international) • Censuses • MTUS • Time Use Survey • LIS (Luxembourg Income Study) • Household Budget Survey Collected and a posteriori harmonized by Universities Archives (MTUS, LIS, IECM) More detailed microdata often accessible at national level

28. B.1 IECM/IPUMS and national dissemination of European censuses IECM + national dissemination IECM soon + national dissemination Only national dissemination No information

29. More censuses microdata available at national level • The case of the 2001 census • The example of 2 European countries : United Kingdom France (new census, yearly since 2006, the example of 2008)

30. United Kingdom Samples of anonymized records (SARs) • The Individual Licensed SAR • 3 % (1 800 000 people) • UK : 9 government office regions for England, and Wales, Scotland and Northern Ireland Available from CCSR (a charge may apply), also available from IPUMS • The Special Licence Household SAR (SL-HSAR) • 1 % (245 000 households) • England and Wales Available from the UK Data Archive • The Small Area Microdata (SAM) • 5 % (2 900 000 people) • ±500 Local authorities Available from CCSR (a charge may apply) The Controlled Access Microdata Sample(s) (CAMS) • Both the individual and household SARs files • Available in much greater detail Accessible in safe settings in all ONS sites, for approved research projects. Only for England and Wales

31. France : Available as PUF from Insee website The standard files (PUF) • The dwelling file • The individual file by regions Available from IECM/IPUMS • The individual file by constituencies • The individual file by place of work 3 standard (PUF) individual files for residential mobility They differ for confidentiality reason, • Place of residence at the census time and previous residence (also abroad). • For departments, great communes or all communes • The individual file for travel to work mobility • From commune to commune) • The individual file for school mobility • From commune to commune

32. France: Available for researchers via Data Archive The SUF files • For the last Census available for the end of 2012. • 2 to 3 files: An individual file with a complete set of variables, at the region level A family file. An individual file of the annual census survey, at the region level. Available from the Réseau Quetelet The Secure Access Center to Data (CASD, Centre d’Accès Sécurisé aux Données, GENES-Réseau Quetelet) • All the microdata census files (and others …) Available in the most detailed form Secure remote access

33. B.2 LIS is a cross-national data center, located in Luxembourg. LIS is home to the Luxembourg Income Study Database (LIS) and the Luxembourg Wealth Study (LWS)Database.

34. Household Budget Survey

35. The Centre for Time Use Research collects Time Use Surveys

36. Time Use Survey

37. Household Budget Surveys, Information and Communication Technology Surveys and Time Use Surveys HBS + TIC + TUS HBS + TIC TIC + TUS HBS + TUS TIC No information

38. C. A lot of other national OS usable for comparative research • Various surveys • Related surveys on living conditions (other than those related to the SILC) • Other examples : • Social mobility • School leavers and transition from school to work • Administrative microdata increasingly combined and used by researchers • Examples • Employers and employees • Social security, pensions

39. Other national surveys on living conditions conditions

40. Social mobility Richard Breen ed., 2004. Social Mobility in Europe. Oxford University Press, Oxford.

41. Sources: Richard Breen. 2004. Social Mobility in Europe. Oxford University Press, Oxford.

42. Youth cohorts and transition from school to work surveys • Numerous surveys conducted in European countries • Statistical Offices, France, Italy, Norway … • Ministry of education, employment, statistical departments Denmark, Germany, France, Ireland, Scotland … • Government agencies (Unemployment, others…) Belgium, France … • Universities, UK… Source CEDEFOP, Christine Mainguet, La transition du système éducatif vers la vie active (The transition from school to work) Exploitation des données statistiques nationales (Statistical sources).

43. Employers-Employees • A guide to Linked Employer-Employee Data Sources in the EU and Beyond • See Tanvi Desai, London School of Economics, 2008

44. Linked Employer-Employee Data sources, examples of national surveys

45. The Impact of Social Security Contributions on Earnings: Evidence from administrative data in France, Germany, Netherlands an UK. A. Bozio, J. Grenet Research project 2011

46. Bozio Antoine. The Impact of Social Security Contributions on Earnings: Evidence from administrative data in France, Germany, Netherlands an UK. Research proposal submitted in September 2011 to the Open Research Area (ORA) call

47. V. National and European microdata: where can I find the information and metadata Sources National microdata dissemination Nesstar Microdata dissemination at European level

48. Sources • Sources of information : NSI websites and CessdaPPP • 1 National Statistical Institute selected per country • These are general trends, there could be little variations due to lack of information available on the web or very recent changes

49. Metadata dissemination for national microdata: NSIs and Archives

50. Metadata dissemination at European level • Each European body • CESSDA only for some national microdata (depending on Archives members) • No single point of access even at national level • Some countries are opening portal for access to official data (open data initiatives) yet mostly for aggregate data • In some countries, archives gather metadata from different government producers (yet not all)