Emilio Di Meglio and Emanuela Di Falco (EUROSTAT)

1. Continuous improvement of EU-SILC quality: standard error estimation and new quality reporting system Emilio Di Meglio and Emanuela Di Falco (EUROSTAT) Q2014 Conference Vienna

2. Why variance estimation? • Requested by regulation • Quality report • Compliance • Requested by users • Policy relevance of indicators • Requested by researchers Q2014 Conference Vienna

3. Current legal requirements • According to Reg.1982/2003, the X and L (initial sample) data are to be based on a nationally representative probability sample of the population residing in private households. • Representative probability samples shall be achieved both for households and for individual persons in the target population. • The sampling frame and methods of sample selection should ensure that every individual and household in the target population is assigned a known and non-zero probability of selection. • Reg. 1177/2003 defines the minimum effective sample sizes to be achieved. Q2014 Conference Vienna

4. Main challenges for EU SILC • Difficulty to find the « best » possible method for variance estimation at Eurostat level • Different designs (flexibility) • Missing information • Debate on methodsongoing • Differentiate the needs: accuracyestimates for policy usage and accuracyestimates for researchers. Q2014 Conference Vienna

5. Sampling design by country (2012) Q2014 Conference Vienna

6. Our objective • Resampling taking into account all the possible elements coming from 32 countries would be extremely computationally and resource intensive • Variance estimation methods balancing between scientific accuracy and administrative considerations (time, cost, simplicity) are the only viable solution • Aim: to quickly provide to users and policy makers standard errors for the SILC-based indicators, particularly the AROPE (At-Risk-Of-Poverty or social Exclusion), its components and its main breakdowns. Q2014 Conference Vienna

7. The method (synthesis) • Linearization is a technique based on the use of linear approximation to reduce non-linear statistics to a linear form, justified by asymptotic properties of the estimator (Särndal et al, 1992 ; Deville, 1999 ; Wolter, 2006 ; Osier, 2009) • The "ultimate cluster" approach (Särndal et al, 1992) is a simplification consisting in calculating the variance taking into account only variation among Primary Sampling Unit (PSU) totals • This method requires first stage sampling fractions to be small which is nearly always the case. • This method allows a great flexibility and simplifies the calculations of variances. • It can also be generalized to calculate variance of the differences of one year to another (Berger, 2004 , 2010 ). • Applicable with the main statistical packages (SAS, R, STATA) Q2014 Conference Vienna

8. Results on AROPE • For 6 countries 95% Confidence Interval for AROPE equal or smallerthat ±1.0% (CZ, IT, SI, DE, FI, NO) • For 9 countries 95% Confidence Interval for AROPE between ± 1% and ±1.5% (ES, PL, UK, EE, AT, SK, CH, SE, IS) • For 8 countries 95% Confidence Interval for AROPE between ±1.5% and ±2% (BE, DK, HR, HU, NL, PT, CY, MT) • For 6 countries 95% Confidence Interval for AROPE largerthan ±2% (BG, EL, IE, RO, LT, LV) • Complete results in EU-SILC quality report Q2014 Conference Vienna

9. Measurement of net changes To measure the significance of the evolution of social indicators Example: When the At-risk-of-poverty or social exclusion rate for Poland goes from 27.2% in 2011 to 26.7% in 2012, are we able to say that this change is significant? Exercise already done for AROPE and other main EU-SILC indicators Q2014 Conference Vienna

10. Output Q2014 Conference Vienna

11. EU-SILC Quality reports(Reg. No 1777/2003) At national level, Member States have to produce: • An Intermediate QR (by the end of the year N+1) Based on cross-sectional data of year N • A Final QR(by the end of the year N+2) Based on longitudinal and cross-sectional data year N At European level, EUROSTAT has to produce : • EU Comparative Intermediate QR(by June of the year N+2) Based on the national Intermediate QRs • EU Comparative Final QR(by June of the year N+3) Based on the national Final QRs Q2014 Conference Vienna

12. Quality reporting Revision process • New template (ESQRS) • Revision of the Contents • Introduction of annexes and questionnaire • ESS Metadata Handler (old NRME) Q2014 Conference Vienna

13. EU SILC key quality dimensions • Accuracy • Comparability • Coherence  National ESQRS • Cost and burden • Statistical processing • Timeliness and punctuality • Relevance  EU ESQRS Q2014 Conference Vienna

14. Availability of quality metadata • Quality reports • Questionnaires • Methodological papers Further action: integrate more information in a wiki platform Q2014 Conference Vienna

15. Conclusion and future plans • The variance estimation methodology is of relatively simple application • It can be considered as a good compromise between scientific soundness and feasibility under current constraints. • The next steps consist in still improving these calculations by asking Member States to provide the necessary information where missing. • Dissemination of further information to users. • Betterdisseminatequality reports Q2014 Conference Vienna