COMPARISON OF VALIDATION PROCEDURES TO DETECT MEASUREMENT ERRORS IN AN AREA FRAME SAMPLE SURVEY

1. COMPARISON OF VALIDATION PROCEDURES TO DETECT MEASUREMENT ERRORS IN AN AREA FRAME SAMPLE SURVEY Laura Martino, Marco Fritz, Marjo Kasanko & Javier Gallego European Conference on quality in official statistics, Rome 8-11th July 2008 European Conference on Quality

2. Contents • Scope of the work • Methods for measurement errors detection • The case study: LUCAS survey • Conclusions European Conference on Quality

3. Scope of the work The scope of the work is comparing the efficacy of three selected procedures to detect measurement errors in an area frame sample survey that uses a combination of orthophoto interpretation and ground survey European Conference on Quality

4. Methods for detection of measurement errors Three methods have been selected to be compared: • double-blind survey; • check of ground data with ausiliary data sources; • cross-check of orto-photo interpretation and ground-survey outcomes to evaluate the accuracy of stratification European Conference on Quality

5. Case study: Land Use and Coverage Area frame sample Survey (LUCAS) • Area frame sample survey carried out by Eurostat since 2001. • Main objectives are providing: • coherent and harmonised statistics on land use and land cover; • b) a common sampling base (frame, nomenclature, data treatment) to be used for further scopes; • c) ground evidence for calibration of satellite images; • d) information on aspects relating to the agro- environment European Conference on Quality

6. East Second phase sample:Ground survey During the ground survey, surveyors: • Reach the point using GPS and cartographic material • Record land cover and land use according to the full nomenclature • Take pictures (Point, Cov,N,E,S,W,irr) • Walk the transect (250 m. East) European Conference on Quality

7. 1° procedure: Double blind survey A second survey was organised in 2006 in parallel to the main LUCAS 2006 survey. • It was conducted by an independent company not in charge of the LUCAS survey in the countries involved. • No information were provided on the results of the main survey. • The double blind-survey sample size represented 5% of the total survey sample and counted almost 8200 points. European Conference on Quality

8. Outcomes of double-blind survey The possible outcomes of the double-blind survey were: • Land Cover/Land Use being the same between the two surveys (correct points observed from the same location); • Land Cover/Land Use being different between the two surveys but both correct (different rules of observation - look north, east, etc. - or change in the land cover between the two visits (crops harvested, sown, building built, etc...); • Land Cover/Land Use being different between the two surveys and lack of sufficient information to say which survey is correct; • Land Cover/Land Use being different between the two surveys and the double-blind survey being correct; • Land Cover/Land Use being different between the two surveys and the main survey being correct. European Conference on Quality

9. Double blind survey D-B survey 8,200 pts Main survey 169,000 pts D-B survey introduced as many errors as the main survey European Conference on Quality

10. 2° procedure: Use of ausiliary data to correct results • IACS data • Pictures • Other georeferenced information Large improvement in data quality through the use of pictures European Conference on Quality

11. 3° procedure: comparison of orthophoto interpretation and ground survey A comparison of the classification of the points according to the ground observation and the photointerpretation has been conducted in a comparable nomenclature of 7 classes. • A proportion of agreement has been computed • Since points belonging to different strata have been subsampled with probability that could be 5 times larger/lower, a weighted proportion of agreement is computed in addition to the unweighted one. European Conference on Quality

12. European Conference on Quality

13. Measurements of agreement • Frequency: The unweighted proportion of agreement is 70.8%, while the weighted agreement is 74.8%. • Kappa index (Bishop et al., 1975) where are the proportions of each cell of the table The unweighted Kappa is 0.58 (0.66 if we take into account the strata weights). European Conference on Quality

14. Rate of agreement European Conference on Quality

15. Mainly confused items Some points that are agricultural land are photo-interpreted as non-agricultural for several reasons: location inaccuracy, land cover change or simply photo-interpretation errors The main confusions occured between: • permanent grass and bare land are often photo-interpreted as arable. • Woodland or shrubland interpreted as grass. This may be partly due to problems to apply in practice the definition of forest or woodland European Conference on Quality

16. Conclusions • The pre-eminent result of the double blind survey was that it introduced as much errors as the main survey for many reasons • The analysis of pictures and the use of auxiliary information appeared to be fundamental to detect measurement errors and improve data quality • Orthophoto interpretation produce some inefficiency (stratification accuracy) -> poststratification??? European Conference on Quality

17. THANK YOU FOR YOUR ATTENTION!!! European Conference on Quality