Introduction

1. Improving the Data Quality monitoring framework (CCSA/Eurostat self assessment initiative) Case of FAO Producer Prices data (methodology and data quality self assessment):byCarola FabiFAO Statistics Division AFCAS meeting, Algiers

2. Introduction • The subject of data quality is consistently addressed in many international forums • countries have always addressed data quality from both practical and theoretical perspectives • growing consensus that policy and decisions are increasingly reliant on better data if they are to be effective • convergence of data quality models AFCAS meeting, Algiers

3. Paper outline • main differences between national and international criteria • data quality dimensions of particular relevance to agricultural price data • results of the self-assessment exercise on producer price data based on the CCSA checklist • suggestions for improving data quality AFCAS meeting, Algiers

4. 1. General concept of quality of official statistics All agree that statistics data quality is multi-dimensional, but there are differences in the criteria identified and in the vocabulary used to define the main data quality axes. AFCAS meeting, Algiers

5. General concept of quality of official statistics Eurostat’s 6 dimensions of data quality • Relevance: degree to which statistics meet users’ needs. • Accuracy: closeness of estimates or computations to the exact or true value. • Timeliness and punctuality: time lag between the reference period and the release data, time lag between the release date and the target date. AFCAS meeting, Algiers

6. 1. General concept of quality of official statistics Eurostat’s 6 dimensions of data quality • Accessibility and clarity: physical conditions in which data can be obtained, data’s information environment, whether they are accompanied by appropriate metadata; extent to which additional assistance is provided. • Comparability: measures of the impact of differences in concepts and procedures when statistics are compared between geographical area, non-geographical domains, or over time. • Coherence: statistics’ adequacy to be reliably combined in different ways and for various uses. AFCAS meeting, Algiers

7. 1. General concept of quality of official statistics • Statistics Canada and OECD call Interpretability what is Clarity for Eurostat (in particular metadata) • The IMF includes four dimensions under Serviceability and Accessibility and adds Pre-Requisites, Integrity, and Methodological Soundness (procedural dimensions for data quality). • FAO Statistics Division is adhering to the Eurostat taxonomy AFCAS meeting, Algiers

8. 2. Quality dimensions revised in an international context • Assessing international databases quality is an important exercise • Frameworks need to be adjusted • because global datasets cover heterogeneous countries • international organisations have a limited normative power on the methodologies implemented in the individual countries. AFCAS meeting, Algiers

9. 2. Quality dimensions revised in an international context • Shared Fundamental principles • Production of international data: • raw data is the output of national statistical offices • transformation/production process aims at adding value AFCAS meeting, Algiers

10. 2. Quality dimensions revised in an international context 3. Quality dimensions • At the national level, the debate on data quality is much more advanced than at the international level. • Many quality dimensions developed for national data sets carry over directly to international data sets. • There are at least two specific dimensions of quality, which apply at the international level: • (i) coverage - for how many countries/regions are data available and • (ii) comparability - to what extent is the information for different countries/regions comparable? AFCAS meeting, Algiers

11. 2. Quality dimensions revised in an international context 4. Degree of coordination/cooperation • harmonize practices • develop standards, (Importance of metadata and information technology) • create integrated international statistical system, with an appropriate division of labor AFCAS meeting, Algiers

12. 3. Characteristics of price data • Price data are generally multi-dimensional • A substantial amount of work has been done for collection of price data to compile CPI, PPI and fro calculating PPP’s • study poverty issues and international comparison of purchasing power. • These studies are oriented towards well organised consumer markets where agriculture is not clearly visible. • Price data concepts depends on the objective of data collection, frequency depends on the economic conditions at large AFCAS meeting, Algiers

13. 3. Characteristics of price data Questions for price data collection • What type of price? Can we collect actual transaction prices rather than list price? • Will we collect basic prices (excluding taxes, including subsidies on products, and excluding transport costs)? • At what time? • How to take into account product variety, quality? Frequency of price collection will depend on how frequently prices change. AFCAS meeting, Algiers

14. 3. Characteristics of price data Guiding principles for data collection • How tightly defined (narrow) or general (broad) the item specification should be an issue of great theoretical and practical importance. • Selection of items should be based upon a complete list of important crops and livestock product being produced by the country. • The rules for selection should also take account of the sampling methodology underlying the selection of markets/traders. AFCAS meeting, Algiers

15. 3. Characteristics of price data Guiding principles for data collection • Special attention to seasonal items. • Considering data collection less often than monthly for some items, thereby enlarging total sample. • Regular timing is particularly important when inflation is rapid. • Preferably days of the week and times of the month should be chosen taking into account socio-economic set-up. AFCAS meeting, Algiers

16. 3. Characteristics of price data Collection of price data is being done by countries either using a probability sampling techniques for getting unbiased estimates or using a purposive sampling selection method. Reasons for adopting non-probability sampling methods are: • No sampling frame is available, • Bias resulting from non-probability sampling is negligible • Possibility that the sample can not be monitored for long-time • Availability of staff • Sample size is too small • To give freedom to price collector to choose locally popular varieties. The representative item method is more suitable for homogeneous products. AFCAS meeting, Algiers

17. 4. Quality criteria applied to price data and problems with price data received in the FAO Relevance: • of FAOSTAT price data: the adopted definition stems from the SNA93 where producer prices enter into the compilation of value of production and economic accounts. • FAO Statistics division has not yet developed derived price-based indicators, whose relevance will have to be assessed in due course. AFCAS meeting, Algiers

18. 4. Quality criteria applied to price data and problems with price data received in the FAO Countries responses AFCAS meeting, Algiers

19. 4. Quality criteria applied to price data and problems with price data received in the FAO ACCURACY: Countries response rate AFCAS meeting, Algiers

20. 4. Quality criteria applied to price data and problems with price data received in the FAO ACCURACY: Ratio of official to estimated data AFCAS meeting, Algiers

21. 4. Quality criteria applied to price data and problems with price data received in the FAO Sources of inaccuracy: • doubt on price concept adopted • use of non standard-units AFCAS meeting, Algiers

22. 4. Quality criteria applied to FAO price data TIMELINESS AND PUNTUALITY AFCAS meeting, Algiers

23. 4. Quality criteria applied to FAO price data Time-lag all countries AFCAS meeting, Algiers

24. 4. Quality criteria applied to FAO price data Time-lag African countries AFCAS meeting, Algiers

25. 4. Quality criteria applied to FAO price data Accessibility and Clarity: • Available on-line and for free • Complemented by metadata on concepts and definitions, country notes, currencies AFCAS meeting, Algiers

26. 4. Quality criteria applied to FAO price data Comparability and Coherence: • Comparability is impaired by the complexity of prices that are a dynamic and multi-dimensional dataset • Countries change concept; six African countries replaced consumer prices with consumer prices • Lack of metadata to distinguish between comparability issues and revisions (accuracy) • Coherence will be strengthened in FAOSTAT with the development of derived indicators (Indices, value of production) AFCAS meeting, Algiers

27. 5. Self-assessment checklist on FAOSTAT’s price datasets • tool for the systematic quality assessment of statistics compiled by international or supranational organisations. • It is a work in progress based on tests and inputs from offices such as the FAO • It has been developed within the Committee for Coordination of Statistical Activities (CCSA) project, • on the use and convergence of international quality assurance frameworks AFCAS meeting, Algiers

28. 5. Self-assessment checklist on FAOSTAT’s price datasets • Generic checklist : it is designed to apply to statistics gathered by international organisations, irrespective of the subject matter area • Modular approach: to account for differences in statistical systems and functions of international organisations and enable to tailor it to specific needs. • Full exercise at intervals to identify strengths and weaknesses • Annual partial exercise to monitor quality over time AFCAS meeting, Algiers

29. 5. Self-assessment checklist on FAOSTAT’s price datasets Chapters 1. Background Information 2. Conceptual Frameworks 3. Users And Customers 4. Data Providers 5. Validation (country data) 6. Validation (international aggregates) 7. Data dissemination 8. Statistical confidentiality 9. It conditions 10. Documentation 11. Follow-up of the statistical production process AFCAS meeting, Algiers

30. 5. Self-assessment checklist on FAOSTAT’s price datasets SELECTED INDICATORS 3. Users And Customers 3/6 Availability of information on user satisfaction 3/8 Assess overall user satisfaction 5. Validation (country data) 5/2 Degree of completeness of received country data 5/3 Degree of completeness of received country metadata 5/7 Extent of unit non-response in country surveys 5/19 Distribution of countries by number of revisions 5/20Average size of revisions by the countries 6. Validation (international aggregates) 6/1 Necessity of editing received data 6/18 Average size of revisions by FAO 6/20 Overall assessment of disseminated data accuracy AFCAS meeting, Algiers

31. 5. Self-assessment checklist on FAOSTAT’s price datasets SELECTED INDICATORS 7. Data dissemination 7/12 Timeliness of preliminary results 7/13 Timeliness of final results 7/15 Is punctuality kept? 7/19 Coherence of data within the domain 7/20 Coherence in areas where it is applicable 7/23 Comparability over time 7/24 Extent of use of standard concepts and methodologies by countries 7/25 Asymmetries in mirror data 10. Documentation • 10/6 Assessment of the completeness of metadata AFCAS meeting, Algiers

32. AFCAS meeting, Algiers

33. USERS METADATA COMPLETENES • VALIDATION– COUNTRY DATA TIMELINESS VALIDATION – INTERNATIONAL DATA AFCAS meeting, Algiers

34. Summarizing quality – Assessment diagrams and FAO Data quality stamp Production Prices Meta data International classifications Update schedule Global coverage Integrated - Integrated Up-to-date AFCAS meeting, Algiers

35. 6. Suggestions for improvement • sound concepts and definitions followed by sufficient explanations on country practices • Capacity building for not only producing primary statistics but also secondary statistics to increase coherence of price data. • Establish a metadata system, which may provide details about approach (sampling design, questionnaire used, concepts and definitions) adopted by the countries to collect data. • Regular feedback between FAO and countries indicating inconsistency and discrepancy noted in the data. • Review all national price data sources and centralisation in one office to progressively increase coordination, improve resource uses, data comparability and coherence • Introduction in FAO’s annual questionnaire of check and pre-validation tools AFCAS meeting, Algiers