The Data Quality Assessment Framework

1. The Data Quality Assessment Framework OECD Meeting of National Accounts Experts October 2001

2. Purpose of this Presentation To describe: • The IMF’s Data Quality Assessment Framework (DQAF), and • Experience to date with the DQAF for Reports on Observance of Standard and Codes (ROSCs) and beyond.

3. Plan for Presentation • Origins of DQAF • DQAF Approach • Framework: what is it? • Process: how was it developed? • Draft framework: an overview • The DQAF suite of assessment tools • The work ahead • Links to SDDS/GDDS • Working with the DQAF

4. Origins of Recent Work • SDDS and GDDS: broadening the scope of data standards to strengthen the link with data quality • Provision of data by members to the IMF: a need to be clearer about what is called for • ROSC’s: a need for an even-handed approach to assessing data quality

5. Increased Interest in Data Quality More widely, interest in quality follows from explicit use of statistics in policy formulation and goal setting: • Inflation targeting (spotlight on CPI) • Stability Pact in the context of EMU (spotlight on debt/deficit ratios to GDP) • UN Conferences on Least Developed Countries (inclusion and graduation from list is based on specified economic indicators)

6. The IMF’s Approach

7. The IMF’s Approach • Data Quality Reference Site at the IMF’s Dissemination Standards Bulletin Board http://dsbb.imf.org/dqrsindex.htm • The Site provides an introduction to the topic of data quality and includes a selection of reference materials and articles on data quality issues.

8. The DQAF: What is its Purpose? • Its potential uses • To guide data users—to complement the SDDS and GDDS • To guide IMF staff • in assessing data for IMF surveillance and operations, • in preparing ROSCs, and • in designing Technical Assistance • To guide country efforts (self-assessment)

9. The DQAF: Requirements • Given these differing potential uses, the framework should be: • Comprehensive • Balanced between experts’ rigor and generalists’ bird’s-eye view • Applicable across various stages of statistical development • Applicable to the major macroeconomic datasets • Designed to give transparent results • Arrived at by drawing on national statisticians’ best practices

10. The DQAF : What Is It? Generic etc. Dataset-Specific etc. etc. GFS BOP NA

11. How the DQAF Was Developed • We engaged a national statistical office to help develop the generic framework • In parallel, IMF staff worked on frameworks for several datasets • National accounts was reviewed in June 2000 • National accounts (revised) and four other specific frameworks were circulated informally in the international statistical community for comment in August-September 2000

12. How the DQAF Was Developed • Drafts were discussed in topical or regional meetings, e.g. • East Asian Heads of NSOs • ECB Working Group on Money and Banking Statistics • IMF BOP Statistics Committee • GFS Expert Group meeting

13. How the DQAF Was Developed • IMF staff tested the frameworks in the field • A paper for the Statistical Quality Seminar in December 2000 presented: • Revised generic framework • Revised BOP dataset-specific framework • Alternatives for a preview (“lite”) tool • Sample summary presentations of results To access the paper: http://dsbb.imf.org/dqrsindex.htm

14. DQAF: an Overview • Uses a cascading structure • Five dimensions of quality - and for each dimension, • Elements that can be used in assessing quality - and for each element, • Indicatorsthat are more concrete and detailed - and for each indicator, • Focal issues that are tailored to the dataset • - and for each focal issue • Key points

15. DQAF: an Overview The five dimensions of the IMF’s Data Quality Assessment Framework 1. Integrity 2. Methodological soundness 3. Accuracy and reliability 4. Serviceability 5. Accessibility

16. DQAF: an Overview • Also, some elements/indicators are grouped as “prerequisites of quality” • Pointers that are relevant to more than one of the five dimensions • Generally refer to the umbrella agency • Example: quality awareness

17. Prerequisites for Quality • Legal and institutional framework • Roles and responsibilities of statistical agencies • Data sharing and coordination between data producing agencies • Access to administrative and other data for statistical purposes • Nature of reporting • Resources • Quality awareness

18. Elements of Integrity • Professionalism • Transparency • Ethical standards

19. Elements of Methodological Soundness • Concepts and definitions • Scope • Classifications • Basis for recording: accounting rules and valuation principles

20. Elements of Accuracy • Source data • Statistical techniques: compilation procedures and statistical methods and adjustment • Assessment and validation

21. Elements of Serviceability • Relevance of the national accounts program • Timeliness and periodicity • Consistency • Revision policy and practice

22. Elements of Accessibility • Data accessibility • Metadata accessibility: documentation • Assistance to users: service and support

23. Indicators of Consistency • Temporal consistency • Internal consistency • Intersectoral consistency

24. Focal Issues for Internal Consistency • Internal consistency of the annual accounts • Internal consistency between quarterly and annual estimates

25. Key Points Internal Consistency of the National Accounts • Discrepancies between approaches shown? • Size of discrepancies? • Differences between growth rates? • Supply and use framework applied? • Do total supply and use match? • Does net lending/borrowing match between sectors?

26. General Reactions • “Welcome initiative” • “Fills important gap” • “Is careful and thoughtful” • “Provides basis for coherent and practical way forward in a complex field”

27. General Reactions • Some other points • Is the framework really operational for small countries? • Can it be used without giving a “black mark” for points that are irrelevant to a country? • Is the framework able to identify “poor” statistics prepared within a developed statistical system?

28. General Reactions • Some other points (cont’d) • Expand the range of datasets covered • Coordination with other organizations working on data quality is important • Continue working in a consultative manner

29. The DQAF Suite of Tools • DQAF “Lite” • Background: interest in a version that might serve as a diagnostic preview or for a non-statistician’s assessment • IMF is field testing a “Lite” made up of 13 indicators.

30. The DQAF Suite of Tools • Summary presentation of results • Background: Interest in a presentation of results for, e.g., policy advisors • IMF is testing a summary presentation • For each dataset, a one-page table • At the two-digit level (21 elements) • On a 4-point scale, from “practice observed” to “practice not observed” • With an “n.a.” column • With a “comments” column

31. Data Quality Assessment FrameworkSummary for [dataset]  Note: O = Practice Observed; LO = Practice Largely Observed: MNO = Practice Materially Nonobserved; NO = Practice Nonobserved; NA= Not Applicable Comment: only if different from O.

32. The DQAF Suite of Tools Generic (3-digit) “Lite” Summary of Results etc. Dataset (6-digit) etc. DatasetSpecific (5-digit) etc. GFS BOP NA

33. Work ahead • Test the suite • in a wider range of country situations • especially with non-statisticians • Refine and revise the suite • Complete supporting materials • A Glossary • Supporting Notes for specific datasets • A Methodology (a how-to-do-it guide) • Develop frameworks for other datasets

34. Links to the SDDS/GDDS Summary: The DQAF complements the SDDS/GDDS • All of the elements of the SDDS/GDDS are also found within the DQAF

35. Links to SDDS/GDDS • The purpose and scope of the SDDS/GDDS and DQAF differ: • In SDDS/GDDS, as dissemination standards, quality is a dimension. • That dimension takes an indirect approach to dealing with, e.g., accuracy--it calls for dissemination of relevant information. • In DQAF, as an assessment tool, quality is the umbrella concept. • That concept covers collection, processing, and dissemination of data.

36. Links to SDDS/GDDS • The DQAF definition of “quality” has been brought into line with the emerging consensus that quality is a multidimensional concept. • Some aspects relate to the product • Some aspects relate to the institution

37. Links to SDDS/GDDS • DQAF is “more active” in dealing with, e.g., conformity with international guidelines, accuracy, and reliability. • SDDS and to a lesser degree GDDS left users on their own to make judgments • DQAF guides users in making such judgments by providing two structured dimensions: • Methodological soundness • Accuracy and reliability

38. Working with the DQAF • The earlier list of potential uses of the DQAF included “To guide IMF staff “ • Largely this refers to staff of the IMF Statistics Department • Interrelated uses: • in assessing data for IMF’s use in surveillance and operations, • in preparing ROSCs, and • in designing technical assistance

39. Working with the DQAF • We are now using the DQAF in the field • In capacity building advisory missions • In ROSCs

40. Working with the DQAF • What do we see from the experiences? • Advantages • Provides more structure to technical assistance • Promotes consistency across staff/experts • Potentially provides input for useful database • Places data standards in the center of work on the international financial architecture • Challenges • Puts premium on consistency • Calls for explicit judgments