1 / 41

The Data Quality Assessment Framework

The Data Quality Assessment Framework. OECD Meeting of National Accounts Experts October 2001. Purpose of this Presentation. To describe: The IMF’s Data Quality Assessment Framework (DQAF), and

jfairley
Download Presentation

The Data Quality Assessment Framework

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The Data Quality Assessment Framework OECD Meeting of National Accounts Experts October 2001

  2. Purpose of this Presentation To describe: • The IMF’s Data Quality Assessment Framework (DQAF), and • Experience to date with the DQAF for Reports on Observance of Standard and Codes (ROSCs) and beyond.

  3. Plan for Presentation • Origins of DQAF • DQAF Approach • Framework: what is it? • Process: how was it developed? • Draft framework: an overview • The DQAF suite of assessment tools • The work ahead • Links to SDDS/GDDS • Working with the DQAF

  4. Origins of Recent Work • SDDS and GDDS: broadening the scope of data standards to strengthen the link with data quality • Provision of data by members to the IMF: a need to be clearer about what is called for • ROSC’s: a need for an even-handed approach to assessing data quality

  5. Increased Interest in Data Quality More widely, interest in quality follows from explicit use of statistics in policy formulation and goal setting: • Inflation targeting (spotlight on CPI) • Stability Pact in the context of EMU (spotlight on debt/deficit ratios to GDP) • UN Conferences on Least Developed Countries (inclusion and graduation from list is based on specified economic indicators)

  6. The IMF’s Approach

  7. The IMF’s Approach • Data Quality Reference Site at the IMF’s Dissemination Standards Bulletin Board http://dsbb.imf.org/dqrsindex.htm • The Site provides an introduction to the topic of data quality and includes a selection of reference materials and articles on data quality issues.

  8. The DQAF: What is its Purpose? • Its potential uses • To guide data users—to complement the SDDS and GDDS • To guide IMF staff • in assessing data for IMF surveillance and operations, • in preparing ROSCs, and • in designing Technical Assistance • To guide country efforts (self-assessment)

  9. The DQAF: Requirements • Given these differing potential uses, the framework should be: • Comprehensive • Balanced between experts’ rigor and generalists’ bird’s-eye view • Applicable across various stages of statistical development • Applicable to the major macroeconomic datasets • Designed to give transparent results • Arrived at by drawing on national statisticians’ best practices

  10. The DQAF : What Is It? Generic etc. Dataset-Specific etc. etc. GFS BOP NA

  11. How the DQAF Was Developed • We engaged a national statistical office to help develop the generic framework • In parallel, IMF staff worked on frameworks for several datasets • National accounts was reviewed in June 2000 • National accounts (revised) and four other specific frameworks were circulated informally in the international statistical community for comment in August-September 2000

  12. How the DQAF Was Developed • Drafts were discussed in topical or regional meetings, e.g. • East Asian Heads of NSOs • ECB Working Group on Money and Banking Statistics • IMF BOP Statistics Committee • GFS Expert Group meeting

  13. How the DQAF Was Developed • IMF staff tested the frameworks in the field • A paper for the Statistical Quality Seminar in December 2000 presented: • Revised generic framework • Revised BOP dataset-specific framework • Alternatives for a preview (“lite”) tool • Sample summary presentations of results To access the paper: http://dsbb.imf.org/dqrsindex.htm

  14. DQAF: an Overview • Uses a cascading structure • Five dimensions of quality - and for each dimension, • Elements that can be used in assessing quality - and for each element, • Indicatorsthat are more concrete and detailed - and for each indicator, • Focal issues that are tailored to the dataset • - and for each focal issue • Key points

  15. DQAF: an Overview The five dimensions of the IMF’s Data Quality Assessment Framework 1. Integrity 2. Methodological soundness 3. Accuracy and reliability 4. Serviceability 5. Accessibility

  16. DQAF: an Overview • Also, some elements/indicators are grouped as “prerequisites of quality” • Pointers that are relevant to more than one of the five dimensions • Generally refer to the umbrella agency • Example: quality awareness

  17. Prerequisites for Quality • Legal and institutional framework • Roles and responsibilities of statistical agencies • Data sharing and coordination between data producing agencies • Access to administrative and other data for statistical purposes • Nature of reporting • Resources • Quality awareness

  18. Elements of Integrity • Professionalism • Transparency • Ethical standards

  19. Elements of Methodological Soundness • Concepts and definitions • Scope • Classifications • Basis for recording: accounting rules and valuation principles

  20. Elements of Accuracy • Source data • Statistical techniques: compilation procedures and statistical methods and adjustment • Assessment and validation

  21. Elements of Serviceability • Relevance of the national accounts program • Timeliness and periodicity • Consistency • Revision policy and practice

  22. Elements of Accessibility • Data accessibility • Metadata accessibility: documentation • Assistance to users: service and support

  23. Indicators of Consistency • Temporal consistency • Internal consistency • Intersectoral consistency

  24. Focal Issues for Internal Consistency • Internal consistency of the annual accounts • Internal consistency between quarterly and annual estimates

  25. Key Points Internal Consistency of the National Accounts • Discrepancies between approaches shown? • Size of discrepancies? • Differences between growth rates? • Supply and use framework applied? • Do total supply and use match? • Does net lending/borrowing match between sectors?

  26. General Reactions • “Welcome initiative” • “Fills important gap” • “Is careful and thoughtful” • “Provides basis for coherent and practical way forward in a complex field”

  27. General Reactions • Some other points • Is the framework really operational for small countries? • Can it be used without giving a “black mark” for points that are irrelevant to a country? • Is the framework able to identify “poor” statistics prepared within a developed statistical system?

  28. General Reactions • Some other points (cont’d) • Expand the range of datasets covered • Coordination with other organizations working on data quality is important • Continue working in a consultative manner

  29. The DQAF Suite of Tools • DQAF “Lite” • Background: interest in a version that might serve as a diagnostic preview or for a non-statistician’s assessment • IMF is field testing a “Lite” made up of 13 indicators.

  30. The DQAF Suite of Tools • Summary presentation of results • Background: Interest in a presentation of results for, e.g., policy advisors • IMF is testing a summary presentation • For each dataset, a one-page table • At the two-digit level (21 elements) • On a 4-point scale, from “practice observed” to “practice not observed” • With an “n.a.” column • With a “comments” column

  31. Data Quality Assessment FrameworkSummary for [dataset]  Note: O = Practice Observed; LO = Practice Largely Observed: MNO = Practice Materially Nonobserved; NO = Practice Nonobserved; NA= Not Applicable Comment: only if different from O.

  32. The DQAF Suite of Tools Generic (3-digit) “Lite” Summary of Results etc. Dataset (6-digit) etc. DatasetSpecific (5-digit) etc. GFS BOP NA

  33. Work ahead • Test the suite • in a wider range of country situations • especially with non-statisticians • Refine and revise the suite • Complete supporting materials • A Glossary • Supporting Notes for specific datasets • A Methodology (a how-to-do-it guide) • Develop frameworks for other datasets

  34. Links to the SDDS/GDDS Summary: The DQAF complements the SDDS/GDDS • All of the elements of the SDDS/GDDS are also found within the DQAF

  35. Links to SDDS/GDDS • The purpose and scope of the SDDS/GDDS and DQAF differ: • In SDDS/GDDS, as dissemination standards, quality is a dimension. • That dimension takes an indirect approach to dealing with, e.g., accuracy--it calls for dissemination of relevant information. • In DQAF, as an assessment tool, quality is the umbrella concept. • That concept covers collection, processing, and dissemination of data.

  36. Links to SDDS/GDDS • The DQAF definition of “quality” has been brought into line with the emerging consensus that quality is a multidimensional concept. • Some aspects relate to the product • Some aspects relate to the institution

  37. Links to SDDS/GDDS • DQAF is “more active” in dealing with, e.g., conformity with international guidelines, accuracy, and reliability. • SDDS and to a lesser degree GDDS left users on their own to make judgments • DQAF guides users in making such judgments by providing two structured dimensions: • Methodological soundness • Accuracy and reliability

  38. Working with the DQAF • The earlier list of potential uses of the DQAF included “To guide IMF staff “ • Largely this refers to staff of the IMF Statistics Department • Interrelated uses: • in assessing data for IMF’s use in surveillance and operations, • in preparing ROSCs, and • in designing technical assistance

  39. Working with the DQAF • We are now using the DQAF in the field • In capacity building advisory missions • In ROSCs

  40. Working with the DQAF • What do we see from the experiences? • Advantages • Provides more structure to technical assistance • Promotes consistency across staff/experts • Potentially provides input for useful database • Places data standards in the center of work on the international financial architecture • Challenges • Puts premium on consistency • Calls for explicit judgments

More Related