Quality Frameworks: Implementation and Impact

1. Quality Frameworks:Implementation and Impact Notes by Michael Colledge

2. Terms of Reference • Some organisations are using quality frameworks others are not. Does it really make a difference? • What is important, the existence of a quality framework, support of top management, quality of the organisation?

3. Content • What is a quality (assurance) framework? • Who are stakeholders in a quality framework? • What are costs and benefits? • Which organisations have a quality framework? • What is impact on quality of a quality framework? • To what extent can quality frameworks converge?

4. What is a quality (assurance) framework? • Umbrella for quality practices • Refers to survey programme rather than single survey • Covers range of survey operations rather than single subject • Involves template that can be used for quality assessment

5. Scope of a quality framework? Transformation Output Input

6. Scope of a quality framework? • Product (output data, services) • Process (collection transformation, dissemination) • Input data • OrganisationDifference in scope limits convergence

7. Dimensions of Data Quality • Generic set: • Relevance, accuracy, timeliness, accessibility, interpretability, comparability, coherence • Could be standardised • Could be validated by feedback from users • Check against ISO 20252

8. Quality Effectiveness Product “Doing the right thing” Quality indicators Performance Efficiency Process “Doing the thing right” Performance indicators The other dimension:cost/performance

9. Quality Related Mechanisms and Topics • Mechanisms • Review (self, peer, external) • Reporting (indicators, reports) • Labelling, declarations • Sharing best practices • Topics • Metadata management • Programme monitoring /evaluation, audit • TQM • Management

10. Types of Quality Framework by Owner and Applier NSO owned and applied • Statistics Canada quality framework • IO owned and applied • OECD quality framework • IO owned NSO applied • IMF DQAF

11. Types of Quality Framework by Owner and Applier • NOS owned IO applied • What value does IO add? • Is every data item actually used? • What efforts have been made to harmonise across IOs? • How does IO quality assure its outputs? • Questions would be answered by IO owned and applied framework

12. Stakeholders in Quality Framework • Designers/owners • Organisational units that apply framework • Users of data collected via framework template

13. Costs and Benefits • Benefits • Increased awareness of quality concepts, components, best practices • Systematic quality assessment • Identification of quality problems • Comparisons over time, across organisational units • Indications of additional resource and training needs • Costs - staff time

14. Which Organisations Have Quality Frameworks? • National • International/Supranational

15. Impacts of Quality Frameworks? • Statistics Canada Quality Guidelines • Statistics Canada Quality Framework • ESS Self-Assessment Checklist (DESAP) • OECD Quality Framework • IMF Data Quality Assessment Framework (DQAF) • Australian Statistical Clearing House

16. Impacts of Quality Frameworks: Conclusions • Quality frameworks serve range of useful purposes • Does not have to be developed from scratch • Can adapt existing ones (best practice) • Development process itself has benefits

17. Convergence of Quality Frameworks • Benefits • Common terminology • Better understanding of concepts • Promoting current best practice • Reducing reporting burden • Limitations • Differing objectives, scope, users • For example, process orientation v product orientation