Session 6 Quality assurance in population and housing censuses UNSD presentation

1. Session 6 Quality assurance in population and housing censuses UNSD presentation

2. Overview • Importance of a quality assurance programme • The quality assurance circle • The role of managers • Dimensions of quality • Quality assurance by census phase/major activity • Evaluation • Process • Data

3. Importance of a quality assurance programme • P&R recommends that “Each country must have a quality assurance and improvement programme in place to measure the quality of each stage of the census” (P&R 2.169) • A major objective of a quality assurance programme is to ensure that quality assessment is consistently incorporated in all phases of the census, focusing on efforts in controlling the occurrence of errors and taking remedial actions to ensure the highest quality of both the processes and their outcomes. • A quality assurance programme should also be viewed as a quality improvement programme, and without such a programme, the census data when finally produced may contain many errors, which can severely diminish its usefulness • The quality assurance and improvement system should be developed as part of the overall census programme - integrated with other census plans, schedules and procedures, - and established at all phases of census operations, including planning, pre‐enumeration, enumeration, coding, data capture, editing, tabulation and data dissemination.

4. Importance of a quality assurance programme • Quality is the outcome of processes, and deficiencies in quality (for example, delays in processing or lack of accuracy in the results) are usually the result of deficiencies in process rather than the actions of individuals working in that process. • Quality is relative, and is based on what is acceptable to data users, or fit for the purpose, rather than on a concept of absolute perfection • The key to quality assurance and improvement is the ability to regularly measure the timeliness and accuracy of a given process so that errors are prevented from reoccurring, to detect errors easily and inform the workers so that they do not continue • this simple feedback loop is represented in the “quality assurance circle”

5. Measure quality Implement corrective action Identify most important problems Identify causes of problems Quality assurance circle The quality assurance circle is a schematic representation of the iterative process by which quality is improved Quality assurance circle is particularly applicable to tasks that are highly repetitive such as the processing phase of the census It is less applicable in processes that are one-off or time-constrained (eg. enumeration) as there is less opportunity to measure performance, identify problems and implement corrective actions The emphasis of the quality circle is on improving the process that caused the “error”, which may be any of the cost, timeliness or accuracy attributes falling below specified levels.

6. The role of managers • Managers have a vital role in establishing quality – their main roles include: • Establishing a culture within the census agency that has a focus on quality issues and to obtain the commitment of staff to strive to achieve high‐quality goals • Creating an environment in which everyone has the opportunity to contribute to quality improvement • Ensuring that clients’ expectations are known and that these expectations are built into planning objectives and into the systems that are to deliver them • Ensuring that processes for implementing quality assurance programmes are documented and such documentation provide information on: • how quality is going to be measured • who is involved in identifying root causes of problems with quality, • how the process improvements are going to be implemented

7. Dimensions of quality • Quality is a multidimensional concept • Outputs of any statistical exercise should possess some or all of the following six main attributes: • Relevance • Accuracy • Timeliness • Accessibility • Interpretability • Comparability • Some of these dimensions are inter-dependent and involve trade-off (eg. timeliness and accuracy) • Additional dimensions of quality: coherence, completeness

8. Dimensions of quality: Relevance • The relevance of statistical information is the degree to which it meets the needs of users -- and suggests the need to avoid the collection and production of data for which there is no significant use • This dimension is important in census content development and dissemination • Relevance is a qualitative assessment of the value of the census data produced, including in terms of meeting the mandate of the agency, legislated requirements, user needs

9. Dimensions of quality: Accuracy • The accuracy of statistical information is the degree to which those data correctly estimate or describe the quantities or characteristics that the statistical activity was designed to measure • It is usually characterized in terms of errors in statistical estimates introduced by major sources of errors such as coverage, sampling, non-response, processing, etc

10. Dimensions of quality: Timeliness • Timeliness refers the length of time between the census reference day and the date on which the information becomes available • It represents the degree to which information is released in a time period that still permits the information to be of value to users • It often involves a trade-off with accuracy • Census results are often made available over several release dates---so to provide an assessment of timeliness, major information releases should have specified publication dates in the dissemination schedule

11. Dimensions of quality: Accessibility • The accessibility of statistical information refers to the ease with which it can be obtained • Takes into account the suitability of the form in which the information is available to users, the media of dissemination • Availability of metadata • Where data products are provided at cost, the affordability of the information to users also affects accessibility

12. Dimensions of quality: Interpretability • The interpretability of statistical information reflects the availability of supplementary information and metadata necessary to interpret and use it • Usually it covers the underlying concepts, definitions, classifications used, the methodology of data collection and processing and indications of the accuracy of the information

13. Dimensions of quality: Comparability • The comparability of statistical information reflects the degree to which statistical information is comparable across countries, regions within a country, and time • Usually underlying concepts, definitions, classifications used, the methodology of data collection and processing provide information on comparability

14. Additional dimensions of data quality Coherence • Coherence reflects the degree to which the census information can be successfully brought together with other statistical information within an integrated framework over time • The use of standard concepts, definitions and classifications – possibly agreed at the international level - promotes coherence Completeness – an extension of relevance • Completeness reflects the degree to which statistics serve the needs of users as completely as possible, taking limited resources and respondent burden in to account

15. Quality assurance by census phase/major activity • Quality assurance can be applied to the entire census cycle with: • Performance in the previous phase being evaluated at any  given level of detail;  • Problems with quality ranked in order of importance;  • Root  causes  identified  and  corrective  action  implemented • The following slides outline the ways in which the concept of the quality circle is applied to the census cycle, with focus on some procedures for improving the quality of important phases or major activities

16. Quality assurance by census phase/major activity • Topic selection • The first step in managing the quality of census statistics to ensure that the data product will be relevant to users and meets requirements outlined in census legislations • The key process is extensive consultation with actual and potential users • Consultation with users would include: consultations with key government departments and agencies; advice from professional advisory committees in major subject matter areas; user feedback; ad hoc consultations with interested groups; etc.

17. Quality assurance by census phase/major activity Questionnaire design and testing • The next quality management task concerns the testing of each census question and the testing of the design of the form. • There are several key internal stakeholders of the questionnaire design process, whose engagement is critical for a successful outcome: • The dissemination team -- to ensure that the questions asked will deliver the data to meet the needs of users; • The subject matter specialist team; • The team responsible for development of the data capture or processing system - especially for data collection using scanning systems or an electronic questionnaire • The field operations team -- which is responsible for training the enumeration workforce and printing the form; • The respondents -- to ensure that the forms are easy to complete in mail‐out/mail‐back as well as Internet based self enumeration

18. Quality assurance by census phase/major activity Printing and distribution of census questionnaires/materials • Estimating the number of questionnaires and census materials is critical step for cost-effective way of printing census materials • Establishment of a system for monitoring the quality of the work done by the printing company is necessary to conduct: • Regular checks of the quality of the printed documents • Monitoring the progress in printing • Monitoring distribution of census materials from printing house to final destination – by number of questionnaire and other census materials

19. Quality assurance by census phase/major activity Recruitment and training of the field staff • Establishing: • Criteria for selection of enumerators/supervisors • Clear procedures for recruitment • Ensuring a standard training materials and programme • Monitoring and supervision of training and recruitment of enumerators/supervisors by relevant census supervisors

20. Quality assurance by census phase/major activity Enumeration • Checking the work of enumerators - coverage/content • Collecting periodical information from enumerators/ supervisors to assess the progress in enumeration • Collecting information on number of enumerated population/housing units, refusal, housing units with no contact • Monitoring non-response and follow-up rates • Assessment for the risk of not completing enumeration as scheduled • Special procedures for refusal/housing units with no contact

21. Quality assurance by census phase/major activity Data capture, coding and editing • Processing procedures should be developed with a view to minimizing the risk of erroneously cancelling, losing or artificially creating households during all phases of data processing • Developing procedures for monitoring the quality of each phase • Repeating certain procedures based on the sample of batches/records and comparing two datasets • Identifying systematic errors • Assessment of the quality of the procedures and difficulties faced

22. Quality assurance by census phase/major activity Dissemination • The dissemination area is responsible for the timely delivery of products and services to the census data users • Therefore insufficient planning and resources for this phase can have the effect of delaying the release of the data and thus compromising the overall achievement of the census objectives • Management of the quality in census dissemination is driven by concerns to: • deliver relevant products and services while maintaining accuracy of the data, and timeliness and predictability of data release within agreed cost constraints

23. Evaluation • P&R recommends that “a complete evaluation takes place and is documented at the end of each phase of the census - particularly for phases such as enumeration, so that the organizational learning is carried forward to the next census” • Evaluation of the overall census operation is vital for identifying strengths and weaknesses of census phases, including planning, enumeration, data processing and dissemination, and also for the purpose of analysing the quality of census statistics, which are the major output of these processes. • A comprehensive evaluation programme should include: • Evaluation of census processes – operational assessments • Evaluation of data quality - assessment of coverage and content errors • The results of evaluations of census operation for both operational aspects and the quality of data should be made available to the stakeholders

24. Evaluation – Operational aspects/processes • Operational assessments provide valuable information on strengths and weaknesses of past operational procedures which should be carefully reviewed prior to the development of the next census • Operational assessments should: • document operational errors • explain the effectiveness of operations and procedures and their likely impact on overall quality of census • Census evaluation with all dimensions of quality requires a comprehensive evaluation programme for assessing and documenting the outcomes of each process using appropriate and customized methodologies – these methodologies should be planned well in advance, in the planning phase of the census

25. Evaluation – Operational aspects/processes • P&R recommends that “the census evaluation programme should be undertaken by subject specialists according to the agreed goals and methodologies covering all possible dimensions of quality.” Some areas for evaluation include: • Identification of the deficiencies and achievements in data capture, coding and editing; • Relevance of census data to user needs and satisfaction of users with dissemination tools and products (based on information collected through user consultation); • Achievements and difficulties in use of new technologies and methodologies and identification of improvements for the next census; • Realization of the census calendar, including the calendar of releasing census results, and, in the case of changes to the calendar, the reasons and consequences.

26. Evaluation – Data quality • P&R recommends that “Evaluation of the accuracy of the census data should be undertaken, to the extent possible, by conducting a post‐enumeration survey for measuring coverage and content errors, by comparing the census results with similar data from other sources (surveys and administrative records in a similar time frame & previous census results) and by applying demographic analysis” • The purposes of evaluating the accuracy of the data are to inform users on the quality of the current census data and to assist in future improvements. • Evaluation of data accuracy may enable the identification of any problem areas that have not been previously detected through the quality management processes in earlier phases of the census

27. Conclusions • Quality assurance and improvement systems should be developed as part of the overall census programme, and integrated with other census plans, schedules and procedures • The systems should be established at all phases of census operations, including planning, pre‐enumeration, enumeration, document flow, coding, data capture, editing, tabulation and data dissemination. • Quality management procedures for each phase/major activity of the census should be assessed with appropriate methods