Evaluation and assessment in civil registration and vital statistics systems

1. Evaluation and assessment in civil registration and vital statistics systems Discussion, United Nations Statistics Division

2. Australia – Evaluating the quality of vital statistics: paper for discussion • Overarching assessment/evaluation framework - WHO and University of Queensland Health Information Systems Knowledge Hub • Several components covering inputs, processes and outputs • An assessment tool for independent evaluation studies • Legal framework/organizational structure/coverage/accuracy/quality check/data dissemination • The tool itself covers a sub-component that asks whether quality check is being conducted regularly (E.1)

3. Australia – Evaluating the quality of vital statistics: paper for discussion • Comments on Chapter IV of the Principles and Recommendations for a Vital Statistics System, Rev. 2 • developing an assessment framework • Technical and dense, may provide a set of guiding questions (see above framework) • Consider the following additional criteria when evaluating vital statistics • Relevance • Ensuring international comparability • Ensuring confidentiality of individuals

4. The vital statistics system in China • Three main sources of population statistics • Population and housing censuses (every 10 years) • Household sample surveys: • 1% population survey (inter-censal) • 1‰ population change survey (every year) • Administrative records: • Population by permanent residents (Ministry of Public Security) • Registered temporary residents (Ministry of Public Security) • Marriages and divorces (Ministry of Civil Affairs) • Health statistics (Ministry of Health)

5. The vital statistics system in China • Data checking: • Resident population and migrants’ information from the household registration administered by the Ministry of Public Security; birth information collected by family planning departments, the health department as well as the community and village committees – used to check and compare information collected in the 2010 censuses.

6. Reflections/some thoughts on the revision of the P&R, rev 2. • The chapter should answer the following questions: • Why is quality assurance and assessment important? • Against what standards are we evaluating? • What is quality assurance and assessment? • How should it be done? • What methods? • How often? • Who should be doing it?

7. Measuring against the following standards • Principles and Recommendations for a Vital Statistics System, Rev. 2 (para. 547) • Completeness • Correctness/accuracy • Availability • Timeliness • Additional ones to consider? • Relevance • International/national comparability • data confidentiality

8. Quality assurance – what is it/how is it done? • Occurs at each operation stage within the civil registration and vital statistics systems, for example • Registration stage: making sure that all events are registered without duplication; information verified; all required information provided • Data collection • Data transfer: Monitoring the submitted statistics report: timeliness, completeness (from all registration areas), and monitoring any irregularities • Data compilation, editing stage • Data dissemination • Making sure that all the checks are in place and are integrated part of the two systems (Australia paper)

9. Quality assurance – how often? Who does it? • How often? • Constant • Who does it? • Civil registrars at all levels • Statisticians compiling, processing and disseminating vital statistics

10. 2.0 Local Registration Example: United States LIVE BIRTH A1 Hospital Completes Birth Record B2 C1 N System Automatically Performs NCHS Validations Submit Record to VR Jurisdiction Via Web Hold for X Days Record Received From Local Registrar? Match Records, Manually Stamp, Log, Key Local File Number Register Record Y Correct Errors and Resubmit for Validation Passes all NCHS edits? Y N VR Database Print and Send Paper Record to Local Registrar Local Registrar Manually Logs, Reviews and Files Record Mail or Deliver Record to VR Jurisdiction END A2 B1 Generate Routine Submissions to NCHS C2 Timeliness Metrics: • Number of days from date of birth to when paper record is received from local registrar (A1 to A2). • Number of days from when record is received from local registrar to when record is registered (B1 to B2). • Number of days from when record is registered to when it is submitted to NCHS (C1 to C2). END

11. Process flow: deaths and stillbirths – South Africa Data Capture (ID variables) Collection Sorting Pasting STORES Data Capture & QA Export to Editing Coding Pre-coding Derivation 0f UCD Data Analysis Publication & Dissemination Data Editing

12. Quality assessment studies – what are they? • Answers specific questions and with specific objectives • What is the coverage of birth registration? • Who are the un-registered? What are their characteristics? • Is there difference in the quality of cause of death data from two provinces? What causes the difference and how to improve? • Is my new cause of death coder competent? • Is the information collected on birth records for District A accurate? • Some studies carried out regularly, some ad-hoc basis • Best if carried out by an independent group

13. Quality assessment studies – how? • Direct methods • matching birth and death records: identify un-registered records; an assessment of both registers • Matching civil registration records with other administrative records (e.g. social security, school records): such check may be automated and ran constantly if a comprehensive population register is in place • Match birth/death records with records from census and surveys: identify un-registered recent births/deaths; also check accuracy of other information • Compare cause of death information with autopsy reports, hospital records. • Assess coder’s qualification: ask 2 coders to work on the same set of statistics report and compare • Select a sample of birth records and send a questionnaire to parents asking similar information as on the birth reporting form • Include a question in census asking whether recent births and deaths are registered

14. Quality assessment studies – how? (cont.) • Indirect methods: • Monitoring births trends over a number of years; • Monthly figure monitoring • Intervals between date of occurrence and registration – timeliness • Work with census data: • Population balance equation: P2 – P1 = B – D + net in-Migration • Comparing births/deaths in the last 12 months • Comparing rates for similar periods

15. Quality assessment studies – how? (cont.) • Overall evaluation studies: touching on all aspects of the two systems but not in-depth • WHO/UQ HIS-Hub framework • Eurostat • UNSD assessment studies for countries through regional workshops and questionnaires • Often carried out on ad-hoc basis • Could be self-assessment

16. Points for discussion • What kind of information we would like this chapter to cover? • Are the evaluation methods (direct/indirect) up-to-date? • Any new methods emerged in the past 10 years?