Micro-data access: Data Reconciliation for Robust Estimations

1. Micro-data access: Data Reconciliation forRobust Estimations Expert Group Meeting on MDG Indicators Astana, 5-8 October 2009

2. Presentation overview: IHSN tools ADP task 1 leading to Task 2 Partnership ADP/WHO Preparations for a new workshop Implications of greater data availability

3. About the IHSN … A mechanism is needed to bring survey sponsors and survey users together. The recommendations propose creation of a Household Survey (HHS) Network, comprising the major sponsors of the global household survey programs... MAPS • A network of international agencies($0.75-1m pa DGF) A coordinating mechanism to: • Improve quality and use of household survey data in developing countries /MDG monitoring • Harmonise international recommendations for survey design, data analysis, etc • Produce and disseminate international good practices

4. In developing countries, survey data are produced mostly by official agencies for monitoring / assessing / targeting purposes High dependency on external financial and technical assistance Poor coordination creates problems: Accessibility and use (no re-purposing of data) Reliability and relevance Comparability across sources and over-time The problems

5. Better use of survey data Lots of data are collected, but various issues must be addressed to make them more useful/used: Limited information on what’s available (no catalogs of surveys) Limited accessibility by secondary users (for legal, technical, psychological and political reasons) Data often poorly documented, hence difficult or risky to use Reliability, comparability, relevance, timeliness, periodicity Loss of many datasets/metadata Work program on data documentation, dissemination and preservation Objective 1

6. Central Survey Catalogue 1. Search by country 2. List all surveys 3. Details on the surveys www.surveynetwork.org Cataloguing of over 3,500 surveys by country, forerunner to NADA

7. Archiving Tools The IHSN Microdata Management Toolkit using the power of the DDI standard for metadata exchange: Input-Metadata Editor:Input of data and metadata and includes bundling of reports, manuals and all Documents: Output to PDF and DDI Output-CD-ROM Builder:Input of data and metadata in DDI: Output to HTML for CD-ROM or web dissemination Catalogue-NADA (National Data Archive):Catalogue and link all the survey data and metadata and facilitate data access through on-line request forms

8. Better quality of future surveys Timing and sequencing (part of NSDS) Comparability (over-time, across countries), and consistency across data sources Reliability (use of improved methods) Work program on coordination and harmonization - Guidelines for dissemination in NSS - International Question Bank Objective 2

9. ADP An Accelerated Data Program (ADP): Country Implementation of IHSN Tools At the country’s request, the Accelerated Data Program (ADP) supports statistical data relevant for policy design, monitoring and evaluation, by making better use of existing data and aligning survey programs and statistical outputs to priority data needs. This goal will be achieved by: • Task 1: Building national capacity in micro-data preservation, analysis, anonymization, and dissemination;   • Task 2: Working with national data producers and secondary users on the production of updated estimates of key indicators, by further exploiting existing datasets and collecting new data.

10. ADP Countries Includes countries where ADP is implemented by at least one official data producer (NSO, line ministry). ADP active ADP Inactive Expressed interest in ADP IHSN Tools used without ADP

11. ADP Nigeria Implementation 2003 FOS starts Statistical Master Plan 2005 Merger of FOS and National Data Bank to NBS NB S 2006 NBS attends regional data archiving workshop in Ghana 2006 Request to enter ADP pilot program 2006 2007 2008 2009 Sept 2008 NADA Launch ADP June 2008 National Archiving Workshop October 2007 Inventory of Data July 2007ADP Documentation Workshop March 2007ADP Project document

12. ADP Nigeria Task 1 Task 1: Achievements in Nigeria • Over 20 surveys documented • Inventory reports over 100 data sets in various ministries • Review process before publishing • Draft data access policy • NADA with 14 reviewed surveys • Microdata available for many surveys on the NADA See: http://www.nigerianstat.gov.ng/nada/

13. ADP Nigeria Data Demand Task 1 Primary Outcome: Measuring Data Demand in Nigeria Total: 89 downloads  Nigeria on-line survey databank - Number of downloads of data (two surveys) in 3 months, by origin of request  Data users in Nigeria

14. ADP Nigeria Task 2 Task 2: Going further in Nigeria The ADP Task 2provides support for the assessment of existing datain the country by specialized national and international researchers. The expected outputs are not only analytical work and policy briefs, but also and mostly a detailed assessment of the strengths and weaknesses of the data (in terms of reliability, relevance, comparability, timeliness, accessibility), and recommendations for improving data collection and analysis in the country. With Task 1 completed, the ADP was looking for a sector and a partner to validate the work and extend the process further…and this…. …is what the presentation is about…a partnership & the results of data access

15. JMP/ADP Start of a partnership • The JMP was in search of New Data: • JMP sought help from the IHSN • Through the World Bank Development Data Group the JMP was • Provided with WAT SAN datasets for 100 countries • Nigeria: • The JMP had done a traditional workshop in 2007 • Was moving toward a workshop focusing on reconciliation and harmonization • Through the IHSN and the ADP the JMP found a partner The JMP/ADP made a perfect match!

16. What is measured and what not? Access to improved sanitation technologies Well regulated systems Total population Toilets flushing to sewerage systems Sustainability Toilets flushing to sewerage with effective wastewater treatment Access to improved sanitation technologies ensuring privacy (not shared) Affordable

17. Major reasons for discrepancies • Differing sources of data • NSO – user based data • Sector – provider based data • Differing methodologies • Even with user based data • Countries sometimes use single data point • Differing population estimates • Most recent census vs. UNPD estimates • Differing definitions of urban/rural • NSO and Sectors not always agree • Differing definitions

18. Comparing JMP and national definitions (after closing data gaps) MDGDefinition • With access • Sewer connection • Septic tank • VIP/improved pit • Without access • Shared facilities • Unimproved facilities • Open defecation NationalDefinition • With Access • Sewer connection • Septic tank • VIP/improved pit • Shared facilities • Unimproved facilities 52% 80% Differences due to differing definitions • Without Access • Open defecation

19. Can JMP and country authorities agree on water and sanitation ladder?

20. ADP/JMP Nigeria Task 2 Preparations • Once the partnership was defined after meeting with JMP and Nigeria • Identify the national agencies monitoring the sector and build consensus • UNICEF, Active Water Sanitation Sector, Water Aid, JMP • (Building on the work of JMP reconciliation in 2007) • Review any previous harmonization work done in the sector • v UNICEF Harmonization work and indicator publication • Identify and prepare the data sets • Review the quality of the data • Review published survey reports and questionnaires • Organization of the workshop • Identify technical resources

21. ADP/JMP Nigeria More Data ADP identified six more surveys with national Water and Sanitation data and added these to the 3 DHS surveys. • New JMP Estimates added the • data available in the NADA • Child Labour Force Survey 2000 • Nigeria Living Standard Survey 2004 • General Household Survey 2006 • Core Welfare Indicator Survey 2006 • Multiple Indicator Cluster Survey 2007 • Sector Baseline Survey 2008 • Old JMP Estimates • based on: • DHS 1990 • DHS 1999 • DHS 2003

22. ADP/JMP Nigeria Compare Sample size

23. ADP/JMP Nigeria Compare Past results

24. ADP/JMP Nigeria Compare Questionnaires

25. CLFS 2000 1 Tap/pipe Inside house 2 Tap/Pipe Outside House 3 Tube/well 4 Manual Well Protected 5. Handpump 6.Ponds/Stream/River/Rainwater CWIQ 2006 1 Pipe borne water treated 2 Pipe borne water untreated 3 Bore hole/hand pump 4 Protected well 5 Unprotected well 6 Rain water 7 River, lake or pond 8 Vendor, truck 9 Other Compare Categories ? MICS 2007 11 Piped into dwelling 12 Piped into yard or plot 13 Public tap/standpipe 21 Tubewell/borehole 31 Dug well/Protected well 32 Unprotected well 41 Protected spring 42 Unprotected spring 51 Rainwater collection 61 Tanker-truck 71 Cart with small tank/drum 81 Surface water (river, stream, dam, lake, pond, canal, irrigation channel) 91 Bottled water 96 Other (specify) GHS 2006 1 Pipe borne water treated 2 Pipe borne water untreated 3 Borehole/hand pump 4 Well/Spring Protected 5 Well/Spring Unprotected 6 Rain Water 7 Streams/Pond/River 8 Tanker/Truck/Vendor 9 Other NLSS 2004 1 Pipe borne water treated 2 Pipe borne water untreated 3 Borehole/hand pump 4 Protected Well 5 Unprotected

26. ADP/JMP Nigeria Task 2 Workshop

27. Workshop Outputs • Harmonized response categories between surveys (i.e. recoded and re-compared) • Confidence intervals on all recoded data • New trend in access to improved water and sanitations • Common definitions (JMP based) • Core Questions Defined

28. CWIQ 2006 1 Pipe borne water treated 2 Pipe borne water untreated 3 Bore hole/hand pump 4 Protected well 5 Unprotected well 6 Rain water 7 River, lake or pond 8 Vendor, truck 9 Other Recoded as Improved or Unimproved CLFS 2000 1 Tap/pipe Inside house 2 Tap/Pipe Outside House 3 Tube/well 4 Manual Well Protected 5. Handpump 6.Ponds/Stream/River/Rainwater MICS 2007 11 Piped into dwelling 12 Piped into yard or plot 13 Public tap/standpipe 21 Tubewell/borehole 31 Dug well/Protected well 32 Unprotected well 41 Protected spring 42 Unprotected spring 51 Rainwater collection 61 Tanker-truck 71 Cart with small tank/drum 81 Surface water (river, stream, dam, lake, pond, canal, irrigation channel) 91 Bottled water 96 Other (specify) IMPROVED 50/50 SPLIT GHS 2006 1 Pipe borne water treated 2 Pipe borne water untreated 3 Borehole/hand pump 4 Well/Spring Protected 5 Well/Spring Unprotected 6 Rain Water 7 Streams/Pond/River 8 Tanker/Truck/Vendor 9 Other NLSS 2004 1 Pipe borne water treated 2 Pipe borne water untreated 3 Borehole/hand pump 4 Protected Well 5 Unprotected

29. Checked intervals • Significant differences to be explained are indicated in light blue. • NLSS may be a problem with weighting coefficients. • DHS2003 may be a sampling issue or a recoding issue due to a high level of ambiguous categories for reclassification. • MISC2007 takes into account shared toilet facility as unimproved. This cannot be properly measured in other surveys.

30. Trend line change

31. Nigeria Estimates Revised

32. Sub-Saharan estimates Revised

33. Core questions And categories IMPROVED UNIMPROVED

34. Open Questions • The fundamental lesson can be boiled down to two questions • For the National Statistical Office: • Is there a • National Core Survey Questionnaire available on-line? • Is there a • National Survey Catalogue (Archive) available on-line? If the answer to the any of the above is no, then please ask why?

35. Open Questions • Is there interest at the country level for this activity? • What countries and what sectors ? • Is National Data Archive of interest? • Is there an interest in developing a core National questionnaire? • Developing training modules/training material • Funding Mechanisms for implementation at the country and supporting development of training material/technical support.