Modernizing Official Statistics: Meeting the Challenges of Data Availability

1. Qatar National Statistics Day Doha, Qatar, 10 December 2013 Modernization of Official Statistics (Session 2) Eszter Horvath United Nations Statistics Division

2. “Never has so much been expected from statistics; never have statisticians had such means at their disposal; and never has there been so much willingness to learn from each other and standardize internationally the outcomes of that learning.” Handbook of Statistical Organization, Third Edition, 2003The Operation and Organization of a Statistical Agency http://unstats.un.org/unsd/publication/SeriesF/SeriesF_88E.pdf

3. Demand Increased demands and higher expectations for a wider range of statistics to be made available more quickly. Challenge Failing to address the challenges and opportunities of a world where data are available in abundance from many sources, sometimes on an almost “real time” basis, will reduce the relevance of producers of official statistics. Statistical organizations need to be sufficiently flexible and agile to provide quality statistics quickly, to meet user needs at an acceptable cost. Statistical organizations have to do more with fewer resources. Why Modernizing Official Statistics?

4. The High-Level Group (HLG) for the Modernization of Statistical Production and Services was created in 2010, to coordinate the response of the official statistics community. This group consists of ten heads of national and international statistical organisations, who oversee a modernisation programme to re-invent products and processes and adapt to a changing world. The HLG has produced a Strategic Vision and a Strategy to Implement the Vision, both of which have been endorsed by the Conference of European Statisticians. The over-arching theme in these documents is to eliminate the unnecessary diversity in statistical processes and to manage the necessary diversity more strategically. High-Level Group (HLG) for the Modernization of Statistical Production and Services

5. Measurement Framework Defining the domain International standards Metadata Data Collection Instruments and Processes How to collect What is the process Data Editing, Analysis and Archiving Output Production Types of output Media mode Interpretation and explanation Overview of the Statistical Production Process

6. Generic Framework on Statistical Production Process • The global statistical community needs a generic framework to review the statistical production process, with international agreed modules on each of the production process and with technical specifications • Two models have been proposed.They aim to provide common terminology, improving communication about the production of statistics, within and between organizations. This, in turn, facilitate collaboration and exchange of good practices, leading to greater efficiency.

7. Generic Statistical Business Production Model (GSBPM)

8. Generic Statistical Business Production Model (GSBPM) The GSBPM: • provides a framework of standard terminology to describe and define the set of business processes needed to produce official statistics; • it is intended to apply to all activities undertaken by producers of statistics, at both national and international levels, which result in data outputs; • it is designed to be independent of the data source, so it can be used for the description and quality assessment of processes based on surveys, censuses, administrative records, and other non-statistical or mixed sources.

9. Generic Statistical Information Model (GSIM) In addition to the processes described by the GSBPM, the information that flows between those processes (data, metadata, rules, parameters, etc.) is also very important. The Generic Statistical Information Model (GSIM): • aims to define and describe these information objects in a harmonized way; • provides a common language to describe information that supports the whole statistical production process from the identification of user needs to the dissemination of statistical products.

10. Generic Statistical Information Model (GSIM)

11. Integration of Official Statistics with Geospatial Information The HLG has also approved initial work to explore the relationships between these standards and models with the emerging range of geo-spatial standards. The geographical dimension of data is becoming increasingly important for data integration, analysis and dissemination.

12. New Data Sources: Unstructured Data and Big Data • Does not reside in traditional databases and data warehouses • May have an internal structure, but does not fit a relationaldata model • Generated by both humans and machines • Examples include • Personal messaging – email, instant messages, tweets, chat • Business documents – business reports, presentations, survey responses • Web content – web pages, blogs, wikis, audio files, photos, videos • Sensor output – satellite imagery, geolocation data, scanner transactions

13. The value of unstructured data sources • Provide a rich source of information about people, households and economies • May enable the more accurate and timely measurement of a range of demographic, social, economic and environmental phenomena • Combined with traditional data sources • As a replacement for traditional data sources • So present unprecedented opportunities for official statistics to • Improve delivery of current statistical outputs • Create new information products not possible with traditional data sources • Need to be checked against accuracy, relevance, consistency, interpretability, timeliness, and cost

14. Modernization of products Detailed and integrated datasets; geocoded data Available more rapidly Combination of various data sources Data solutions created by the users Modernization of production processes Use of new devices for data collection Use of SDMX to facilitate the integration of IT systems Use of common generic business processes across all statistical domains Modernization of organizational and human resources dimensions Organization to adapt to the new data environment Staff to be trained and equipped with relevant new skills How to Modernize?

15. Addressing increasing demand: Restructuring of the NSOs? • All reorganizations should accomplish the following: • Create new ideas • Lead to efficiency gains • Improve the organization’s focus on strategic objectives • However, an organizational restructuring is not necessary to achieve these goals in many circumstances

16. Institutional Set up Today • Centralized systems • As administrative data is increasingly used for statistical purposes, more work is undertaken outside of NSOs • Decentralized systems • Even in these systems, there is always a central statistical agency • Tendency to improve coordination to improve reliability and cohesion of statistical system • Centralized/Decentralized • Recent trend is for these two approaches to be coming together • Not the issue anymore but we need to: • Coordinate • Modernize • Communicate

17. As demand for data increases, consistency across data sources becomes more important (quality assurance framework) Improved efficiency can result from sharing infrastructure across statistical agencies within a National Statistical System The larger picture: from NSO to NSS: Improving Coordination

18. Leadership, vision, strategy Adequate Statistical Legislation Fundamental Principles of Official Statistics endorsed by the ECOSOC and General Assembly provide a strong foundation of NSOs Improved structure and coordination – vertical and horizontal Rethought business processes along which data are produced and disseminated Modernized resource management – human, financial and IT resources upgrade computers – upgrade human skills NSOs Driving the Modernization

19. Key points • National Statistical Offices should modernize to survive • Modernization is not only IT related • Modernization is strategic – to define the future of Official Statistics International Seminar on Modernizing Official Statistics:Meeting Productivity and Data ChallengesTianjin, China, 24-26 October 2013

20. Thank You!