Statistics Portugal | Planning, Control and Quality Unit Magda Ribeiro | magda.ribeiro@ine.pt

1. Statistics Portugal | Planning, Control and Quality Unit Magda Ribeiro | magda.ribeiro@ine.pt Vera Morais | vera.morais@ine.pt Measuring Relevance of Statistical Information « European Conference on Quality in Official Statistics 2008 July 2008 «

2. The official statistical information is an essential tool for the knowledge of any society The statistical information should allow a “real” quality picture on the economic and social situation The permanent society development is a major demand for the statistical activity Keeping the statistics relevant implies that they must closely follow the continuous societal changes, identifying at each moment what is important to measure Statistics should be evaluated according to international standards, assuring its quality and comparability among other national or international data Moreover, statistics are to be released to all users at the same time Statistical information

3. The decision to keep or start a statistical project is directly associated with the relevance of that project. Those decisions are taken under the national and international statistical systems. Relevance means Relevance Pertinence, Utility, Importance … Strategical Domain Multi-annual statistical work programmes Annual statistical work programmes Operacional Domain The relevance of the disseminated statistical information is defined by the users recognising the utility and importance of that information.

4. Relevance - Standard Quality Report Relevance is one of 6 dimensions of the definition on Quality in Statistics and the Standard Quality Report Relevance accuracy timeliness and punctuality accessibility and clarity comparability and coherence “Relevance is the degree to which statistics meet current and potential users’ needs. It refers to whether all statistics that are needed are produced and the extent to which concepts used (definitions, classifications etc.) reflect user needs” [Eurostat 2003a].

5. Relevance - Standard Quality Report Accuracy, timeliness and punctuality, accessibility and clarity, comparability and coherence (depends) Degree of relevance Example: If a statistical data is not released on time, it could loose its utility and consequently become not relevant. Suppose that: The announcement of the unemployment rate for the 1st quarter 2008 is done in 2010. Is this information useful for any decision on the labour market? «

6. A description and classification of users A description of the variety of users’ needs (by class of users, mainly), the possible contradictory expectations among them and priorities made. If a class of users is given strategic importance, a more thorough description of their needs can prove useful Reference of specific documents where the description of more comprehensive needs could be found, if any Main results regarding the satisfaction of users. In particular, appraisals by the most important class of users, if any. Main reasons for lack of relevance The number or percentage of unavailable results, compared to what should be available. Reasons for incompleteness as well as the prospects for future solutions Follow-up of the user satisfaction assessment, i.e. the measures and actions taken to improve user satisfaction Circulation and /or readership of publications (paper of electronic) Number of Web hits for the relevant web-pages and/or number of downloads of specific products Relevance - Standard Quality Report Relevance - Guidelines Source: Doc. Eurostat/A4/Quality/03/General/Standard_Report

7. Relevance - Standard Quality Indicators • Rate of available statistics • It is calculated by dividing the number of values provided in a concrete data set divided by the total number of fields for which data has to be provided: • Rate of available statistics = Number of values provided / Number of fields applicable • User Satisfaction Index A recommended methodology does not exist. Normally it is evaluated through user satisfaction surveys Source: Doc. ESTAT/02/Quality/2005/9/Quality Indicators «

8. Relevance – Code of practice European Statistics Code of Practice PRINCIPLE 11: RELEVANCE European statistics must meet the needs of users Indicators: • Processes are in place to consult users, monitor the relevance and practical utility of existing statistics in meeting their needs, and advise on their emerging needs and priorities. • Priority needs are being met and reflected in the work programme. • User satisfaction surveys are undertaken periodically. «

9. Reflections on relevance Aspects related to the relevance of the statistical information : 1. Conception of the statistical project: What to do and how to do it? 2. Users of the statistical informatios: For whom? 3. Results of the statistical project:How to disseminate them and how will they be understood by the users? Life cycle – Statistical project «

10. Life cycle – Statistical project Statistical Production Handbook Procedures II. Operation III. Dissemination IV. Evaluation I. Conception II.1. Planing and preparation of data collection I.1. Viability Study III.1. Dissemination IV.1. Evaluation I.2. Methodological Study II.2. Data collection • The phases directly related with the relevance are: • Conception • Dissemination • Evaluation II.3. Data treatment and analysis I.3. Technical approval «

11. II. Operation III. Dissemination IV. Evaluation I. Conception Life cycle – Conception • Conception: • 1st phase of a statistical operation, when the methodological study is developed I.1. Feasibility study • Sub-processes related to relevance I.2. Methodological study I.3. Survey approval «

12. Life cycle - Conception Feasibility study – Document /Template Main tasks related with relevance Documentation: • Objectives of the survey • Identify user's needs • General characterization of the survey - Communitarian Legal Acts - Main documents about users needs - Document with the generic characterization of the survey

13. Life cycle - Conception Methodological study – Document /Methodological Document Main points related with relevance • Purpose • Summarize the objectives of a statistical operation • Users information • Users description by type and strategical importance • Description of information needs by users type Products Quality standards – desirable release calendar Products to make available – associated information for each product: Name; Product type; Periodicity; Geographic level of disaggregation; Availability type; Strategic users Information available for each variable Name, Measure unit, Classification variables

14. II. Operation III. Dissemination IV. Evaluation I. Conception Life cycle - Dissemination Dissemination:selection, adaptation, promotion and publishing of the information resulting from statistical operations. Its major purpose is to put available to the community statistical data with quality, to support decisions and further research. Main tasks related with relevance: • Adapt contents and products to users needs • Adapt the content’s promotion strategy • Adapt communication strategy to the users • Prepare the dissemination • Execute the dissemination • Promote «

15. II. Operation III. Dissemination IV. Evaluation I. Conception Life cycle - Evaluation Evaluation:life cycle evaluation for a statistical operation; strong points and improvement opportunities to implement or adapt to other projects. Main tasks related with relevance: • Survey quality evaluation a posteriori • Products and dissemination services quality evaluation • Audits Quality Report, by statistical project Example: User satisfaction surveys Example: Peer review «

16. Evaluation - Peer Review – Principle 11 Iceland Statistics Iceland benefits from a very close cooperation with users. Intensive cooperation with users and their frequent consultation has been put on an institutional level. Two user groups, on price statistics and national accounts, and one advisory group on wages have been set up. The user and advisory groups unite both users and cooperative partners in the production of statistics. The meetings are organised in an open manner, with participants raising subject to be discussed. Minutes are recorded and published. This creates a very open style of managing statistics. Liechtenstein Users' recommendations are integrated in publications whenever possible. Lithuania User satisfaction surveys. The user satisfaction surveys cover general users, web users and specific user groups. The surveys – some outsourced to private opinion institutes - ask for opinions on the following subjects: visibility and image perception, quality of official statistics, internet accessibility, statistical publications, monitoring of user-requests, alert-me services, library-bookshop in Statistics Lithuania head-office and visitors’ corners in Regional statistical offices. Source: Summary of good practises identified during the European Statistics Code of Practice peer reviews carried out during 2006-2008 - Version 1.0 of 10 June 2008

17. Evaluation – Peer Review – Principle 11 Peer Review Reports associated to each Member State and Eurostat - principle 11: RELEVANCE Most frequent initiatives: User satisfaction surveys Contact/communications with users «

18. Evaluation – Customer satisfaction surveys Examples: Researchers • The results analysis of a user satisfaction survey to researchers has showed that the satisfaction levels were different among them, according to the statistical area considered. • This analysis useful the definition of improvements for different statistical projects and users. • Media As a result of the analysis of a user satisfaction survey to the media, Statistics Portugal has promoted several workshops. The purpose of this workshops was to inform the media about the statistical activity along the life cycle of each project (information availability, release calendars, methodologies, etc.) «

19. How can we measure the relevance of statistical information Measuring Relevance By project • Evaluation of the relevance of statistical data on decision making • Analysis of users (strategic or non strategic) • Analysis of user data requests by statistical project and user type • Website statistics analysis by statistical project and service type • User satisfaction surveys by statistical project and user type • Promotion of focus groups during product definition and set-up Know users and users needs: Users type – users classification «

20. Conception: Expectations on relevance level are high 1 Low high Decrease the distance between 1 and 2 After data dissemination Relevance level depends on user perception about the information It can be increased with promotion activities Continuous information on users Adapt contents to users requests Improvement actions 2 3 Relevance level

21. Relevance level The relevance level of a statistical project can be increased Even assuming that the relevance level of a statistical project is high, that might not be perceivable or understood by users. Relevance is a dynamic attribute! Communication • Efficient communication between NSI’s and their users • Meet users’ needs according to a users typology • Understand and act on users perception - users interaction • Oriented promotion of contents and products, according to users typology Explaining statistical literacy Promotion «

22. Promotion Promotion efficiency • Examples: • 1. Statistics Portugal has conducted a promotional campaign for the Demographic Studies Review, through a mailing to potential users. The results were: • 76% of sales were a direct result of this mailing • Among other products with similar promotion, this Studies Review had the highest number of downloads (3449) • 2. A mailing about the publication “Tourism Statistics” was responsible for an additional number of new users (33%). «

23. Communication/Perception Some user types are active actors on the promotion of statistical information, contributing to increase the relevance level of statistical information Example: Portuguese media When searching by the keywords: “Employment”, “Unemployment” and “Labour Market”, 862 results were obtained referring to news published on Portuguese media mentioning statistical data produced by Statistics Portugal, during 1st quarter 2008. The statistical information relevance might be compromised by the user perception Example: Media During 2005 news were published in the Portuguese media suggesting that the data disseminated in “Study on the Local Purchasing Power” were manipulated due to political pressure. Statistics Portugal then invited journalist to a press conference in order to clarify the methodology of the referred study. «

24. Relevance - Users Considerations on the users • Official statistical producers can have a key role on increasing the recognised relevance of statistical products through their adequacy to different users’ needs. • Implies knowing how to classify users: groups of users with common needs. • Specialized users of statistics are normally request more detailed data whereas non specialized users are interested in general and more aggregated data. • Satisfied users tend to recognize greater relevance to statistics. • The users literacy level is directly related with their recognition of statistical information relevance. Relevance level Users oriented activity «

25. Conclusions Relevance • We can measure relevance in advance: In the conception phase of a survey • We can measure relevance afterwards (à posteriori): Using the results of Customer Satisfaction Surveys; Using customer’s Database information – Customer profile, Typology of information requested, Customers level of experience, Demand indicators The process of measuring relevance must be harmonized among countries and statistical projects. Demand statistical information Index Relevance User Satisfaction Index «

26. Conclusions Relevance level • Producers and disseminators of statistical information have the option to increase statistical relevance level by: - Studying the relevance: using quantitative, qualitative and descriptive indicators - Improving processes - Communicating with users By statistical project «

27. Future developments • Cross analysis of quality reports • A case study of a statistical project - What is the relevance level of the Labour Force Survey? «

28. Statistics Portugal | Planning, Control and Quality Unit Magda Ribeiro | magda.ribeiro@ine.pt Vera Morais | vera.morais@ine.pt Thank you July 2008 «