Maximizing Statistical Metadata Strategy in Portugal

1. Isabel Morgado (isabel.morgado@ine.pt) Monica Isfan (monica.isfan@ine.pt) « Statistics Portugal’s Case Study 9 –11 April 2008

2. Overview • Organization • Metadata Strategy • Statistical metadata system • Statistical metadata in each phase of the statistical business process • Systems and design issues • Organizational and workplace culture issues • Lessons learned

3. Organization The National Statistical System • Statistical Council • National Statistical Institute (Statistics Portugal)

4. The Statistical Council is the state body that supervises and coordinates the National Statistical System. Organization This duty is carried out by the “Planning, Coordination and Dissemination” Standing Section, which has the power: “…to analyse and approve concepts, definitions, nomenclatures and other technical instruments of statistical coordination of the National Statistical System and to approve regular changes to these documents resulting from work done at the EU or national level”(Structure and Functioning of Statistical Council – SC Deliberation No. 286; 2005).

5. Organization Statistics Portugal: • Record, • Refine, • Coordinate, • Disseminate, official data • Delegate duties to other public departments, • Conceive and manage the statistical metadata system of the National Statistical System.

6. Organization Statistical Council Secretariat Board Delegations (Porto, Coimbra, Évora, Faro) Planning, Control and Quality Unit Dissemination Unit Legal Support Unit Communication and Image Unit Administration and Management Department National Accounts Department Methodology and Information System Department Economic Statistics Department Demographic and Social Statistics Department Data Collection Department

7. Organization Methodology and Information System Department ....... Metadata Unit Applications Development Unit Coordinates all the work related to the statistical metadata system.

8. Metadata Strategy Approval of concepts Statistics Portugal centralises the management of statistical concepts. Concepts classified by subject area and, entered into database with the status “proposed concept”. The working groups (Statistical Council) analyse and recommend the concepts for approval to the Planning, Coordination and Dissemination Standing Section. After approval, the concepts receive the status: “Statistical Council – approved concept” and, are of obligatory use.

9. Metadata Strategy Approval of classifications: • Portuguese Classification of Economic Activities, • National Classification of Occupations, • National Classification of Goods and Services, • Administrative Division Code, • List of Countries, • Etc. Approved by the Statistical Council formandatory usein theNational Statistical System.

10. Metadata Strategy Approval of methodological documentation: 2005 Statistics Portugal submitted to appreciation to the Planning, Coordination and Dissemination Standing Section a standard format for the methodological documentfor theNational Statistical System’s surveys. The standard format was approved and adopted as mandatory in the National Statistical System. 2007 75% of the surveys in the National Statistical System were documented in this format.

11. I.Design II. Production III. Dissemination IV. Evaluation Processes Processes Processes Processes I.1 Feasibility Study II.1 Planning and Preparation III.1 Dissemination IV.1 Evaluation I.2 Methodological Study II.2 Collection I.3 Technical Approval II.3 Process and Analyse Metadata Strategy Phases

12. Metadata Strategy Methodological Document (preliminary) Intranet I.3 Technical Approval Methodological Documents Methodological Document (final) Internet

13. Metadata Strategy Technical approval of surveys: • Thepreliminary methodological documentand thequestionnaire(s)are sent: • units directly involved, • users of the results of surveys, • Planning Unit, • Data Collection Department, • Methodology and Information Systems Department. opinion • The Metadata Unit analyses: • concepts and classifications, • ensures correct application of the standard format of the methodological document, • analyses questionnaires, • introduces new concepts into the concept base. opinion

14. Metadata Strategy Technical approval of surveys: • The department responsible for the survey: • updates the methodological document and/or the questionnaire(s) with the proposed changes or justifies its rejection, • submit the new version of the methodological document and questionnaire(s) for approval by the Board. • The Metadata Unit prepares a memo, on the basis of all the opinions of the different units and respective answers, to send to the Board, proposing their approval or rejection. • The Board then approves or rejects the survey. • Metadata Unit records the questionnaire(s) in the data collection instruments database and publish the methodological documentation the Intranet and in the Official Statistics website.

15. Integrated Statistical Metadata System

16. To support the wholelife cycleof surveys; To act as acentral repositoryfor statistical metadata serving as a source for other databases that support: design, production, dissemination of statistics and management; To establishterminologyfor statistical metadata; To constitute aninstrument for statistical harmonisation and coordinationof the National Statistical System, standardising the documentation of surveys, among other elements; To implement ahomogeneous environmentfor its technological infrastructure. Integrated Statistical Metadata System The main purposes of the Integrated Statistical Metadata System are:

17. Corr. Entry Corr. Table Classifications Item Level Family Classif. Version Dom. Unit of Value Measure Variables Concepts Theme Variable Variable Conceptual Theme Source Unit Stat. Glossary Concepts Admin. Sources ires Questions Questionna Instr. Collection Relationship Data Files Doc. Survey Method. Statistical sources Products Integrated Statistical Metadata System

18. Concepts Subsystem Glossary Theme Relationship Concept Source

19. Concepts Subsystem

20. Concepts Subsystem

21. Classifications Subsystem Family Classification Correspondence table Version Level Item

22. Classifications Subsystem

23. Classifications Subsystem

24. Variables Subsystem Family/ Theme Property Object Class Conceptual Variable Representation Class Value Domain Variable Unit of Measure Statistical Indicator

25. Variables Subsystem

26. Variables Subsystem

27. Variables Subsystem

28. Data Collection Instruments Subsystem Data Collection Instruments Substitution/ Prorogation Questionnaires Files

29. Data Collection Instruments Subsystem

30. Methodological Document Subsystem Survey Observation Concepts used Variables Derived Indicators Dimensions Classifications used Methodological Document Products Questionnaires Collection Instruments Data Files

31. Methodological Document Subsystem Standard format: Methodological document • General characterization, • Methodological characterization, • Concepts, • Classification, • Variables, • Data collection instruments, • Abbreviation and acronym, • Bibliography.

32. Methodological Document Subsystem

33. Methodological Document Subsystem

34. Methodological Document Subsystem Metadata classification: • Statistical metadata • Survey metadata, • Methodological metadata, • Definitional metadata, • Quality metadata • System metadata

35. Statistical metadata in each phase of the statistical business process

36. Statistical metadata in each phase of the statistical business process Phase: Production/II.1 Planning and preparation Universes and Samples Management System Sample frame and samples Short – term surveys Sub-universe frame WEbInq Annual universe frame Sample frame and samples Sub-universe frame Structural surveys Statistical Unit Metadata SUFS Statistical Units File System ISMS Integrated Statistical Metadata System

37. Statistical metadata in each phase of the statistical business process Phase: Production/II.2 Collection Collection Process Management System Universes and Samples Management System USMS System metadata Integrated Statistical Metadata System ISMS Surveys Metadata Data collection instruments Survey instance Collection instance Collection Statistical Unit self-completion SUFS Statistical Units File System WEbInq

38. Statistical metadata in each phase of the statistical business process Phase: Production/II.2 Collection

39. Statistical metadata in each phase of the statistical business process Phase: Dissemination/III.1 Dissemination Data Base Data Metadata Statistical Indicators(view) Metadata Variables Subsystem Statistical Indicators DB Data Internet DataWarehouse

40. Statistical metadata in each phase of the statistical business process Phase: Dissemination/III.1 Dissemination

41. Statistical metadata in each phase of the statistical business process Phase: Dissemination/III.1 Dissemination

42. Systems and design issues

43. Organizational and workplace culture issues Metadata Unit • Technicians with general profile, • Nomenclaturists, • Terminologists. 17 technicians + head unit

44. Organizational and workplace culture issues Training and knowledge management Training plan • Integrated Statistical Metadata System, • Terminology, • Classifications subsystems, • Variables subsystems, • Methodological documentation (2009).

45. Organizational and workplace culture issues Training and knowledge management I. Integrated Statistical Metadata System • The integrated metadata system’s different components and its role as a technical statistical coordination tool; • The basic concepts and terminology underlying this system; • The role of each subsystem in the life cycle of surveys; • Improving the statisticians’ competences in the use of this system as a tool in designing the information subsystems in which they work.

46. Organizational and workplace culture issues Training and knowledge management II. Terminology • Terminology theory; • The notion of conceptual system and the application of a methodology in its design; • Semantic relationships in terminology; • Terminography; • Definition.

47. Organizational and workplace culture issues Training and knowledge management III. Classifications subsystem • The main international, EU and national classifications systems used in statistics, the classifications comprising them and the relationships between them; • The approval process for classifications at different levels and harmonisation in the use of classifications in the NSS; • Improving competences in the use of the classification subsystem.

48. Organizational and workplace culture issues Training and knowledge management IV. Variables subsystem • Recognition and designation of entities making up the conceptual model; • The rules and principles in the name, standardisation and harmonisation of variables; • How this subsystem relates to other subsystems in the integrated metadata system and other outside subsystems, such as the Dissemination Database and WebInq (electronic data collection); • Use of the consultation applications and the survey manager (SM) profile of the management application; • Defining indicators to be disseminated on the Official Statistics website.

49. Organizational and workplace culture issues Training and knowledge management V. Methodological Documentation (2009) • The Statistical Council-approvedmethodological document format for documenting the National Statistical System’s surveys; • Use of the consultation applications and the survey manager profile of the management application.

50. Lessons learned • Involvement of the institution's top management, fundamental Creation of documentation with formal and standardised procedures. • Development of prototype systems before final implementation, fundamental To test the system’s design and to detect the strong and the weak points.