1 / 11

Data validation in Statistical Office of the Republic of Serbia

Explore how the Statistical Office of the Republic of Serbia utilizes IST, a unique metadata-driven system, for efficient data validation, processing, and reporting. Learn about the comprehensive validation rules, logical control procedures, and integration with Eurostat for high-quality statistical output.

benjaming
Download Presentation

Data validation in Statistical Office of the Republic of Serbia

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data validation in Statistical Office of the Republic of Serbia

  2. Statistical Office of the Republic of Serbia • As the backbone of the national statistical agency, SORS is the main producer of official statistics in the service of citizens and policy-makers. SORS has over 400 employees and 14 regional offices throughout Serbia; central office is located on three different locations in Belgrade. • We annually conduct over 300 studies and various one-off projects. Many of the surveys are based on a monthly dynamic (that may span many years and decades), so at any moment the SORS has over 250 active applications (IT solutions). • More than 90% of statistical production in SORS is done in IST • IST is a software tool, metadata driven system, completely designed and developed in the Statistical Office of Serbia

  3. Metadata driven codeless development environment • IST concept: • simple metadata database - IST metadata • program that interprets metadata - IST program

  4. Metadata driven codeless development environment IST has been fully developed on Microsoft development tools, on the .net technology and the IST metadata is on the MS SQL server IST is a .net application and a simple metadata database (only six tables) These two equally important components deal with a large number of different databases containing individual and aggregate data IST.net application is an interpreter. It, literally, reads data from the IST metadata that represents a description of each application, and based on the read data it generates and in real time executes each stage of statistical research or any other project, from data entry, data editing and validation to the dissemination of the data. The most important components of the IST.net application are: updating of the IST metadata, generated input data ('data entry interpreter'), generator for 'web' applications, data editing and data correction, automatic correction, advanced search of individual and aggregated data from the database, generating reports and the ability to manage procedures in database containing individual and aggregate data.

  5. Rules of logical control in metadata database

  6. Rules of logical control in metadata database • All data validation rules for all statistical surveys conducted in our office are stored in ONE table of IST metadata database • Simple rules for validation: built-in functions and the SQL statements • Every physical table has aditional columns (boolean data type) for keeping information about errors in records for each validation rule described in IST metadata database and also one for whole record as a flag that record is not valid • All Validation rules are described with SQL statements (just„WHERE“ clause of the UPDATE statement) • After running batch logical control IST generates validation report • Data validation is followed by corrective actions, such as data editing or data imputation.We are accessing records with errors trough same application

  7. Logical control

  8. Logical control - report

  9. Data editing

  10. Sending data to Eurostat • First part of preparation of data for EUROSTAT and validation is done on IST platform • All reporting procedures for all surveys are stored in one table in IST metadata database. Interpreter get data from database and generate report in .xls, .xml or .json format • Although all statistical production has been carried out in the SORS on IST platform, there are no standard procedures for the way that data that is being sent to Eurostat at the institutional level. Every department and responsible statistician validate and send data using different tools and platforms provided by EUROSTAT. If data sent to Eurostat is not valid, in cooperation with subject meter experts, developers are adjusting validation rules in IST and after corrections they prepare new reports for Eurostat again • Future plans: Cooperation with Eurostat regarding services for data and metadata validation and to explore is there a way to integrate developed services in oursystem, in order to improve statistical system and improve validation, data and metadata exchange

  11. Thank you for your attention!

More Related