1 / 27

CORE Architecture: Designing Statistical Processes for Abstract Services and Data Exchanges

This article provides an overview of the CORE architecture, including design and implementation issues, the data model, IAPIs, repository, and GUIs and web applications. It discusses the objective of CORE and its design principles for abstract services and data exchanges. The article also explores the CORE data and metadata design, as well as the logical architecture, GUI integration APIs, and the CORE service repository. Additionally, it explains the CORE process engine and its functional and organizational perspectives.

carre
Download Presentation

CORE Architecture: Designing Statistical Processes for Abstract Services and Data Exchanges

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CORE Architecture Mauro Bruno, Monica Scannapieco, Carlo Vaccari, Giulia Vaste Antonino Virgillito, Diego Zardetto (Istat) 1

  2. Outline • Overview of CORE architecture • Design & implementation issues • Data Model • IAPIs • Repository • GUIs and Web application 2

  3. CORE Objective • Provide a unique environment for: • Designing • Statistical processes in terms of abstract services • Exchanged data and metadata • Running • Designed processes by invoking existing (wrapped) tools

  4. CORE Design: Services • Abstract services: specify a well-defined functionality in a technology-independent way • An abstract service can be implemented by one or more concrete services, i.e. IT tools • Examples: sample allocation, record linkage, estimates and errors computation, etc.

  5. CORE Design: Services • GSBPM classification • Documentation purpose • Provided that a CORE service can be linked to several IT tools, GSBPM tagging enables to perform searches retrieving, for instance • “all the IT tools implementing the 5.4 Impute subprocess of GSBPM proposal”

  6. CORE Design: Services • Service inputs and outputs • Specified by logical names • Characterized wrt their “role” in data exchanges • Non-CORE: if they are not provided by/to other services of the process, but are only “local” to a specific service • CORE: otherwise, i.e. they are passed by/to other services and hence they do need to undergo CORE transformations

  7. CORE Design: Data and Metadata • They are specified as service inputs and outputs • Logical names link them to previously specified services • Non-CORE data do not need any further specification but the file system path where they can be retrieved

  8. CORE Design: CORE Data • 3 elements concur to the specification of CORE data • Domain descriptor • CORE data model • Mapping model

  9. Domain descriptor model • Entity • Like Entity Relationships entities • Entity properties • Like Entity Relationships attributes • Very simple (meta-)model: can easily describe “in fieri” models like GSIM

  10. Example of Domain Descriptor <schema name="DEMO_Domain_Descriptor"> <entity name="SamplePlan"> <property name="STRATIFICATION_VAR"/> <property name="STRATUM_SAMPLE_SIZE"/> <property name="STRATUM_POPULATION_SIZE"/> </entity> <entity name="Enterprise"> <property name="IDENTIFIER"/> <property name="STRATIFICATION_VAR"/> <property name="WEIGHT"/> <property name="SAMPLING_FRACTION"/> <property name="ENTERPRISE_FLAG"/> <property name="EMPLOYEES_NUM"/> <property name="VALUE_ADDED"/> <property name="AREA"/> </entity> </schema>

  11. Domain Descriptor: role • Role of the domain descriptor: from service-to-service data mapping to service-to-global data mapping i1 S1 O1 mapped to i2 Via ad-hoc mapping o1 i2 o1 DD o1 DD O1 mapped to i2 Via DD i2 DD i2 S2 o2

  12. CORE Data Model Rectangular data set CORE tag: Data set level (mandatory) Column level (optional) Rows level (optional) Data set kind Column kind 12

  13. CORE Data Model: role • Specified once and valid for all processes • Extensible, i.e. core tag, data set kind, column kind can be modified • Adds more semantics to data • Example of usage: mapping to other models

  14. Mapping model Rectangular data assumption Mapping is intended to be specified wrt Domain Descriptor Columns are to be mapped to properties of an entity It contains the specification of how CORE data model concepts are associated to data 14

  15. CORE Logical Architecture GUI Integration APIs SERVICES CORE Repository … Runtime Process Engine 15

  16. CORE GUIs Process design Ad-hoc customization of an existing tool (Oryx) Service design Set of interfaces for the definition of services and related data flow Data design Set of interfaces for the specification of domain descriptors and mapping files

  17. Integration APIs • Purpose: making a tool a CORE service • Translates inputs and outputs of the tool in a completely transparent and automatic way CORE Service

  18. Repository • Processes and their instances • Services with their GSBPM and CORE classification • Tools and their runtime features • Data with their logical classification within CORE processes

  19. Process Engine Official statistics processes can be viewed from two perspectives: Functional: they are data-oriented,reflecting acommon feature of scientific workflows Organizational: they are workflow-oriented,have the complexity of real production lines, with the need of harmonizing the work of different actors

  20. Process Engine Hence our process engine has two layers WF ENGINE • Complex control flows • Syncronizing constructs, cycles, conditions, etc. • E.g.: Interactive multi-user editing imputation • Simple control flows • Sequence of tasks is composed by connecting the output of one task to the input of another • Data intensive operations DATA FLOW CONTROL SYSTEM 20

  21. Implementation issues Java web application implementing: GUIs CSV-CORE Integration API Data flow control system Layered design firmly based on frameworks: Hibernate: database mapping Struts2: model-view-controller approach Repository implementation: MySQL dbms

  22. Web Application Design Business Logic Data access View (GUI) Model Controller Forms Actions Services DAOs Entities Input validation Hibernate Struts2

  23. Architecture Deployment • Web-based architectured centered on a centralized component • CORE Environment • Different CORE deployments can co-exist • Intra- or Inter- organization • Services can be remotely executed • Support is needed in the form of a distributed component for tool execution and data transfer

  24. Types of service runtime • Batch • Tool executed by a command line call • Can be automated • Interactive • User interact with the tool through a tool-provided GUI • Cannot be automated • Web service • No tool – procedure distributed on a web service actived by a programming language call • Can be automated

  25. CORE Distributed Deployment CORE Environment Batch-Interactive runtime Runtime Definition Repository GUI Runtime agent Process Engine Integration APIs Web service runtime Runtime Remote activation Web container Web service client

  26. Conclusions and Possible Future Work • CORE implementation is a proof-of-concept prototype showing: • Real implementation of industrialized (standardized and automated) statistical processes • Reuse of IT tools possibly developed on different platforms and by different NSIs • GSBPM-aware services implementation • A unique common data model enabling integration of heterogeneous data exchanged between services • Openess to evolving statistical information models (e.g. GSIM) through a dedicated slot

  27. SUMMARIZING: What we will see in the DEMO

More Related