Introduction to Stat Suite: Open Source Framework

1. Introduction to .Stat Suite Open Source Framework Prepared and presented by Nicolas Briemant from RedPelicans

2. Foreword & demo • Objective: provide an overview of the new .Stat Suite • Setup: 60/30 min (presentation/questions) • Demo: • http://data-explorer.staging.oecd.redpelicans.com (tenant OECD) • Other tenants: Cambodia, iStat, ILO • Search page, Visualisation page, Share feature, Viewer app • Dependencies in inspector

3. Architecture overview

4. Multi-tenant architecture(1/4): proxy service • repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-proxy • docker: https://cloud.docker.com/u/siscc/repository/docker/siscc/dotstatsuite-kube-proxy • Entry point of the architecture • Handle route request depending on urls: https://<app>.<env>.<tenant>.redpelicans.com • Set tenant header to instruct target application • Note: <tenant>.redpelicans.com could be replaced by a dedicaded DNS entry, iehttps://<app>.<env>.oecd.org • Routes definition in data/route.json

5. Multi-tenant architecture (2/4): config service • repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-config • docker: https://cloud.docker.com/u/siscc/repository/docker/siscc/dotstatsuite-config-dev • Centralize all configuration resources used by other services • Web server which provide requested configuration, not exposed to users • Gitversioned configuration data • Tenants definition in /configs/tenants.json

6. Multi-tenant architecture(3/4): webapps • All webapps are forkedfromhttps://gitlab.com/sis-cc/.stat-suite/dotstatsuite-webapp • Boilerplate multi-tenant ready: webapp project includes a nodejsbackend to dynamically generate html (Partial Server Side rendering) with data embedded in script tags • When webapp starts, data are available to be processed without any later requests to the backend • https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-webapp/forks • Data-explorer • Data-viewer

7. Multi-tenant architecture(4/4): flow • Browser requests https://data-explorer.staging.oecd.redpelicans.com • oecd.redpelicans.com is an alias to the GCP cluster static IP address • Kubernetes ingress service routes the request to proxy service • Proxy was setup through routes definition: • to map oecd.redpelicans.comwith tenant oecd • to map data-explorer.stagingto the data-explorer service in staging namespace of the GCP cluster • Data-explorer nodeJS server receives request http://data-explorer?tenant=oecd • Data-explorerserver requests config server to get: • /oecd/data-explorer/i18n/<lang>.json • /oecd/data-explorer/settings.json • Data-explorer server returns index.html + config data + data-explorer's local config • Data-explorer single page application is mounted with dedicated configuration

8. Components & Functions • Components are building blocks, functions are steps of a flow • Hosts (webappslike data-explorer) composes UI from components and behaviorsfromfunctions • ui-components • Description: set of visual components in React(localised, themable, responsive) • Repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-ui-components • Demo: in progress (local storybook) • sdmxjs • Description: set of functions to manipulatesdmx • Repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-sdmxjs • Demo: http://sdmxjs.staging.oecd.redpelicans.com/ • npmregistry: https://www.npmjs.com/settings/sis-cc/packages

9. DevOps (1/3): CI • Gitlab • all repositories are under https://gitlab.com/sis-cc/.stat-suite • each repository defines its pipelines in gitlab-ci.ymlfile • Develop branch represents qa and master branch staging* • Dockerhub • all images are under https://cloud.docker.com/u/siscc/repository/list • tags are latest, develop and commit hash • Each repository defines its image(s) in Dockerfilefile(s) • Npm • npm packages are published under https://www.npmjs.com/settings/sis-cc/packages • tags are only on commits in master and trigger a publish • Tests • Tests are mandatory • https://gitlab.com/snippets/1847834#list-of-webappsservicespackages

10. DevOps(2/3): CD • Repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-kube-rp • Centralize all kubernetesstrategies • Rely on a kubernetes cluster hosted in Google Cloud platform • 2 namespaces: qa and staging*

11. DevOps(3/3): flow

12. Search service (1/2): overview • repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-sdmx-faceted-search • docker: https://cloud.docker.com/u/siscc/repository/docker/siscc/dotstatsuite-sdmx-faceted-search • a static schema is defined in the config • a dynamic schema is derivated from dataflows • current config: http://sfs.staging.oecd.redpelicans.com/admin/config?api-key=secret

13. Search service (2/2): features • UI features • Paginated • Highlights for free textfields • Handleshierarchicalfacets • Multilang* • Free textabilities: • single term search • phrase term search • tagged search • multiple tagged search with phrase • negate search • Technicalfeatures • Solrisdecoupledfrom the webservice (sfs) • Not multitenant • Limitations • Reconciliatesdmx and searchdata

14. Share service • repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-share • docker: https://cloud.docker.com/u/siscc/repository/docker/siscc/dotstatsuite-share • Web server which store and retrieve shared data using a redis database • share server is not auth protected, so any robot can spam it; to avoid that, many mechanisms are in place: • charts are temporary stored only during redisChartTTL seconds before beeing deleted unless beeingconfirmed • share server check POST calls rates, over maxRatePerIP per second, POST calls, per IP, are rejected with a 419 HTTP code • POST body are size limited to maxChartSize

15. Questions Thankyou and seeyou at the basecamp…