.Stat Suite documentation

.Stat Suite Open Source Framework

Table of Content

Component-based technical architecture

The high-level technical architecture can be schematised as follow with a topology example (with two .Stat Suite SDMX data spaces):

Stat Suite high-level architecture

Each .Stat Suite main module (as well as their components) can be used together or separately.

In the following are a few typical use cases on how an organisation can make use of these modules:

A) Adding the Data Explorer to an existing fully-fledged back-end solution

Pre-requisites: A statistical database with its own data management features and with a public SDMX API (SDMX 2.1 Rest incl. SDMX-JSON); hardware (see the system requirements in the installation guides)

Objectives: Add a public web application as GUI for end-users to ease to consumption of the data

Approach: Use only the .Stat Data Explorer components and plug them to the public SDMX API

B) Adding the Data Lifecycle Manager to an existing back-end solution to extend its management functionalities

Pre-requisites: A statistical database with a protected SDMX API (SDMX 2.1 Rest incl. SDMX-JSON); hardware (see the system requirements in the installation guides)

Objectives: Add a protected web application as GUI for data producers to ease to management of the data

Approach: Use only the .Stat Data Lifecycle Manager components and plug them to the protected SDMX API

C) Adding the .Stat Core as a back-end solution (SDMX data store)

Pre-requisites: Management and Data portal GUIs that can connect to an SDMX end-point through SDMX API calls; SQL Server licenses; hardware (see the system requirements in the installation guides)

Objectives: Add a protected SDMX datastore

Approach: Use only the .Stat Core components and plug them into the existing GUIs software

D) Adding the .Stat Suite as a full-stack statistical data management and dissemination solution

Pre-requisites: SQL Server licenses; hardware (see the system requirements in the installation guides)

Objectives: Add the full .Stat Suite with one or more SDMX data stores (to handle independent data lifecycle versions)

Approach: Define specific topology for your needs using all .Stat Suite components or partially

Technical stack

The following technologies are used for the development of the .Stat Data Explorer components and the .Stat Data Lifecycle Manager components:

kubernetes gcp docker solr mongo es6 nodejs npmjs git gitlab

The following technologies are used for the development of the .Stat Core components:

kubernetes gcp docker git gitlab sql server .net core visual studio nuget

.Stat Data Explorer module

.Stat Suite Data Explorer logo

Non-technical overview

graph LR; id0{user}; id1((data-explorer)); id6((share-viewer)); id2[share service]; id3[config service]; id4[search service]; id5(sdmx public endpoint); id0-->id1; id2-->id6; id0-->id6; id1-->id2; id1-->id3; id1-->id4; id4-->id3; id2-->id3; id6-->id3; id1-->id5;

Data Explorer app

short description

This web app is the main GUI for (external) users to find, understand and use the statistical data stored in the SDMX end point(s).

demo (light)
  1. go to https://de-qa.siscc.org (tenant: oecd, env: staging)
  2. home page
  3. search by facets (click on a facet)
  4. fine-tune search results (play with facet selection)
  5. data view with search params as data query (click on a dataflow)
  6. fine-tune data selection (play with filter selection)
  7. fine-tune data view (play with customize)
technical aspects

Share service

short description

This service (and related database) is used to store and retrieve user-defined data tables and charts as small JSON objects containing the related configurations.

demo
  1. go to any visualisation page (table view or chart view) of https://de-demo.siscc.org/
  2. click on share button
  3. fill the form (email adress)
  4. click on publish ‘Request URL and embed code’
  5. check email
  6. follow the confirmation link
  7. check the shared view
flow
graph LR id0[user] id1((share-requester)) id2[share service] id3(share db) id4((share-viewer)) id0 --- id1 id1 -->|1. share chart request| id2 id2 -->|2. returns id and chart url| id1 id2 -->|3. chart temporary stored| id3 id2 -->|4. confirmation email| id0 id0 -->|5. user confirmation| id2 id2 -->|6. chart confirmed| id3 id2 -->|7. redirect user|id4
technical aspects

Share-Viewer App

short description

This web app is a companion GUI for (external) users to display user-defined, shared data tables and charts e.g. in embedded views or through shared links.

demo (light)
  1. Go to https://dv-demo.siscc.org/ (to get a dedicated viewer representation, follow the Share demo)
technical aspects

Search service

short description

This service is a .Stat-specific proxy to an SolR engine to index SDMX dataflows categorised and accessible in one or more SDMX end points and to execute faceted search queries on these SDMX dataflows.

features/demos
  • free-text faceted search
  • results are sorted by relevance and paginated
  • free-text search hit: highlight the first occurance of all search terms
  • handles hierarchical facet content (from SDMX codelists)
  • multi-language enabled
  • single term search: Unemployment rate by sex and disability status 1 result
  • phrase term search: "Unemployment rate by sex and disability status" 1 result
  • tagged search: name:enterprises
  • multiple tagged search with phrase: name:enterprises description:"tourism"
  • negate search: -seasonally 25/33 results (witness: seasonally 8/33 results)
  • synonyms
  • stemming
  • ASCII-folding
architecture
graph LR id0[ExpressJS + evtX] id2[SolR] id4(mongo) id5[config] id7((user)) id9((admin)) id0 -->|GET| id7 id9 -->|POST| id0 id0 --- id4 id0 --- id5 id0 --- id2 subgraph sdmx-faceted-search id0 end
technical aspects

Proxy Service

short description

The Proxy service handles route request depending on urls (https://<app+env+organisation+scope>.<domain>), and sets tenant headers depending on host to instruct target application. <app+env+organisation+scope>.<domain> could be replaced by a dedicated DNS entry, e.g. https://<app>.<env>.<organisation>.<scope>.<domain>.

technical aspects

Config Service

short description

The Configuration service centralises all configuration resources used by other services. It is a web server providing requested configuration, not exposed to users. Based on git versioned configuration data.

technical aspects

Dependencies

short description

A footer view component for Table and Charts.

technical aspects
(Dependency) rcw-charts
short description

Set of configurable charts based on d3.

technical aspects
(Dependency) visions
short description

Visions is a library of React components. The goal is to handle interfaces and very limited logic.

technical aspects
(Dependency) sis-header
short description

A header view component for Tables and Charts.

technical aspects
(Dependency) dotstatsuite components
short description

Is a library of components. [more discription to come]

technical aspects
(Dependency) dotstatsuite-sdmxjs
short description

Is a library of components for the parsing of SDMX-JSON messages. [more discription to come]

technical aspects

Multi-tenant architecture

demo (staging)
tenant specs url
oecd own sdmx oecd staging endpoint, OECD-staging https://de-demo.siscc.org/
technical overview
graph LR id1(browser) id2((internet)) id3[proxy] id4((data-explorer)) id12((share-viewer)) id5[search] id6[share] id7[config] id8[solr] id9(mongo) id10(mongo) id11((x)) id13[sdmx/nsi] id14[transfer] id15(mssql) id1 --> id2 id2 --> id3 subgraph oecd cluster on GCP id3 -->|app & organisation & scope| id11 id3 -->|assets| id7 id11 --> id4 id11 --> id5 id11 --> id6 id11 --> id12 id4 --> id7 id5 --> id7 id5 --> id8 id5 --> id9 id6 --> id7 id6 --> id10 id12 --> id7 end id2 --> id13 id2 --> id14 subgraph oecd cluster on GCP id13 --> id15 id14 --> id15 end
demo: add/update a tenant
  1. update list of tenants: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-config-data/-/blob/develop/configs/tenants.json
  2. add tenant config (data-explorer & sdmx): https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-config/tree/develop/data/dev/configs
  3. add tenant referenced assets (data-explorer): https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-config-data/-/tree/develop/assets/
  4. check the tenant at http://de-staging-tenant=.siscc.org/
  5. update something in the configuration and check the update
demo: add/update an app for the tenant (siscc workshop hands-on)
  1. fork from wepapp
  2. inject pre-coded app (to create)
  3. declare it somewhere
  4. check the app at http://<app>.staging.<tenant>.siscc.org
  5. update the app
  6. check the app at http://<app>.staging.<tenant>.siscc.org

.Stat Data Lifecycle Manager module

.Stat Suite Data Lifecycle Manager logo

Data Data Lifecycle Manager app

short description

This web app is the main GUI for statistical data teams to efficiently produce and prepare high-quality statistical data and metadata for dissemination.

demo (light)
  1. go to https://dlm-qa.siscc.org/
technical aspects

.Stat Core module

.Stat Suite CORE logo

technical overview

graph LR id1((de & share-viewer)) id2((dlm)) id3[sdmx/nsi] id4[sdmx/nsi plugin] id5[transfer] id6[data-access] id7[common] id8(structure-db) id9(data-db) id10[auth-management] id11(auth-log-db) id1 --> id3 id2 --> id3 id2 --> id5 id2 --> id10 subgraph sdmx-ri / eurostat id3 --> id8 end subgraph .stat core id3 --> id4 id4 --> id6 id4 --> id7 id5 --> id6 id5 --> id7 id6 --> id9 id6 --> id11 id10 --> id11 end id6 --> id8

Transfer service

short description

This web service allows uploading statistical data and their referential metadata to a .Stat Core data space as well as transfering them between different .Stat Core data spaces.

demo (light)
  1. go to https://transfer-qa.siscc.org/swagger (tenant: oecd, env: qa)
technical aspects

SDMX service

(also named SDMX-RI NSI web service (c) Eurostat)

short description

This web service allows uploading data structures to a .Stat Core data space as well as retrieving data structures, data and their referential metadata from a .Stat Core data space.
This RESTful web service is developed by Eurostat and is fully SDMX compliant.

demo (light)
  1. go to http://nsi-stable-qa.siscc.org/ (tenant: oecd, env: qa)
technical aspects

Authorisation service

short description

This web service allows managing user access rights for data structures and data/referential metadata in .Stat Core data spaces.

demo (light)
  1. go to http://authz-qa.siscc.org/swagger (tenant: oecd, env: qa)
technical aspects

Dependencies

(Dependency) Data Access library to access the .Stat Core Data Store
short description

This library is used by the Transfer service to upload and transfer statistical data structures to and between .Stat Core data spaces.

technical aspects
(Dependency) Core Common library
short description

This library is used for shared code in the .Stat Core components. These inlude logs, localisation and configuration.

technical aspects

DevOps implementation

See: .Stat DevOps implementation

Component list and dependency matrix

FlowMatrixInfo__1_.xlsx

Contributing to an open source repository

https://www.gienverschatse.com/oss/2018/02/06/flow-of-open-source/