.Stat Suite documentation

Description of .Stat Suite Docker images

Table of Content

Using Docker technology, the three .Stat Suite main modules Data Explorer, Data Lifecycle Manager and .Stat Core or their components are containerised as ready-to-use Docker images, which can be freely re-used to easily compose a new topology (system architecture) by anyone in their own cloud or premises.
This section describes where to find and how to use the .Stat Suite Docker images.

All .Stat Suite Docker images are located under https://hub.docker.com/u/siscc. Please see each repository for detailed information on how to use these.

Note: Docker technology is a commonly used containerisation technology, and we will mainly list here our ready-to-use Docker images.


.Stat Data Explorer components

Data Explorer app

This web app is the main GUI for (external) users to find, understand and use the statistical data stored in the SDMX end point(s).

Share service

This service (and related database) is used to store and retrieve user-defined data tables and charts as small JSON objects containing the related configurations.
A Redis database is used to store shared objects (tables or charts). Share server is not auth protected, so any robot can spam it. In order to avoid it, many mechanisms are in place:

Share-Viewer App

This web app is a compagnon GUI for (external) users to display user-defined, shared data tables and charts e.g. in embedded views or through shared links.

Search service

This service is a .Stat-specific proxy to an SolR engine to index SDMX dataflows categorised and accessible in one or more SDMX end points and to execute faceted search queries on these SDMX dataflows. A static schema is defined in the config. A dynamic schema is derivated from the indexed SDMX dataflows.

Limitations:

  • reconciliate sdmx and search data (e.g. dataset order, facet value order) -> cache server
  • datasources & config
  • performance (benchmark to do)
  • how to index (e.g. individual dataset = SDMX dataflow)

Proxy service

The Proxy service handles route requests by mapping the hostname part of URLs to tenant-application pairs. Requests with hostnames that match the defined mappings are forwarded to the matched target application, and tenant headers are set for the matched tenant.

Config Service

The Configuration service centralises all configuration resources used by other services. It is a web server providing requested configuration, not exposed to users. Based on git versioned configuration data.


.Stat Data Lifecycle Manager components

Data Lifecycle Manager app

This web app is the main GUI for statistical data teams to efficiently produce and disseminate high-quality statistical data and metadata.


.Stat Core components

Transfer service

This web service is used for statistical data (and later referential metadata) for their upload, download and transfer between different .Stat Core Data Stores:

Configuration

Configuration values can be set using a config file located in the root of application or by using environment variables.
All files with *.json extension and which are located in the config folder are considered as configuration files. The name of the file is arbitrary (except for the log4net.config file) and configuration values can be loaded from a single or multiple files.

Example configurations

Setting environment variables when using Docker-compose

Setting environment variables when using Kubernetes

log configuration using a configuration file

Dataspaces configuration using a configuration file with connection strings to Structure, Management & Data databases. At least 1 dataspace is required.

Localized user and error messages

Sample usage of docker:

  • Docker with using configuration from file:

Before running sample you would need to copy required configuration files to your locally mapped folder </path-to/my/config>

docker run -it --rm -p 80:80 \
-v </path-to/my/config>:/app/config \
-v </path-to/my/logs>:/app/logs \
--name transfer \
siscc/dotstatsuite-core-transfer
  • Docker with connection strings from environment variables:
docker run -it --rm -p 80:80 \
-v /path-to/my/config:/app/config \
-v /path-to/my/logs:/app/logs \
--name transfer \
-e spacesInternal:0:msConnectionString=my-structure-db-connection \
-e spacesInternal:0:managementConnectionString=my-managemenet-db-connection \
-e spacesInternal:0:dataStoreConnectionStrings:0:connectionString=my-data-db-connection \
siscc/dotstatsuite-core-transfer

Schema:

graph LR dlm((dlm)) transfer[transfer] common-nuget[common] access-nuget[data-access] ms_db(structure-db) data_db(data-db) auth-log-db(auth-log-db) dlm-->transfer subgraph transfer backend transfer--> common-nuget transfer--> access-nuget end subgraph database layer access-nuget--> ms_db access-nuget--> auth-log-db access-nuget --> data_db end

SDMX service

(also named SDMX-RI NSI web service (c) Eurostat)

This web service is used for statistical data structures for their upload and download to and from a .Stat Core Data Store.

Configuration

Configuration is loaded from config directory located in the root of application.
All files with *.json extension are considered as configuration files. The name of the file is not important (except app.config & log4net.config), and it’s not important if the configuration values are loaded from 1 single file or multiple files.

docker run -it --rm -p 80:80 \
-v /path-to/my/config:/app/config \
-v /path-to/my/logs:/app/logs \
--name nsi-ws \
siscc/dotstatsuite-core-sdmxri-nsi

Authorisation service

This web service is used for managing user access rights to data structures and data in .Stat Core Data Stores.


See more about the Recommended infrastructure requirements for all the .Stat Core components listed above; Including the Transfer service, SDMX service and Authorisation service.