.Stat Suite documentation

Description of .Stat Suite Docker images

Table of Content

Using Docker technology, the three .Stat Suite main modules Data Explorer, Data Lifecycle Manager and .Stat Core or their components are containerised as ready-to-use Docker images, which can be freely re-used to easily compose a new topology (system architecture) by anyone in their own cloud or premises.
This section describes where to find and how to use the .Stat Suite Docker images.

All .Stat Suite Docker images are located under https://hub.docker.com/u/siscc. Please see each repository for detailed information on how to use these.

Note: Docker technology is a commonly used containerisation technology, and we will mainly list here our ready-to-use Docker images.

Docker image tag policy

We tag our releases with different tags, which have different purposes: image

  • master tag: Currently latest available official release (‘production-ready’) version of a component. This constantly evolves, including in a non-backward-compatible manner.
  • Name tag, e.g., dragonfruit: Currently latest available version of a specific official release (‘production-ready’) across the components of either .Stat CORE (.Net components) or the .Stat JS-based components. This includes all subsequent backward-compatible hotfix versions of that release.
  • Semantic version number, e.g., v10.0.0: Unique semantic identifier of a component version.
  • Hex-number, e.g. 6c6c64c5: Unique numeric identifier of a component version.

At release time, all these tags lead to the same digest number, e.g., c067b78c99cc. However, when new releases are added, some of these tags start to point to newer versions.

Please consult the changelog and the components’ readme files for details on the content of releases and their versions.

It is up to the installing organisation to decide the tags to use depending on the organisational needs and constraints.

  • A fully static approach would be to use the hex-number per component.
  • The easier understandable but still static approach would be to use the semantic version number per component.
  • A compromise between stability and minimizing important issues (security, …) would be to use the release name tag per release (per .Net and JS components). This makes also sure that all the component versions are compatible between each other. Hotfixes will be deployed automatically.
  • An opportunistic approach would be to use the master tag, which makes sure that the installation is always most up to date within the target environment. This might be useful for functional staging/quality assurance purposes.

.Stat Data Explorer components

Data Explorer app

This web app is the main GUI for (external) users to find, understand and use the statistical data stored in the SDMX end point(s).

Share service

This service (and related database) is used to store and retrieve user-defined data tables and charts as small JSON objects containing the related configurations.
A Mongo database is used to store shared objects (tables or charts). Share server is not auth protected, so any robot can spam it. In order to avoid it, many mechanisms are in place:

Share-Viewer App

This web app is a compagnon GUI for (external) users to display user-defined, shared data tables and charts e.g. in embedded views or through shared links.

Search service

This service is a .Stat-specific proxy to an SolR engine to index SDMX dataflows categorised and accessible in one or more SDMX end points and to execute faceted search queries on these SDMX dataflows. A static schema is defined in the config. A dynamic schema is derivated from the indexed SDMX dataflows.

Limitations:

  • reconciliate sdmx and search data (e.g. dataset order, facet value order) -> cache server
  • datasources & config
  • performance (benchmark to do)
  • how to index (e.g. individual dataset = SDMX dataflow)

Proxy service

The Proxy service handles route requests by mapping the hostname part of URLs to tenant-application pairs. Requests with hostnames that match the defined mappings are forwarded to the matched target application, and tenant headers are set for the matched tenant.

Config Service

The Configuration service centralises all configuration resources used by other services. It is a web server providing requested configuration, not exposed to users. Based on git versioned configuration data.

.Stat Data Lifecycle Manager components

Data Lifecycle Manager app

This web app is the main GUI for statistical data teams to efficiently produce and disseminate high-quality statistical data and metadata.

.Stat Core components

Transfer service

This web service is used for statistical data (and later referential metadata) for their upload, download and transfer between different .Stat Core Data Stores:

Configuration

Configuration values can be set using a config file located in the root of application or by using environment variables.
All files with *.json extension and which are located in the config folder are considered as configuration files. The name of the file is arbitrary (except for the log4net.config file) and configuration values can be loaded from a single or multiple files.

Example configurations

Setting environment variables when using Docker-compose

Setting environment variables when using Kubernetes

log configuration using a configuration file

Dataspaces configuration using a configuration file with connection strings to Structure, Management & Data databases. At least 1 dataspace is required.

Localized user and error messages

Sample usage of docker:

  • Docker with using configuration from file:

Before running sample you would need to copy required configuration files to your locally mapped folder </path-to/my/config>

docker run -it --rm -p 80:80 \
-v </path-to/my/config>:/app/config \
-v </path-to/my/logs>:/app/logs \
--name transfer \
siscc/dotstatsuite-core-transfer
  • Docker with connection strings from environment variables:
docker run -it --rm -p 80:80 \
-v /path-to/my/config:/app/config \
-v /path-to/my/logs:/app/logs \
--name transfer \
-e spacesInternal:0:msConnectionString=my-structure-db-connection \
-e spacesInternal:0:managementConnectionString=my-managemenet-db-connection \
-e spacesInternal:0:dataStoreConnectionStrings:0:connectionString=my-data-db-connection \
siscc/dotstatsuite-core-transfer

Schema:

graph LR dlm((dlm)) transfer[transfer] common-nuget[common] access-nuget[data-access] ms_db(structure-db) data_db(data-db) auth-log-db(auth-log-db) dlm-->transfer subgraph transfer backend transfer--> common-nuget transfer--> access-nuget end subgraph database layer access-nuget--> ms_db access-nuget--> auth-log-db access-nuget --> data_db end

SDMX service

(also named SDMX-RI NSI web service (c) Eurostat)

This web service is used for statistical data structures for their upload and download to and from a .Stat Core Data Store.

Configuration

Configuration is loaded from config directory located in the root of application.
All files with *.json extension are considered as configuration files. The name of the file is not important (except app.config & log4net.config), and it’s not important if the configuration values are loaded from 1 single file or multiple files.

docker run -it --rm -p 80:80 \
-v /path-to/my/config:/app/config \
-v /path-to/my/logs:/app/logs \
--name nsi-ws \
siscc/dotstatsuite-core-sdmxri-nsi

Authorisation service

This web service is used for managing user access rights to data structures and data in .Stat Core Data Stores.

Database configuration tool

The Database Configuration Tool can be used for the configuration of MS SQL server databases in situations where those databases cannot be accessed directly, e.g. in containerised environments like the SIS-CC DevOps environments.

It currently supports the following tasks:

  • Set database recovery mode set to SIMPLE (always executed)
  • Adjust maximum size of Data & Log files (optional)

Detailed information can be found here.

See more about the Recommended infrastructure requirements for all the .Stat Core components listed above; Including the Transfer service, SDMX service and Authorisation service.