.Stat Suite documentation

Description of .Stat Suite Docker images

Table of Content

Using Docker technology, the three .Stat Suite main modules Data Explorer, Data Lifecycle Manager and .Stat Core or their components are containerised as ready-to-use Docker images, which can be freely re-used to easily compose a new topology (system architecture) by anyone in their own cloud or premises.

This section describes where to find and how to use the .Stat Suite Docker images.

Note: Docker technology is a commonly used containerisation technology, and we will mainly list here our ready-to-use Docker images.


All .Stat Suite Docker images are located under https://hub.docker.com/u/siscc. Please see each repository for detailed information on how to use these.

.Stat Data Explorer components

Data Explorer app

This web app is the main GUI for (external) users to find, understand and use the statistical data stored in the SDMX end point(s).

Share service

This service (and related database) is used to store and retrieve user-defined data tables and charts as small JSON objects containing the related configurations.
A Redis database is used to store shared objects (tables or charts). Share server is not auth protected, so any robot can spam it. In order to avoid it, many mechanisms are in place:

  • tables/charts are only temporarily stored for redisChartTTL seconds before being deleted unless confirmed via email link
  • share server checks POST calls rates. Over maxRatePerIP POST calls per second, per IP, are rejected with a 419 HTTP code
  • POST bodies are limited in size to maxChartSize

Share-Viewer App

This web app is a compagnon GUI for (external) users to display user-defined, shared data tables and charts e.g. in embedded views or through shared links.

Search service

This service is a .Stat-specific proxy to an SolR engine to index SDMX dataflows categorised and accessible in one or more SDMX end points and to execute faceted search queries on these SDMX dataflows. A static schema is defined in the config. A dynamic schema is derivated from the indexed SDMX dataflows.

Limitations:

  • reconciliate sdmx and search data (e.g. dataset order, facet value order) -> cache server
  • datasources & config
  • performance (benchmark to do)
  • how to index (e.g. individual dataset = SDMX dataflow)

Proxy service

The Proxy service handles route request depending on urls (https://<app>.<env>.<tenant>.redpelicans.com), and sets tenant headers depending on host to instruct target application. <tenant>.redpelicans.com could be replaced by a dedicated DNS entry, e.g. https://<app>.<env>.oecd.org.

Config Service

The Configuration service centralises all configuration resources used by other services. It is a web server providing requested configuration, not exposed to users. Based on git versioned configuration data.


.Stat Data Lifecycle Manager components

Data Lifecycle Manager app

This web app is the main GUI for statistical data teams to efficiently produce and disseminate high-quality statistical data and metadata.


.Stat Core components

Transfer service

This web service is used for statistical data (and later referential metadata) for their upload, download and transfer between different .Stat Core Data Stores:

  • Upload of SDMX data files (csv, xml) into a .Stat Core data store
  • Upload of Excel data files (using a specific data mapping definition) into a .Stat Core data store
  • Transfer of data between two .Stat Core data stores

Configuration

Configuration is loaded from config directory located in the root of application. All files with *.json extension are considered as configuration files. The name of the file is not important (except log4net.config), and it's not important if the configuration values are loaded from 1 single file or multiple files.

docker run -it --rm -p 80:80 \
-v /path-to/my/config:/app/config \
-v /path-to/my/logs:/app/logs \
--name transfer \
siscc/dotstatsuite-core-transfer
  • docker with connection strings from environment variables:
docker run -it --rm -p 80:80 \
-v /path-to/my/config:/app/config \
-v /path-to/my/logs:/app/logs \
--name transfer \
-e spacesInternal:0:msConnectionString=my-structure-db-connection \
-e spacesInternal:0:managementConnectionString=my-managemenet-db-connection \
-e spacesInternal:0:dataStoreConnectionStrings:0:connectionString=my-data-db-connection \
siscc/dotstatsuite-core-transfer

Schema:

graph LR dlm((dlm)) transfer[transfer] common-nuget[common] access-nuget[data-access] ms_db(structure-db) data_db(data-db) auth-log-db(auth-log-db) dlm-->transfer subgraph transfer backend transfer--> common-nuget transfer--> access-nuget end subgraph database layer access-nuget--> ms_db access-nuget--> auth-log-db access-nuget --> data_db end

SDMX service (also named SDMX-RI NSI web service (c) Eurostat)

This web service is used for statistical data structures for their upload and download to and from a .Stat Core Data Store. The docker image is using a vanilla Eurostat NSI web service image as a base image. It is enriched with a special .Stat Core plugin used to retrieve statistical data structures from a .Stat Core Data Store.

Configuration

Configuration is loaded from config directory located in the root of application. All files with *.json extension are considered as configuration files. The name of the file is not important (except app.config & log4net.config), and it's not important if the configuration values are loaded from 1 single file or multiple files.

  • example configuration: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-kube-core-rp/tree/master/qa/nsi-config

    nsi web service main configuration, more info here In addition to default Eurostat configuration there 2 new values in the appSettings, that are needed for Data plugin DataspaceId - what dataspace is used by NSI for data retrieval
    ConfigDirectory - where to find Plugin *.json configuration files, by default located in the same directory as NSI configs themselves.

    log configuration

    dataspaces configuration with connection strings to Structure, Management & Data databases.

    Localized messages returned back to a user

  • sample usage of docker (provided log4net.config instructs to write to a file in /app/logs directory):

docker run -it --rm -p 80:80 \
-v /path-to/my/config:/app/config \
-v /path-to/my/logs:/app/logs \
--name nsi-ws \
siscc/dotstatsuite-core-sdmxri-nsi

Schema:

graph LR dlm((dlm)) de((data explorer)) nsi[sdmx/nsi] plugin[sdmx/nsi plugin] common-nuget[common] access-nuget[data-access] ms_db(structure-db) data_db(data-db) auth-log-db(auth-log-db) dlm-->nsi de-->nsi subgraph sdmx-ri / eurostat nsi --> plugin end subgraph nuget packages plugin --> common-nuget plugin --> access-nuget end subgraph database layer access-nuget--> ms_db access-nuget--> auth-log-db access-nuget --> data_db end

Authorisation service

This web service is used for managing user access rights to data structures and data in .Stat Core Data Stores.

See more about the Recommended infrastructure requirements for all the .Stat Core components listed above; Including the Transfer service, SDMX service and Authorisation service.


Docker-compose example(s)

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. To learn more about all the features of Compose, see the list of features.

Compose works in all environments: production, staging, development, testing, as well as CI workflows. You can learn more about each case in Common Use Cases.

A .Stat Docker compose example for developers and contributors combining all apps and services and one .Stat Core Data Storage is currently in development and will be available soon.

A first preview of a Docker compose yaml file with .Stat Core services is here:

version: "3"

services:
  nsi-ws:
    image: siscc/dotstatsuite-core-sdmxri-nsi:7.2.0
    container_name: nsi-ws
    ports:
      - "85:80"
    volumes:
      - "./nsi-config:/app/config"
      - "./logs:/app/logs"
    networks:
      - back-network
    depends_on:
      - db

  transfer:
    image: siscc/dotstatsuite-core-transfer:latest
    container_name: transfer
    ports:
      - "86:80"
    volumes:
      - "./transfer-config:/app/config"
      - "./logs:/app/logs"
    networks:
      - back-network
    depends_on:
      - db

  db:
    build: ./mssql/docker
    image: siscc/mssql-with-init
    container_name: mssql
    ports:
      - "1434:1433"
    volumes:
      - "db-data:/var/opt/mssql/data"
      - "./mssql/init:/docker-entrypoint-initdb.d"
    environment:
      ACCEPT_EULA: Y
      SA_PASSWORD: My-Mssql-Pwd-123
      MSSQL_PID: Developer
    networks:
      - back-network

volumes:
  db-data:

networks:
  back-network: