Description of .Stat Suite Docker images
Table of Content
- .Stat Data Explorer components
- .Stat Data Lifecycle Manager components
- .Stat Core components
- Docker compose example(s)
Using Docker technology, the three .Stat Suite main modules Data Explorer, Data Lifecycle Manager and .Stat Core or their components are containerised as ready-to-use Docker images, which can be freely re-used to easily compose a new topology (system architecture) by anyone in their own cloud or premises.
This section describes where to find and how to use the .Stat Suite Docker images.
Note: Docker technology is a commonly used containerisation technology, and we will mainly list here our ready-to-use Docker images.
All .Stat Suite Docker images are located under https://hub.docker.com/u/siscc. Please see each repository for detailed information on how to use these.
.Stat Data Explorer components
Data Explorer app
This web app is the main GUI for (external) users to find, understand and use the statistical data stored in the SDMX end point(s).
- demo: http://de-staging-oecd.redpelicans.com/
- docker: https://hub.docker.com/r/siscc/dotstatsuite-data-explorer
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-data-explorer
This service (and related database) is used to store and retrieve user-defined data tables and charts as small JSON objects containing the related configurations.
A Redis database is used to store shared objects (tables or charts). Share server is not auth protected, so any robot can spam it. In order to avoid it, many mechanisms are in place:
- tables/charts are only temporarily stored for
redisChartTTLseconds before being deleted unless confirmed via email link
- share server checks POST calls rates. Over
maxRatePerIPPOST calls per second, per IP, are rejected with a 419 HTTP code
- POST bodies are limited in size to
- demo: http://share-staging-oecd.redpelicans.com/api/charts/3
- docker: https://hub.docker.com/r/siscc/dotstatsuite-share
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-share
This web app is a compagnon GUI for (external) users to display user-defined, shared data tables and charts e.g. in embedded views or through shared links.
- demo: http://dv-staging-oecd.redpelicans.com/?chartId=3
- docker: https://hub.docker.com/r/siscc/dotstatsuite-data-viewer
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-data-viewer
- server-side rendered (configuration is injected in index.html), no request required from the client to get the configuration
- client bundle expects configuration in
window.CONFIGfor those who want to use directly the static files
This service is a .Stat-specific proxy to an SolR engine to index SDMX dataflows categorised and accessible in one or more SDMX end points and to execute faceted search queries on these SDMX dataflows. A static schema is defined in the config. A dynamic schema is derivated from the indexed SDMX dataflows.
- demo: http://sfs-staging-oecd.redpelicans.com/api/config
- docker: https://hub.docker.com/r/siscc/dotstatsuite-sdmx-faceted-search
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-sdmx-faceted-search
- reconciliate sdmx and search data (e.g. dataset order, facet value order) -> cache server
- datasources & config
- performance (benchmark to do)
- how to index (e.g. individual dataset = SDMX dataflow)
The Proxy service handles route requests by mapping the hostname part of URLs to tenant-application pairs. Requests with hostnames that match the defined mappings are forwarded to the matched target application, and tenant headers are set for the matched tenant.
- docker: https://hub.docker.com/r/siscc/dotstatsuite-kube-proxy
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-proxy
The Configuration service centralises all configuration resources used by other services. It is a web server providing requested configuration, not exposed to users. Based on git versioned configuration data.
- docker: https://hub.docker.com/r/siscc/dotstatsuite-config-dev
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-config
.Stat Data Lifecycle Manager components
Data Lifecycle Manager app
This web app is the main GUI for statistical data teams to efficiently produce and disseminate high-quality statistical data and metadata.
- demo: http://dlm-staging-oecd.redpelicans.com/
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-data-lifecycle-manager
- docker: https://hub.docker.com/r/siscc/dotstatsuite-data-lifecycle-manager
.Stat Core components
This web service is used for statistical data (and later referential metadata) for their upload, download and transfer between different .Stat Core Data Stores:
- Upload of SDMX data files (csv, xml) into a .Stat Core data store
- Upload of Excel data files (using a specific data mapping definition) into a .Stat Core data store
- Transfer of data between two .Stat Core data stores
- demo: http://transfer.qa.core.oecd.redpelicans.com/swagger/index.html
- docker: https://hub.docker.com/r/siscc/dotstatsuite-core-transfer
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-core-transfer
Configuration values can be set using a config file located in the root of application or by using environment variables.
All files with
*.json extension and which are located in the config folder are considered as configuration files. The name of the file is arbitrary (except for the log4net.config file) and configuration values can be loaded from a single or multiple files.
Setting environment variables when using Docker-compose
Setting environment variables when using Kubernetes
log configuration using a configuration file
Dataspaces configuration using a configuration file with connection strings to Structure, Management & Data databases. At least 1 dataspace is required.
Localized user and error messages
Sample usage of docker:
- Docker with using configuration from file:
Before running sample you would need to copy required configuration files to your locally mapped folder </path-to/my/config>
docker run -it --rm -p 80:80 \ -v </path-to/my/config>:/app/config \ -v </path-to/my/logs>:/app/logs \ --name transfer \ siscc/dotstatsuite-core-transfer
- Docker with connection strings from environment variables:
docker run -it --rm -p 80:80 \ -v /path-to/my/config:/app/config \ -v /path-to/my/logs:/app/logs \ --name transfer \ -e spacesInternal:0:msConnectionString=my-structure-db-connection \ -e spacesInternal:0:managementConnectionString=my-managemenet-db-connection \ -e spacesInternal:0:dataStoreConnectionStrings:0:connectionString=my-data-db-connection \ siscc/dotstatsuite-core-transfer
SDMX service (also named SDMX-RI NSI web service (c) Eurostat)
This web service is used for statistical data structures for their upload and download to and from a .Stat Core Data Store. The docker image is using a vanilla Eurostat NSI web service image as a base image. It is enriched with a special .Stat Core plugin used to retrieve statistical data structures from a .Stat Core Data Store.
- demo: http://nsi-staging-oecd.redpelicans.com/
- docker: https://hub.docker.com/r/siscc/dotstatsuite-core-sdmxri-nsi
- docker of original Eurostat SDMX-RI NSI web service: https://cloud.docker.com/u/siscc/repository/docker/siscc/sdmxri-nsi
- repository of .Stat Core plugin: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-core-sdmxri-nsi-plugin
- repository of original Eurostat SDMX-RI NSI web service: https://citnet.tech.ec.europa.eu/CITnet/stash/projects/SDMXRI/repos/nsiws.net
Configuration is loaded from config directory located in the root of application.
All files with *.json extension are considered as configuration files. The name of the file is not important (except app.config & log4net.config), and it’s not important if the configuration values are loaded from 1 single file or multiple files.
example configuration: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-kube-core-rp/tree/master/qa/nsi-config
nsi web service main configuration, more info here In addition to default Eurostat configuration there 2 new values in the appSettings, that are needed for Data plugin DataspaceId - what dataspace is used by NSI for data retrieval
ConfigDirectory - where to find Plugin *.json configuration files, by default located in the same directory as NSI configs themselves.
dataspaces configuration with connection strings to Structure, Management & Data databases.
Localized messages returned back to a user
sample usage of docker (provided log4net.config instructs to write to a file in /app/logs directory):
docker run -it --rm -p 80:80 \ -v /path-to/my/config:/app/config \ -v /path-to/my/logs:/app/logs \ --name nsi-ws \ siscc/dotstatsuite-core-sdmxri-nsi
This web service is used for managing user access rights to data structures and data in .Stat Core Data Stores.
- demo: http://authz-siscc.redpelicans.com/swagger/index.html
- docker: https://hub.docker.com/r/siscc/dotstatsuite-core-auth-management
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-core-auth-management
See more about the Recommended infrastructure requirements for all the .Stat Core components listed above; Including the Transfer service, SDMX service and Authorisation service.
Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. To learn more about all the features of Compose, see the list of features.
Compose works in all environments: production, staging, development, testing, as well as CI workflows. You can learn more about each case in Common Use Cases.
A .Stat Docker compose example for developers and contributors combining all apps and services and one .Stat Core Data Storage is currently in development and will be available soon.
A first preview of a Docker compose yaml file with .Stat Core services is here:
version: "3" services: nsi-ws: image: siscc/dotstatsuite-core-sdmxri-nsi:7.12.0 container_name: nsi-ws ports: - "85:80" volumes: - "./nsi-config:/app/config" - "./logs:/app/logs" networks: - back-network depends_on: - db transfer: image: siscc/dotstatsuite-core-transfer:latest container_name: transfer ports: - "86:80" volumes: - "./transfer-config:/app/config" - "./logs:/app/logs" networks: - back-network depends_on: - db db: build: ./mssql/docker image: siscc/mssql-with-init container_name: mssql ports: - "1434:1433" volumes: - "db-data:/var/opt/mssql/data" - "./mssql/init:/docker-entrypoint-initdb.d" environment: ACCEPT_EULA: Y SA_PASSWORD: My-Mssql-Pwd-123 MSSQL_PID: Developer networks: - back-network volumes: db-data: networks: back-network: