.Stat DLM configuration
Table of Content
- Authentication
- Data space names
- Data space colors
- Data space InsertNewItems parameter
- Use data space as internal space or external space
- Auto-authenticate to external source
- Use native NSI WS authentication for external source
- Define data preview per space
- List of SDMX artefact types
- Default data validations option
- Upload size limit
- Logbook submission time boundaries
- List of the standard roles for user permissions
- Override the default Authorisation-Management service URL
- Allow managing permissions on a specific external space
- FMR Workbench integration
For the tenant (organisation
and scope
) and data space definitions please see here.
Authentication
The Data Lifecycle Manager requires an authentication configuration through an OpenID-Connect compliant identity provider.
For more information, see here.
Data space names
Define the name of an internal or external data space displayed to the end-user in the DLM UI, regardless the datasourceIds
.
- in
dotstatsuite-config-data/<env>/configs/tenants.json
"spaces": {
"staging:SIS-CC-stable": {
"label": "SIS-CC stable"
},
}
Data space colors
Define the font color and background color for each internal and external data space. The colors defined are applied in all options, features and representations of the data spaces (e.g. in the selectors and list of results, upload and transfer features, etc.)
- in
dotstatsuite-config-data/<env>/configs/tenants.json
under a DLMscope
where"color"
refers to the font color of the name, and"backgroundColor"
refers to the background colored area.
"scopes": {
"dlm": {
"type": "dlm",
"label": "dlm"
},
"spaces": [
{
"id": "staging:SIS-CC-stable",
"color": "#0549ab",
"backgroundColor": "#b7def6",
"label": "staging:SIS-CC-stable"
}
}
Data space InsertNewItems parameter
Released in March 29, 2020 Release .Stat Suite .NET 3.3.0
The InsertNewItems
parameter located in the app.config
of the NSI web service allows to update final or non-final SDMX artefacts (of ItemScheme types, e.g. codelist) without changing the version of it.
When the parameter is set to:
- name: INSERT_NEW_ITEM_SCHEME_VALUES
value: "true"
Then it is possible for instance to add a new code in a codelist without modifying its Agency/ID/Version.
Use data space as internal space or external space
Internal spaces in the DLM are those that allow for data imports or copies from other spaces. For this, a related transfer service is required and the transferUrl
parameter instructs the DLM to automatically show that data space as internal, automatically authenticate against that space with the common auth token and use the given transfer service instance for data imports. Otherwise, the data space is shown as external.
The transferUrl
parameter is to be set in the dotstatsuite-config-data/<env>/configs/tenants.json
file under a DLM scope
for a given space
to the rest root URL of the related transfer service.
"scopes": {
"dlm": {
"type": "dlm",
"label": "dlm",
"oidc": {
"authority": "https://keycloak.siscc.org/auth/realms/OECD",
"client_id": "app"
},
"spaces": [
{
"id": "staging:SIS-CC-stable",
"color": "#0549ab",
"backgroundColor": "#b7def6",
"label": "staging:SIS-CC-stable",
"transferUrl": "https://transfer-demo.siscc.org/2",
"dataExplorerUrl": "https://de-qa.siscc.org"
},
Auto-authenticate to external source
Released in August 3, 2022 Release .Stat Suite JS quark
In case a .Stat CORE data space is external because its definition doesn’t define a transfer service to be used for imports, but it uses the same identity provider that is defined for the internal spaces, then the authenticateToRemoteURL
parameter instructs the DLM to automatically authenticate against that source with the common auth token as well as to allow the transfer web service defined for an internal space (when called through one of the /import/sdmxFile
or /validate/sdmxFile
methods) to authenticate against that data source with that same auth token.
The authenticateToRemoteURL
parameter is to be set to true
in the dotstatsuite-config-data/<env>/configs/tenants.json
file under a DLM scope
for a given space
.
"scopes": {
"dlm": {
"type": "dlm",
"label": "dlm"
},
"spaces": [
{
"id": "staging:SIS-CC-reset",
"color": "#0549ab",
"backgroundColor": "#e2f2fb",
"label": "staging:SIS-CC-reset",
"dataExplorerUrl": "https://de-qa.siscc.org",
"authenticateToRemoteURL": true
}
}
Use native NSI WS authentication for external source
Introduced in December 14, 2021 Release .Stat Suite JS 11.0.0
In case an external data source is accessible through an SDMX web service based on Eurostat’s SDMX-RI “NSI” component (so called NSI web service), with the hasExternalAuth
parameter the DLM can be instructed to authenticate against that web service using the native NSI authentication mechanism (implemented by Eurostat) based on HTTP basic access authentication (BA).
The user will have to enter the required credentials through a specific dialog box, as described here.
For more information on how to set the hasExternalAuth
parameter, please see here
Define data preview per space
With the dataExplorerUrl
parameter the DLM is instructed to allow previewing structures and data with the Data Explorer for an internal or external DLM data space.
The dataExplorerUrl
parameter is to be set in the dotstatsuite-config-data/<env>/configs/tenants.json
file under a DLM scope
for a given space
to the URL of the Data Explorer to be used.
"scopes": {
"dlm": {
"type": "dlm",
"label": "dlm"
},
"spaces": [
{
"id": "staging:SIS-CC-reset",
"color": "#0549ab",
"backgroundColor": "#e2f2fb",
"label": "staging:SIS-CC-reset",
"dataExplorerUrl": "https://de-qa.siscc.org"
}
}
List of SDMX artefact types
Define the types of SDMX structural artefacts selectable in the DLM ‘Filter by type’ selector (see functional specs). Only artefacts of those types can be displayed in the DLM user interface.
- in
dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/settings.json
"sdmx": {
"typeIds": [
"agencyscheme",
"categoryscheme",
"categorisation",
"codelist",
"conceptscheme",
"contentconstraint",
"datastructure",
"dataflow",
"hierarchicalcodelist",
"metadatastructure",
"metadataflow",
"structureset"
]
}
Default data validations option
Introduced in December 5, 2022 Release .Stat Suite JS spin
Define, in the ‘Data upload’ and ‘Data transfer’ menus, what is the data validations type option selected by default:
- “basic” for Favour speed with basic validations
- “advanced” for Apply advanced validations
- in
dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/settings.json
"app": {
"defaultOptionDataValidation": "advanced"
}
See more about the functional specifications of this feature here.
Upload size limit
Define the file size limitation when uploading data or strutures files applied by the DLM user interface as follow:
Provided in number of bites, the upload file size limit is usually set to 30MB by default (like in the example below).
- in
dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/settings.json
"sdmx": {
"uploadSizeLimit": 30000000
}
Specific configurations are also necessary also for the underlying .Stat CORE services (transfer and NSI). The hosting web server(s) (e.g. IIS, Kestrel, ..) need(s) to be configured to accept http requests with bodies of the required size. All web servers have their own defaults, e.g. the .Net Core builtin web server, called Kestrel, has a default maximum request body size of 30,000,000 bytes, which is approximately 28.6 MB.
In order to change the MaxRequestBodySize
parameter in the transfer service, it will depend on the installation mode of back-end services:
- in case of a docker installation an environment variable of the container should be used to specify the max. size, e.g.:
Kestrel__Limits__MaxRequestBodySize: 52428800
- if you have source code installation, then the config file you created should be moved into the
config
folder of transfer service. Please note that if you also use IIS, then IIS may also have limitations on the maximum file size.
More information on how to make this configuration in Kestrel (on the example of the NSI web service) can be found here. For other web server technologies, please consult their specific documentation.
Logbook submission time boundaries
Released in December 5, 2022 Release .Stat Suite JS spin
Define the boundaries of the submission time logbook filter. Rules are:
- configuration applies to the organisation;
- valid values for start and end param. are “now” or any ISO date string;
- by default it is set to
"start": "2019-01-01"
and"end": "now"
; - if settings are not set, or invalid, then default boundaries are [30 days - now].
- in
dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/settings.json
"logs": {
"transfer": {
"timeExtents": {
"start": "2019-01-01",
"end": "now"
}
}
}
List of the standard roles for user permissions
Introduced in December 5, 2022 Release .Stat Suite JS spin
Define, per organisation, the Standard roles set for the ‘DLM –> Manage permissions –> Add/View/Edit’ form, based on any combination of the granular permissons as defined in the Authorisation-Management Service.
By default, the below set of combinations is pre-defined as standard roles.
References:
- Authorisation-Management API features and granular permissions
- Functional specifications of the DLM GUI
- in
dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/settings.json
"sdmx": {
"permissionGroups": [
{
"id": 3,
"permissions": [1, 2]
},
{
"id": 145,
"permissions": [1, 16, 128]
},
{
"id": 657,
"permissions": [1, 16, 128, 512]
},
{
"id": 3363,
"permissions": [1, 2, 32, 256, 1024, 2048]
},
{
"id": 2051,
"permissions": [1, 2, 2048]
},
{
"id": 4095,
"permissions": [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]
}
]
}
The "permissionGroup"
lists the standard roles using per role a unique "id"
, which represents the sum of the IDs of the underlying granular permissons as defined in the Authorisation-Management service, as well as the list of those IDs.
Each role ID also requires the definition of a localised label in an “override” i18n
folder under dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/
with each language’s translation file. For example, here are the English labels to add corresponding to the above set of roles: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-config-data/-/blob/develop/configs/default/data-lifecycle-manager/i18n/en.json.
Override the default Authorisation-Management service URL
Released in December 5, 2022 Release .Stat Suite JS spin
Define an Authorisation-Management service URL at the DLM scope level that will override the one defined as an env. variable during the deployment. This override is made especially to allow for a specific Authorisation-Management service URL per DLM scope when a DLM instance has several scopes.
- in
dotstatsuite-config-data/<env>/configs/tenants.json
file under a DLMscope
"dlm2": {
"type": "dlm",
"label": "dlm2",
"oidc": {
"authority": "https://...",
"client_id": "app"
},
"authzServerUrl": "https://authz-env.org/1.1",
Allow managing permissions on a specific external space
Released in December 20, 2023 Release .Stat Suite JS yay
Allow, for a specific data space that is defined as external, to manage the users’ permissions through the DLM UI Manage Pemrissions feature. When the config. parameter "allowPermissionMgmt"
is set to ’true’ (default: false) for an external data space, then the data space appears in the DLM “Manage Permissions” tab and users with succifient rights are allowed to manage user access accordingly.
- in
dotstatsuite-config-data/<env>/configs/tenants.json
file under a DLMtenant > spaces
"spaces": {
"staging:SIS-CC-stable": {
"label": "staging:SIS-CC-stable",
"allowPermissionMgmt": true,
},
}
FMR Workbench integration
Released in June 13, 2024 Release .Stat Suite JS arc
FMR Workbench can be integrated with .Stat DLM to manage SDMX structures in synchronisation. See details functionalities here.
- in
dotstatsuite-config-data/<env>/configs/tenants.json
file under a DLMscope
{
"tenant":
(...)
"scope": {
"dlm": {
(...)
"linkFMR": "https://fwb.sdmxcloud.org"
}
}
}
In FMR Workbench configuration, provide the necessary information in the .json configuration file about .Stat DLM dataspaces and authentication.
Example:
"qa:reset" : {
"api" : {
"query" : https://nsi-qa-reset.siscc.org/rest,
"delete" : https://nsi-qa-reset.siscc.org/rest,
"persist" : https://nsi-qa-reset.siscc.org/rest/structure,
"version" : "1.5.0",
"timeout" : 600
},
"auth" : "OIDC_PKCE",
"oidc" : {
"config" : https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration,
"clientId" : "app"
},
"locale" : ["en", "fr"],
"format" : "mlv2.1"
},
More details about FMR Workbench in Getting started.