.Stat Suite documentation

.Stat DLM configuration

Table of Content

For the tenant (organisation and scope) and data space definitions please see here.


Authentication

The Data Lifecycle Manager requires an authentication configuration through an OpenID-Connect compliant identity provider.

For more information, see here.


Data space names

Define the name of an internal or external data space displayed to the end-user in the DLM UI, regardless the datasourceIds.

  • in dotstatsuite-config-data/<env>/configs/tenants.json
    "spaces": {
      "staging:SIS-CC-stable": {
        "label": "SIS-CC stable"
      },
    }

Data space colors

Define the font color and background color for each internal and external data space. The colors defined are applied in all options, features and representations of the data spaces (e.g. in the selectors and list of results, upload and transfer features, etc.)

  • in dotstatsuite-config-data/<env>/configs/tenants.json under a DLM scope
    where "color" refers to the font color of the name, and "backgroundColor" refers to the background colored area.
    "scopes": {
      "dlm": {
        "type": "dlm",
        "label": "dlm"
        },
        "spaces": [
          {
            "id": "staging:SIS-CC-stable",
            "color": "#0549ab",
            "backgroundColor": "#b7def6",
            "label": "staging:SIS-CC-stable"
          }
      }

data space color


Data space InsertNewItems parameter

Released in March 29, 2020 Release .Stat Suite .NET 3.3.0

The InsertNewItems parameter located in the app.config of the NSI web service allows to update final or non-final SDMX artefacts (of ItemScheme types, e.g. codelist) without changing the version of it.
When the parameter is set to:

    - name: INSERT_NEW_ITEM_SCHEME_VALUES
      value: "true"

Then it is possible for instance to add a new code in a codelist without modifying its Agency/ID/Version.


Use data space as internal space or external space

Changed in December 14, 2021 Release .Stat Suite JS 11.0.0

Internal spaces in the DLM are those that allow for data imports or copies from other spaces. For this, a related transfer service is required and the transferUrl parameter instructs the DLM to automatically show that data space as internal, automatically authenticate against that space with the common auth token and use the given transfer service instance for data imports. Otherwise, the data space is shown as external.

The transferUrl parameter is to be set in the dotstatsuite-config-data/<env>/configs/tenants.json file under a DLM scope for a given space to the rest root URL of the related transfer service.

    "scopes": {
      "dlm": {
        "type": "dlm",
        "label": "dlm",
        "oidc": {
          "authority": "https://keycloak.siscc.org/auth/realms/OECD",
          "client_id": "app"
        },
        "spaces": [
          {
            "id": "staging:SIS-CC-stable",
            "color": "#0549ab",
            "backgroundColor": "#b7def6",
            "label": "staging:SIS-CC-stable",
            "transferUrl": "https://transfer-demo.siscc.org/2",
            "dataExplorerUrl": "https://de-qa.siscc.org"
          },

Auto-authenticate to external source

Released in August 3, 2022 Release .Stat Suite JS quark

In case a .Stat CORE data space is external because its definition doesn’t define a transfer service to be used for imports, but it uses the same identity provider that is defined for the internal spaces, then the authenticateToRemoteURL parameter instructs the DLM to automatically authenticate against that source with the common auth token as well as to allow the transfer web service defined for an internal space (when called through one of the /import/sdmxFile or /validate/sdmxFile methods) to authenticate against that data source with that same auth token.

The authenticateToRemoteURL parameter is to be set to true in the dotstatsuite-config-data/<env>/configs/tenants.json file under a DLM scope for a given space.

    "scopes": {
      "dlm": {
        "type": "dlm",
        "label": "dlm"
        },
        "spaces": [
          {
            "id": "staging:SIS-CC-reset",
            "color": "#0549ab",
            "backgroundColor": "#e2f2fb",
            "label": "staging:SIS-CC-reset",
            "dataExplorerUrl": "https://de-qa.siscc.org",
            "authenticateToRemoteURL": true
          }
      }

Use native NSI WS authentication for external source

Introduced in December 14, 2021 Release .Stat Suite JS 11.0.0

In case an external data source is accessible through an SDMX web service based on Eurostat’s SDMX-RI “NSI” component (so called NSI web service), with the hasExternalAuth parameter the DLM can be instructed to authenticate against that web service using the native NSI authentication mechanism (implemented by Eurostat) based on HTTP basic access authentication (BA).

The user will have to enter the required credentials through a specific dialog box, as described here.

For more information on how to set the hasExternalAuth parameter, please see here


Define data preview per space

With the dataExplorerUrl parameter the DLM is instructed to allow previewing structures and data with the Data Explorer for an internal or external DLM data space.

The dataExplorerUrl parameter is to be set in the dotstatsuite-config-data/<env>/configs/tenants.json file under a DLM scope for a given space to the URL of the Data Explorer to be used.

    "scopes": {
      "dlm": {
        "type": "dlm",
        "label": "dlm"
        },
        "spaces": [
          {
            "id": "staging:SIS-CC-reset",
            "color": "#0549ab",
            "backgroundColor": "#e2f2fb",
            "label": "staging:SIS-CC-reset",
            "dataExplorerUrl": "https://de-qa.siscc.org"
          }
      }

List of SDMX artefact types

Define the types of SDMX structural artefacts selectable in the DLM ‘Filter by type’ selector (see functional specs). Only artefacts of those types can be displayed in the DLM user interface.

  • in dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/settings.json
  "sdmx": {
    "typeIds": [
      "agencyscheme",
      "categoryscheme",
      "categorisation",
      "codelist",
      "conceptscheme",
      "contentconstraint",
      "datastructure",
      "dataflow",
      "hierarchicalcodelist",
      "metadatastructure",
      "metadataflow",
      "structureset"
    ]
  }

Default data validations option

Introduced in December 5, 2022 Release .Stat Suite JS spin

Define, in the ‘Data upload’ and ‘Data transfer’ menus, what is the data validations type option selected by default:

  • “basic” for Favour speed with basic validations
  • “advanced” for Apply advanced validations
  • in dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/settings.json
  "app": {
    "defaultOptionDataValidation": "advanced"
  }

See more about the functional specifications of this feature here.


Upload size limit

Define the file size limitation when uploading data or strutures files applied by the DLM user interface as follow:
Provided in number of bites, the upload file size limit is usually set to 30MB by default (like in the example below).

  • in dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/settings.json
  "sdmx": {
    "uploadSizeLimit": 30000000
  }

Specific configurations are also necessary also for the underlying .Stat CORE services (transfer and NSI). The hosting web server(s) (e.g. IIS, Kestrel, ..) need(s) to be configured to accept http requests with bodies of the required size. All web servers have their own defaults, e.g. the .Net Core builtin web server, called Kestrel, has a default maximum request body size of 30,000,000 bytes, which is approximately 28.6 MB.

In order to change the MaxRequestBodySize parameter in the transfer service, it will depend on the installation mode of back-end services:

  • in case of a docker installation an environment variable of the container should be used to specify the max. size, e.g.:

Kestrel__Limits__MaxRequestBodySize: 52428800

  • if you have source code installation, then the config file you created should be moved into the config folder of transfer service. Please note that if you also use IIS, then IIS may also have limitations on the maximum file size.

More information on how to make this configuration in Kestrel (on the example of the NSI web service) can be found here. For other web server technologies, please consult their specific documentation.


Logbook submission time boundaries

Released in December 5, 2022 Release .Stat Suite JS spin

Define the boundaries of the submission time logbook filter. Rules are:

  • configuration applies to the organisation;
  • valid values for start and end param. are “now” or any ISO date string;
  • by default it is set to "start": "2019-01-01" and "end": "now";
  • if settings are not set, or invalid, then default boundaries are [30 days - now].
  • in dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/settings.json
 "logs": {
    "transfer": {
      "timeExtents": {
        "start": "2019-01-01",
        "end": "now"
      }
    }
  }

List of the standard roles for user permissions

Introduced in December 5, 2022 Release .Stat Suite JS spin

Define, per organisation, the Standard roles set for the ‘DLM –> Manage permissions –> Add/View/Edit’ form, based on any combination of the granular permissons as defined in the Authorisation-Management Service.
By default, the below set of combinations is pre-defined as standard roles.

References:

DLM config basic permissions

  • in dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/settings.json
  "sdmx": {
    "permissionGroups": [
      {
        "id": 3,
        "permissions": [1, 2]
      },
      {
        "id": 145,
        "permissions": [1, 16, 128]
      },
      {
        "id": 657,
        "permissions": [1, 16, 128, 512]
      },
      {
        "id": 3363,
        "permissions": [1, 2, 32, 256, 1024, 2048]
      },
      {
        "id": 2051,
        "permissions": [1, 2, 2048]
      },
      {
        "id": 4095,
        "permissions": [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]
      }
    ]
  }

The "permissionGroup" lists the standard roles using per role a unique "id", which represents the sum of the IDs of the underlying granular permissons as defined in the Authorisation-Management service, as well as the list of those IDs.

Each role ID also requires the definition of a localised label in an “override” i18n folder under dotstatsuite-config-data/<env>/configs/<organisation>/data-lifecycle-manager/ with each language’s translation file. For example, here are the English labels to add corresponding to the above set of roles: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-config-data/-/blob/develop/configs/default/data-lifecycle-manager/i18n/en.json.


Override the default Authorisation-Management service URL

Released in December 5, 2022 Release .Stat Suite JS spin

Define an Authorisation-Management service URL at the DLM scope level that will override the one defined as an env. variable during the deployment. This override is made especially to allow for a specific Authorisation-Management service URL per DLM scope when a DLM instance has several scopes.

  • in dotstatsuite-config-data/<env>/configs/tenants.json file under a DLM scope
  "dlm2": {
    "type": "dlm",
    "label": "dlm2",
    "oidc": {
      "authority": "https://...",
      "client_id": "app"
    },
    "authzServerUrl": "https://authz-env.org/1.1",

Allow managing permissions on a specific external space

Released in December 20, 2023 Release .Stat Suite JS yay

Allow, for a specific data space that is defined as external, to manage the users’ permissions through the DLM UI Manage Pemrissions feature. When the config. parameter "allowPermissionMgmt" is set to ’true’ (default: false) for an external data space, then the data space appears in the DLM “Manage Permissions” tab and users with succifient rights are allowed to manage user access accordingly.

  • in dotstatsuite-config-data/<env>/configs/tenants.json file under a DLM tenant > spaces
    "spaces": {
      "staging:SIS-CC-stable": {
        "label": "staging:SIS-CC-stable",
        "allowPermissionMgmt": true,
        },
    }

FMR Workbench integration

Released in June 13, 2024 Release .Stat Suite JS arc

FMR Workbench can be integrated with .Stat DLM to manage SDMX structures in synchronisation. See details functionalities here.

  • in dotstatsuite-config-data/<env>/configs/tenants.json file under a DLM scope
{
  "tenant":
    (...)
    "scope": {
      "dlm": {
        (...)
        "linkFMR": "https://fwb.sdmxcloud.org"
      }
    }
}

In FMR Workbench configuration, provide the necessary information in the .json configuration file about .Stat DLM dataspaces and authentication.
Example:

        "qa:reset" : {
            "api" : {
                "query" : https://nsi-qa-reset.siscc.org/rest,
                "delete" : https://nsi-qa-reset.siscc.org/rest,
                "persist" : https://nsi-qa-reset.siscc.org/rest/structure,
                "version" : "1.5.0",
                "timeout" : 600
            },
            "auth" : "OIDC_PKCE",
            "oidc" : {
                "config" : https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration,
                "clientId" : "app"
            },
            "locale" : ["en", "fr"],
            "format" : "mlv2.1"
        },

More details about FMR Workbench in Getting started.