.Stat Suite documentation

Changelog

Upgrade Disclaimers:

  • From .Stat Suite .NET v6.4.0 (structure db v6.14) to .Stat Suite .NET v7.1.0 (structure db v6.17) directly: link
  • From a .Stat Suite .NET version below 5.0.0 to .Stat Suite .NET v5.0.0 or higher: link

May 19, 2022

Release .Stat Suite .NET 8.1.2

This release includes a patch version of the core-transfer service.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

patch changes:

May 12, 2022

Release .Stat Suite JS “photon”

This release includes a new version of the data-explorer, data-viewer, and data-lifecycle-manager services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

Disclaimer: .Stat Suite adopts version names
From now on, each .Stat Suite release will be given a unique name, e.g. ‘photon’.
Semantic versioning is kept at the component/service level (see milestone for details).
See the details of our updated release process here

minor changes:

patch changes:

May 11, 2022

Release .Stat Suite .NET 8.1.1

This release includes a patch version of the core-transfer and dlm-excel-addin services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

patch changes:

April 28, 2022

Release .Stat Suite .NET 8.1.0

This release includes a minor version of the dlm-excel-addin, core-transfer and core-data-access services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

Performance evolutions in this release: Here below is a summary of comparison of the performance between this release and the release .Stat Suite .NET 8.0.0. All performance tests processes, definitions and types are documented here. Note that performance tests are now done directly in the quality assurance environment and may be slowed down by other activities, or could not yet be produced for all test types, therefore we provide here also the statistics made in a separate isolated environment (marked in brackets).

Smoke-test data imports 71.42% ✓ 5 ✗ 2 –> 100.00% ✓ 13 ✗ 0

data_import_time................: avg=12.68s --> 24.21s (Note: the increase is due to the extension of tests to more and larger examples)
✓ { datasetSize:extraSmall }....: avg=2.19s --> 2.37s
✓ { datasetSize:small }.........: avg=28.43s --> 25.48s
✓ { datasetSize:large }.........: ---------- avg=38.14s

Smoke-test data extractions for 100.00% ✓ 160 ✗ 0 –> 100.00% ✓ 344 ✗ 0 (100.00% ✓ 344 ✗ 0 –> 100.00% ✓ 344 ✗ 0)

http_req_duration...............: avg=451.33ms (304.54ms) --> 590.36ms (269.97ms) (Note: the increase is due to the extension of tests to more and larger examples)
✓ { datasetSize:extraSmall }....: avg=381.5ms (260.97ms) --> 502.6ms (219.58ms)
✓ { datasetSize:small }.........: avg=472.27ms (244.05ms) --> 467.37ms (209.68ms)

Load-test data extractions (96.97% ✓ 11398 ✗ 356 –> 92.81% ✓ 9303 ✗ 720)

http_req_duration...............: (avg=2.12s) --> (avg=2.67s)
✓ { datasetSize:extraSmall }....: (avg=2.40s) --> (avg=2.71s)
✓ { datasetSize:small }.........: (avg=1.93s) --> (avg=2.41s)

Stress-test data extractions (85.74% ✓ 10428 ✗ 1734 –> 97.91% ✓ 9464 ✗ 202)

http_req_duration................: (avg=2.78s) --> (avg=3.74s)
✓ { datasetSize:extraSmall }.....: (avg=2.67s) --> (avg=3.65s)
✓ { datasetSize:small }..........: (avg=2.47s) --> (avg=3.5s)
✓ { datasetSize:small_paginated }: (avg=1.63s) --> (avg=2.74s)

Spike-test data extractions (70.74% ✓ 3516 ✗ 1454 –> 68.14% ✓ 3480 ✗ 1627)

http_req_duration................: (avg=5.19s) --> (avg=4.79s)
✓ { datasetSize:extraSmall }.....: (avg=5.41s) --> (avg=5.54s)
✓ { datasetSize:small }..........: (avg=5.06s) --> (avg=4.52s)
✓ { datasetSize:small_paginated }: (avg=4.27s) --> (avg=3.82s)

Soak-test data extractions for (100.00% ✓ 42590 ✗ 0 –> 100.00% ✓ 46873 ✗ 0)

http_req_duration..........: avg=1.34s --> avg=1.13s

minor changes:

patch changes:

April 25, 2022

Release .Stat Suite JS 14.0.2

This release includes a patch version of the data-explorer and data-viewer services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

patch changes:

April 14, 2022

Release .Stat Suite JS 14.0.1

This release includes a patch version of the data-explorer and data-viewer services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

patch changes:

April 11, 2022

Release .Stat Suite JS 14.0.0

This release includes a new major version of the data-explorer, data-viewer, data-lifecycle-manager, sdmx-faceted-search, and proxy services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

major changes:

minor changes:

patch changes:

March 24, 2022

Release .Stat Suite .NET 8.0.1

This release includes a patch version of the core-transfer and core-data-access services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

patch changes:

March 23, 2022

Release .Stat Suite .NET 8.0.0

This release includes a new major version of the sdmxri-nsi-ws, core-transfer, core-auth-management, and core-data-access services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

Disclaimer:
Starting with NSIWS v8.9.1 (MSDB v6.19), some of the dataset properties related to a dataflow are stored in a new table called DATASET_PROPERTY. This new dataset property table is populated with all the necessary data by the .Stat Suite transfer service methods:

  • at data/referential metadata import for the targeted dataflow
  • at execution of init/dataflow method for a given dataflow
  • at execution of init/allMappingsets method for all dataflows

When upgrading to MSDB v6.19, the content of the formely used DATASET.OREDER_BY_CLAUSE column is migrated into the new DATASET_PROPERTY table for each dataset present in the structure database. In order to enable usage of improvements related to firtNObservations and lastNObservations queries on existing dataflows in .Stat Suite, a number of other dataset properties must be added to the existing dataflows. This can be achieved by executing the .Stat Suite transfer service method init/allMappingsets (Documentation).
This method must be called manually and as the last very last step of the deployment process of the new version, after

  • all components are deployed/updated, and
  • the DBUP tool has been run to update the databases, and
  • the maapi.net tool has been run to upgrade the structure (mapping store) databases.

/!\ Make sure that the value provided for the configuration setting DataImportTimeOutInMinutes has a value large enough for the init/allMappingsets function to complete. Once this has been ran, the value for DataImportTimeOutInMinutes can be restated to the former value.

Performance evolutions in this release: Here below is a summary of comparison of the performance between this release and the release .Stat Suite .NET 6.4.0. All performance tests processes, definitions and types are documented here. Note that performance tests are now done directly in the quality assurance environment and may be slowed down by other activities, or could not yet be produced for all test types, therefore we provide here also the statistics made in a separate isolated environment (marked in brackets).

Smoke-test data imports 71.42% ✓ 5 ✗ 2 –> 71.42% ✓ 5 ✗ 2

data_import_time................: avg=13.52s --> 12.68s
✓ { datasetSize:extraSmall }....: avg=2.72s --> 2.19s
✓ { datasetSize:small }.........: avg=29.71s --> 28.43s

Smoke-test data extractions for 100.00% ✓ 159 ✗ 0 –> 100.00% ✓ 160 ✗ 0 (99.41% ✓ 342 ✗ 2 –> 100.00% ✓ 344 ✗ 0)

http_req_duration...............: 406.62ms (avg=311.59ms) --> 451.33ms (avg=304.54ms)
✓ { datasetSize:extraSmall }....: 358.35ms (avg=222.77ms) --> 381.5ms (avg=260.97ms)
✓ { datasetSize:small }.........: 421.96ms (avg=238.10ms) --> 472.27ms (avg=244.05ms)

Load-test data extractions (93.58% ✓ 9971 ✗ 684 –> 96.97% ✓ 11398 ✗ 356)

http_req_duration...............: (avg=2.46s) --> (avg=2.12s)
✓ { datasetSize:extraSmall }....: (avg=2.21s) --> (avg=2.40s)
✓ { datasetSize:small }.........: (avg=2.19s) --> (avg=1.93s)

Stress-test data extractions (95.67% ✓ 8516 ✗ 385 –> 85.74% ✓ 10428 ✗ 1734)

http_req_duration................: (avg=4.15s) --> (avg=2.78s)
✓ { datasetSize:extraSmall }.....: (avg=3.51s) --> (avg=2.67s)
✓ { datasetSize:small }..........: (avg=3.91s) --> (avg=2.47s)
✓ { datasetSize:small_paginated }: (avg=3.47s) --> (avg=1.63s)

Spike-test data extractions (72.12% ✓ 2921 ✗ 1129 –> 70.74% ✓ 3516 ✗ 1454)

http_req_duration................: (avg=6.39s) --> (avg=5.19s)
✓ { datasetSize:extraSmall }.....: (avg=6.59s) --> (avg=5.41s)
✓ { datasetSize:small }..........: (avg=6.06s) --> (avg=5.06s)
✓ { datasetSize:small_paginated }: (avg=6.13s) --> (avg=4.27s)

Soak-test data extractions for 99.06% ✓ 17503 ✗ 165 –> 98.37% ✓ 14434 ✗ 238

http_req_duration..........: avg=4.65s --> avg=5.81s

major changes:

  • dotstatsuite-core-transfer#310 HTTP 502 Bad ‘Bad Gateway’ error in cleanup/orphans. In cludes a new transfer API version 2 with the following specificity with v1.2 for the /{version}/cleanup/orphans method (documentation) :
    • Version 1.2: synchronous request
    • Version 2: asynchronous request

minor changes:

patch changes:

March 8, 2022

Release .Stat Suite JS 13.0.1

This release includes a new patch version of the data-explorer and data-viewer services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

patch changes:

March 4, 2022

Release .Stat Suite JS 13.0.0

This release includes a new major version of the data-explorer, data-viewer, data-lifecycle-manager, and config (no Docker image) services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.2.

major changes:

  • dotstatsuite-config#36 (Support) JS v12 upgrade returns a “Maximum call stack size exceeded” error. The default tenant is now mandatory and must be defined in any case in your configs/tenants.json file:
"myTenant": {
    "id": "myTenantId",
    "label": "myTenantLabel",
    "default": true
}

minor changes:

patch changes:

February 21, 2022

Release .Stat Suite JS 12.1.0

This release includes a new minor version of the data-explorer, sdmx-faceted-search, data-viewer, config, and data-lifecycle-manager services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.9.1.

minor changes:

patch changes:

January 13, 2022

Release .Stat Suite JS 12.0.0

This major release includes a new version of the data-explorer, data-viewer, data-lifecycle-manager, and share services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.8.0.

major changes:

minor changes:

patch changes:

December 14, 2021

Release .Stat Suite JS 11.0.0

This major release includes a new version of the data-explorer, data-viewer, data-lifecycle-manager, sdmx-faceted-search, share, config, proxy, and keycloak services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.8.0.

major changes:

  • dotstatsuite-data-explorer#606, dotstatsuite-data-lifecycle-manager#230 Generic openID-Connect compliance for .Stat DE and .Stat DLM:
    • in tenants.json, keycloak replaced by oidc entry (see example in this diff.), and the env. variable AUTH_SERVER_URL is removed and replaced by the new oidc entry.
    • Documentation
  • ‘MailGun’ API key is removed from the share service (see the documentation about setting up your own SMTP share configuration)
  • dotstatsuite-config#34 Allow reading config files from either a GCP bucket, a MinIO bucket or a local filesystem (see more details in Readme and check the env. variables changes in our DevOps).
  • Changes related to the settings.json configuration file: see the technical changelog and related diff. applied in our DevOps.
  • Changes related to the tenants.json file: see the technical changelog and related diff. applied in our DevOps.
  • share service deployment related: the SITE_URL must be a link that redirects to the data-explorer, as SITE_URL: "${PROTOCOL}://${HOST}:${DE_PORT}/share"

minor changes:

patch changes:

Performance improvements of the DE data table: here below is a summary of comparison of the performance between the previous release and this one, introducing significant evolutions of the time required to build the data table in a web browser page :

Table content & layout JS 10.0.0 JS 11.0.0 Diff.
250 observations 468 ms ~200 ms - 260 ms
2,500 observations 4207 ms ~100 ms - 4,1 s
240 observations - No column 751 ms ~400 ms - 350 ms
2,500 observations - No column 5499 ms ~2500 ms - 3 s
2,500 observations - 3 Columns 4860 ms ~1500 ms - 3,3 s
2,500 observations - 1 Row section - 1 Row 4747 ms ~2300 ms - 2,4 s
2,500 observations - 3 Row sections - 1 Row 15498 ms ~4000 ms - 11,5 s

Note: the new performance results are accurate to only tenth of a second because we have corrected the initial performance issues related to scripting, rendering and system. Therefore the calculation only provides an approximate but yet true result for building the data table, and does not impact anymore scripting, rendering or system.

December 3, 2021

Release .Stat Suite .NET 7.2.0

This release includes a new version of the sdmxri-nsi-ws, core-transfer, core-auth-management, and core-data-access services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.8.0.

minor changes:

patch changes:

.NET upgrade disclaimer

An error was reported when upgrading a mapping store (structure) database from v6.14 to v6.17. .Stat Suite affected versions are when upgrading from .Stat Suite .NET v6.4.0 (structure db v6.14) to .Stat Suite .NET v7.1.0 (structure db v6.17) directly. Most likely, the issue can also be present when upgrading from any .Stat Suite .NET release prior to .Stat Suite .NET v6.4.0 (structure db prior v6.14) to .Stat Suite .NET v7.1.0 directly.
The problem was solved with dotstatsuite-core-sdmxri-nsi-ws#188 in the Release .Stat Suite .NET 7.2.0.
As a temporary workaround, structure database v6.14 should be upgraded to v6.15 first and then from v6.15 to v6.17. This can be achieved by one of the following methods:

  • upgrading all .Net components incrementally
  • from .NET v6.4.0 to .NET v7.0.1 and then (in docker installations at least the NSI WS containers should be started in order to perform db upgrade to v6.15)
  • upgrade .NET v7.0.1 to .NET v7.1.0
  • upgrading incrementally only the problematic component to achieve incremental upgrade of structure databases:
  • Docker installation:
    • upgrade image: siscc/sdmxri-nsi-maapi:8.2.0-2410af09 to image: siscc/sdmxri-nsi-maapi:8.5.0-4fd592b5 (v7.0.1) and start the related services to let them upgrade the structure db-s to v6.15;
    • upgrade image: siscc/sdmxri-nsi-maapi:8.5.0-4fd592b5 (v7.0.1) to image: siscc/sdmxri-nsi-maapi:8.7.1-36e7592d (v7.1.0)
    • upgrade the other .NET images from .NET v6.4.0 to .NET v7.1.0
  • Source code installation:
    • download source code and build maapi.net tool v8.5.0, then use it to upgrade your structure database(s) to v6.15
    • use maapi.tool v8.7.1 to upgrade your structure database(s) from v6.15 to v6.17
    • upgrade the other .NET components from .NET v6.4.0 to .NET v7.1.0 directly

October 11, 2021

Release .Stat Suite .NET 7.1.0

This release includes a new version of the sdmxri-nsi-ws, core-transfer, core-auth-management, and excel-addin services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.7.1.

Disclaimers:

  1. Known issue: Time periods below hours are currently represented with hourly period time ranges. We recommend not to upgrade to this .Stat Suite release if you deal with minutely or secondly data, and wait until we release the patch change dotstatsuite-core-data-access#79.
  2. The SdmxSource library of Eurostat now accepts time fragments with fraction of seconds as valid date/time values (e.g. 1222-02-22T22:20:02.202). .Stat Suite does not support time periods below seconds, only down to seconds. (Documentation)

minor changes:

patch changes:

October 5, 2021

Release .Stat Suite JS 10.0.0

This major release includes a new version of the data-explorer, data-viewer, and data-lifecycle-manager services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.5.0.

Performance evolutions of the DE data table in this release: here below is a summary of comparison of the performance between the previous release and this one which introduces a new HTML table in the DE visualisations (dotstatsuite-data-explorer#523). Note that further performance improvements of the table are expected once we release dotstatsuite-data-explorer#591 :

DE table (version, content & layout) Total Scripting Rendering System
JS 10.0.0 Table 250 observations 468 ms 347 ms 33 ms 87 ms
JS 9.0.2 Table 250 observations 449 ms 343 ms 28 ms 78 ms
JS 10.0.0 Table 2500 observations 4207 ms 3817 ms 102 ms 288 ms
JS 9.0.2 Table 2500 observations 5623 ms 4245 ms 1016 ms 361 ms
JS 10.0.0 Table 240 observations - Rows 751 ms 497 ms 84 ms 170 ms
JS 9.0.2 Table 240 observations - Rows 713 ms 502 ms 81 ms 129ms
JS 10.0.0 Table 2500 observations - Rows 5499 ms 4869 ms 248 ms 382 ms
JS 9.0.2 Table 2500 observations - Rows 6661 ms 5398 ms 817 ms 446 ms
JS 10.0.0 Table 2500 obs - Columns * 3 4860 ms 4358 ms 139 ms 363 ms
JS 9.0.2 Table 2500 obs - Columns * 3 5754 ms 4621 ms 798 ms 335 ms
JS 10.0.0 Table 2500 obs - Sections (1 dimension) 4747 ms 4135 ms 223 ms 389 ms
JS 9.0.2 Table 2500 obs - Sections (1 dimension) 14137 ms 7422 ms 2067 ms 337 ms
JS 10.0.0 Table 2500 obs - Sections (3 dimensions) 15498 ms 14796 ms 198 ms 504 ms
JS 9.0.2 Table 2500 obs - Sections (3 dimensions) 16808 ms 15158 ms 963 ms 687 ms

major changes:

  • New translation keys were added to the source code (DLM UI related). See the i18n changelog. Note that these new keys do not all have a default translation provided in all the supported languages, hence you might have missing translations for some distinct (DLM) UI labels, unless you provide translations in your overrides, or you submit translations to the code source.
  • dotstatsuite-data-viewer#22 Use i18n in the title of the data-viewer. It affects the settings.json file by removing the following property (moved to locales):
"app": {
  "title": "SIS-CC .Stat Data viewer",
}

significant and minor changes:

patch changes:

September 16, 2021

Patch Release .Stat Suite .NET 7.0.1

This release includes a new patch version of the core-transfer and core-data-access services.

patch changes:

September 03, 2021

Release .Stat Suite .NET 7.0.0

This major release includes a new version of the sdmxri-nsi-ws, core-transfer, core-data-access, and core-auth-management services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.5.0.

Disclaimers:

  1. With the release of dotstatsuite-core-transfer#231 and the introduction of pagination into the NSIWS for data requests, and in order for the /rest/data/... queries to fully use the pagination mechanism (thus improving performance and properly using Range:start-end), the mapping sets of the dataflows have to be reinitialized. To do so, there are 3 options:
    Option 1: Initialize all mappingsets - Right after the release upgrade, run the transfer-service function /init/allMappingsets. This should be ran when no other users are using the transfer-service The system administrator should schedule for some down time period. The function might be long depending on the amount of dataflows.
    Option 2: Initialize the mappingset of specific dataflow - The initialization can be done per dataflow using the transfer-service function /{version}/init/dataflow for a given dataflow. This option does not require a down time period.
    Option 3: Import data to initialize the mappingset of a specific dataflow - At the end of a data import, the mappingset will be initialized as part of the import process. This option does not require a down time period.
    If none of the above is applied, the /rest/data/... queries will continue to work, but the performance will not be improved, nor the full pagination feature will work (only the first page will be retrieved).
  2. this latest version of the core-transfer service does not save the actual data db connection info into structure database by default, but instead it puts ‘NOT SAVED’ entries in the related DB_CONNECTION table.
    As a result, the nsiws configuration must be extended with the following two elements:
    disseminationDbConnection.dbType
    disseminationDbConnection.connectionString
    Please see the related documentation for further information.

General advice on Dataflows deletion: For a complete deletion of the MappingSet related to a dataflow, it is necessary, when deleting a dataflow, to either use the delete option from the DLM (the MappingSet cleanup is automatic), or manually delete the MappingSet using the Transfer Swagger UI (https://transfer-env.tenant.org/swagger/) before deleting the dataflow. This feature will be improved once releasing dotstatsuite-core-sdmxri-nsi-ws#164.

Known issues: Be aware that this release has known issues described below. If you are concerned by one of these use cases, we suggest to upgrade using the next .NET Patch version.

  • dotstatsuite-core-data-access#76: the migration/upgrade script will throw an error for DSDs with non-coded attributes at dimensions/group level.
  • dotstatsuite-core-data-access#77: the migration/upgrade script will throw an error in the logs for DSDs with no Time dimension. There is no significant impact and should be ignored.
  • dotstatsuite-core-transfer#255: the migration/upgrade script can return errors on Dataflows such as : Incomplete mapping set. Please check if all dimensions, measure(s) and mandatory attributes are mapped: Dimension 'FREQ' is not mapped. Dimension 'REF_AREA' is not mapped ...
    This can happen when several dataflows related to the same DSD were previously deleted at once using the DLM, and it will result in failing to request the data view(s). If so, running a) the /cleanup/mappingsets method, then b) the /init/dataflow method should fix the issue.

Regression:

  • dotstatsuite-core-sdmxri-nsi-ws#167: regression in the LastNObservations feature. Querying for LastNObservations will return the FirstNObservations. A fix will be submitted with high-priority to the SDMX source and integrated to the .Stat Suite with a future version of the nsiws.

major (backward-incompatible) changes:

significant and minor changes:

patch changes:

September 2, 2021

Patch Release .Stat Suite JS 9.0.2

This release includes a new patch version of the data-explorer and data-lifecycle-manager services.

patch changes:

August 17, 2021

Patch Release .Stat Suite JS 9.0.1

This release includes a new patch version of the data-explorer service.

patch changes:

July 8, 2021

Release .Stat Suite JS 9.0.0

This major release includes a new version of the data-explorer, sdmx-faceted-search, data-viewer, config, and data-lifecycle-manager services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.3.0.

TECHNICAL DISCLAIMERS:
Config folder contents significantly change: tenants.json contains more ; settings.json has no more relation to dataspace or datasource ; datasources.json is removed.

  • New tenant model:
    • tenants.json is bigger with centralized spaces/datasources information
    • separation between spaces and datasources concepts
    • new concept of scope used to bind spaces/datasources to apps
    • references to datasources in settings.json moved in tenants.json
    • routes (when proxy is used) can use tenant slug: <tenantId> or <tenantId>:<scopeId>
  • Major change in the search service (new tenant model adaptation):
    • reminder from previous release, a collection is required for each tenant (Documentation)
    • clean and re-index all data for all tenants to avoid side-effects
  • ROBOTS_POLICY env var in DE and DV to configure search engine indexation (default enabled), don’t forget to override robot.txt file accordingly to the env var value
  • AUTHZ_SERVER_URL env var is now used to bind the DLM to AUTHZ (previously in settings.json)
  • TRANSFER_SERVER_URL env var dropped (was used in DLM) and replaced by transferUrl in tenants.json (at scope level): DLM is not bound to a transfer, it is the spaces used within its scope that are bound to a transfer (which is more flexible because different transfers can be used without deploying several DLM)
  • To override space definition in search service:
    • temporary fix waiting for a better integration in the new tenant model
    • no more datasources.json file to mount at sfs level
    • define searchUrl in spaces definition, sfs will use it if defined in tenants.json
    • define headers in datasources definition in tenants.json

For your migration/upgrade process, think about:

  1. datasources are inside the spaces
  2. Define your datasources in tenants.json
  3. From your settings.json file, retrieve your datasources IDs and add it in scope(s) inside your tenants.json, then once this is done, remove the obsolete datasources IDs from your settings.json file.
  4. Then delete your datasources.json file (becoming useless).
  5. When using proxy, change of root: the tenant identification is made of a tenant and a scope for applications supporting scopes: DLM, DE, but not DV.

major (backward-incompatible) changes:

  • dotstatsuite-config-data#4 New tenant model adaptation. See above disclaimer and Documentation. Also available a video record demoing and explaining the adaptation of this new tenant model.
  • dotstatsuite-sdmx-faceted-search#28 Allow individually adding new dataflows in the search index. The previous sfs query to update an individual dataflow PATCH /admin/dataflow is replaced by the new ‘upsert’ POST /admin/dataflow query (Documentation).
  • dotstatsuite-config#29 (Refactoring) config data i18n follow up. Final migration of app configs from .stat-suite/dotstatsuite-config to .stat-suite/dotstatsuite-config-data

significant and minor changes:

patch changes:

June 16, 2021

Release .Stat Suite .NET 6.4.0

This release includes a new version of the sdmxri-nsi-ws service.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.2.0.

Performance evolutions in this release: Here below is a summary of comparison of the performance of release .Stat Suite .NET 4.4.0 (before using the NSI default plugin) with release .Stat Suite .NET 6.0.0 (after using the NSI default plugin) and with this new release. All performance tests processes, definitions and types are documented here).
Smoke-test data imports

data_import_time................: avg=16.85s --> avg=19.42s --> avg=10.14s
✓ { datasetSize:extraSmall }....: avg=2.36s --> avg=2.6s --> avg=6.13s (med=1.77s)
✓ { datasetSize:small }.........: avg=31.34s --> avg=36.24 --> avg=18.17s

Smoke-test data extractions

http_req_duration...............: avg=227.65ms --> avg=182.51ms --> avg=158.72ms

Load-test data extractions

http_req_duration...............: avg=1.33s --> avg=8.05s --> avg=1.07s 
✓ { datasetSize:extraSmall }....: avg=1.6s --> avg=7.07s --> avg=1.08s 
✓ { datasetSize:small }.........: avg=1.39s --> avg=8.85s --> avg=1.16s

Stress-test data extractions

http_req_duration................: avg=1.89s --> avg=2.43s --> avg=1.79s
✓ { datasetSize:extraSmall }.....: avg=2.27s --> avg=2.3s --> avg=1.9s
✓ { datasetSize:small }..........: avg=1.99s --> avg=2.63s --> avg=1.92s

Spike-test data extractions

http_req_duration................: avg=3.8s --> avg=3.68s --> avg=3.76s
✗ { datasetSize:extraSmall }.....: avg=4.25s --> avg=4.11s --> avg=4.22s  
✗ { datasetSize:small }..........: avg=3.88s --> avg=3.76s --> avg=4.01s

Soak-test data extractions

http_req_duration..........: avg=670.42ms --> avg=1.32s --> avg=1.34s

minor changes:

patch changes:

June 1, 2021

Release .Stat Suite .NET 6.3.0

This release includes a new version of the sdmxri-nsi-ws service.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.3.

minor changes:

patch changes:

Release .Stat Suite JS 8.0.1 - Keycloak

This release only includes a new custom image of the keycloak Docker image.
.Stat Suite compatibility to JS 8.0.0 and .NET 6.3.0
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.3.

significant change:

  • Connect Keycloak to Active Directory ADFS. This is a release of the new and ‘custom’ Keycloak Docker image including the new plugin that allows to connect an Active Directory ADFS as a third-party provider to Keycloak:
  • Docker hub image
  • Newly made public Keycloak project
  • ADFS-to-keycloak plugin
  • Documentation on how to configure ADFS as a SAML provider to Keycloak

May 19, 2021

Release .Stat Suite JS 8.0.0

This major release includes a new version of the data-explorer, sdmx-faceted-search, data-viewer, share, config, and data-lifecycle-manager services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

major (backward-incompatible) changes:

  • dotstatsuite-sdmx-faceted-search#40 Solr upgrade to version 8.7. (Updated API documentation)
    Disclaimer:
    • sfs is now multi-tenant, which means that each API call should include a tenant. Providing a tenant can be done in 2 ways:
      • as a query param: &tenant=test
      • as a header: x-tenant=test
    • Existing data can be dropped, it is reindexed (new solr major version, no need to struggle with a migration). No need to create a SOLR Core anymore!
    • Create a Collection in solr named defaulttenant: http://0.0.0.0:8983/solr/admin/collections?action=CREATE&name=defaulttenant&numShards=1&collection.configName=_default
    • Add DEFAULT_TENANT env. variable with the value defaulttenant to sfs
    • Note that sfs API is unchanged when using DEFAULT_TENANT, all calls without a tenant will use the value of DEFAULT_TENANT as a tenant
    • data-explorer will request sfs without a tenant until the upcoming new sfs model is released (see this list of iteration D issues)
    • On-premise installation: http call to solr api locally (since solr should be exposed)
    • Docker installation: http call to solr locally or from sfs container (see latest docker-compose)
  • keycloak#8 & keycloak#11 (DevOps) Keycloak upgrade to version 12.0.4.
    Disclaimer:
    • Migration process to this new major version is quite sensitive even though not backward-incompatible
    • Backup your data before processing with the upgrade!
    • See this tutorial page about Keycloak configuration, and also Kubernetes strategy and historical migrations documented here (as source of inspiration from our DevOps deployment process)
  • dotstatsuite-share#9 (Refactoring) Remove hard-coded dependency on third-party mail service MailGun.
    Disclaimer:
    • This allows the full SMTP setup of the share service: see updated documentation about using SMTP connection.
    • Add the new MAIL_FROM env. variable with the contextual value '"Share" share@your-domain.org' (was previsouly hard-coded)
    • Mailgun API key is kept as a param for an extra period of time (until September 30, 2021). After this date, the Mailgun API key will be definitely removed from the source code.
  • dotstatsuite-data-explorer#525 (Refactoring) Align i18n versions of data-viewer with data-explorer.
    Disclaimer:
    • data-viewer i18n is now using the default translations similarly to the data-explorer, hence no need to create overrides specific to data-viewer.

Additional disclaimer: This release implies to temporarily retrieve any new and default i18n key and translation values in a new public project and from its master branch https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-config-data/-/tree/master/i18n. The fluid process of releasing i18n default translations as part of the open-source .Stat Suite source code will be complete with the next iteration C delivery.

significant and minor changes:

patch changes:

May 11, 2021

Release .Stat Suite .NET 6.2.0

This release includes a new minor version of the core-data-access service.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

minor changes:

April 27, 2021

Release .Stat Suite .NET 6.1.0

This release includes a new version of the core-transfer, core-data-access, core-auth-management, and excel-addin services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

minor changes:

patch changes:

April 8, 2021

Patch Release .Stat Suite JS 7.2.1

This release includes a new patch version of the data-explorer and data-viewer services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

patch changes:

April 1, 2021

Release .Stat Suite JS 7.2.0

This release includes a new version of the data-explorer, data-lifecycle-manager, data-viewer, share, config, and proxy services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

Disclaimer for .Stat Suite Docker setup: as a result of dotstatsuite-config#27, a new Docker image needs to be used for Docker setup: siscc/dotstatsuite-config-prod:master -> siscc/dotstatsuite-config:master. Note that the previous siscc/dotstatsuite-config-prod:master will still work and will be removed at some point in the future.
In addition, the i18n translations were implicitly loaded, now it is done explicitly: folder i18n to mount config/data/prod/i18n -> opt/i18n.

minor changes:

patch changes:

March 15, 2021

Patch Release .Stat Suite JS 7.1.1

This release includes a new patch version of the data-explorer service.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

patch change:

  • dotstatsuite-data-explorer#500 Search results additional downloads for external resources does not work. All use cases for internally/externally stored dataflows, and externally stored materials to download must be handled.

March 10, 2021

Release .Stat Suite JS 7.1.0

This release includes a new version of the data-explorer, data-viewer, sdmx-faceted-search, and data-lifecycle-manager services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

significant and minor changes:

patch changes:

March 5, 2021

Release .Stat Suite .NET 6.0.0

This major release includes a new version of the core-transfer, core-sdmxri-nsi-ws, core-auth-management, and core-data-access services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

Disclaimer on the performance evolutions in this release. Here below is a summary of comparison of the performance before and after using the SDMX-RI NSI web service default data plugin in .Stat Suite (all performance tests processes, definitions and types are documented here). Although a series of improvements can already be noticed with this release, it does not yet meet the expected performance improvements for high-load situations (many parallel data queries). Additionally required improvements (related mainly to the default nsi-ws-plugin) are being addressed with dotstatsuite-core-sdmxri-nsi-ws#111.

  • Comparison of the performance before and after using the SDMX-RI NSI web service default data plugin
    Smoke-test data imports (time increase by 15%) 
    data_import_time................: avg=16.85s --> avg=19.42s  
✓ { datasetSize:extraSmall }....: avg=2.36s --> avg=2.6s  
✓ { datasetSize:small }.........: avg=31.34s --> avg=36.24
    Smoke-test data extractions (time decrease by 20%) 
    http_req_duration...............: avg=227.65ms --> avg=182.51ms
    Load-test data extractions (time increase by 605%) 
    http_req_duration...............: avg=1.33s --> avg=8.05s  
✓ { datasetSize:extraSmall }....: avg=1.6s --> avg=7.07s  
✓ { datasetSize:small }.........: avg=1.39s --> avg=8.85s
    Stress-test data extractions (time increase by 29%) 
    http_req_duration................: avg=1.89s --> avg=2.43s  
✓ { datasetSize:extraSmall }.....: avg=2.27s --> avg=2.3s  
✓ { datasetSize:small }..........: avg=1.99s --> avg=2.63s
    Spike-test data extractions (time decrease by 3%) 
    http_req_duration................: avg=3.8s --> avg=3.68s  
✗ { datasetSize:extraSmall }.....: avg=4.25s --> avg=4.11s  
✗ { datasetSize:small }..........: avg=3.88s --> avg=3.76s
    Soak-test data extractions (time increase by 97%) 
    http_req_duration..........: avg=670.42ms --> avg=1.32s

major (backward-incompatible) changes:

  • dotstatsuite-core-sdmxri-nsi-plugin#37 Always protect non-public data. Whether openid-connect authentication is turned ON or OFF in the NSI web-service configuration, anonymous users ALWAYS only get data that is especially made public (through appropriate permissions).
    This represents a major change from previous releases, where with NSI openid-connect authentication was turned OFF, all users (necessarily unauthenticated) could get all data. Now, Docker image with authorization is enabled (turned ON) by default.
  • Known limitation: the current DLM feature for viewing artefact content (in xml format) in a new web-browser tab by clicking on the (hyperlinked) name of an artefact does not work anymore with the new nsi-ws authentication in place, unless a generic CanReadStructures permission is granted to all users incl. anonymous users. (Updated documentation)
  • dotstatsuite-core-transfer#124 Allow for non-numeric and coded measures (observation values) in .Stat CORE. Now, according to the SDMX standard, String is the default SDMX data TextType format when it is not specified in the data structure definition (previsouly, Float was the default TextType format). (Documentation)

minor changes:

patch changes:

General upgrade disclaimer

If you upgrade a .Stat Suite installation from a .NET version below 5.0.0, to .NET v5.0.0 or higher, you must follow the migration procedure explained as follows. All MappingSets must be generated specifically in the context of the .Stat Suite. This is to be done through the following methods:

  • Before you generate the MappingSets (see next bullet), if during the migration/upgrade to version 5.0.0 or higher with the DBUP tool, some DSDs/Dataflows migrations fail (check the logs using the transfer service /status/requests method), then you should migrate these dataflows manually using the Transfer service method /init/dataflow. Note that this should happen only extremely rarely, and would be caused by previous inconsistencies in the DB state. In case the manual dataflow migration is still unsuccessful then it is recommended to delete the underlying DSD, cleanup the related DB objects (using the Transfer service method /cleanup/dsd), recreate the data structures and reload the related data.
  • Generate the MappingSets for all already existing dataflows when the .Stat Suite .NET version is migrated to 5.0.0 or higher, using the .Stat Suite Transfer service method /init/allMappingsets. This method must be called manually as the very last step of the deployment of the new version (after all components are deployed/updated, and after the DBUP tool has run to update the databases). (Documentation)
  • Generate the MappingSet for any newly added dataflow using the .Stat Suite Transfer service method init/dataflow. This can be done using the Transfer service Swagger UI. (Documentation)
  • Generate the MappingSet for any newly added dataflow by uploading any data (in DLM or with the .Stat Suite Transfer service). In other words, the MappingSet of a newly added dataflow will be automatically generated once you upload data for this dataflow.

January 25, 2021

Release .Stat Suite .NET 5.0.0

This major release includes a new version of the core-transfer, core-sdmxri-nsi-ws, core-auth-management, and core-data-access services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

WARNING: While this first .Stat Suite Core (.NET) version using the SDMX-RI NSI web service default data plugin approach (according to the DB model review step 1) includes the related basic features such as PIT features, authorisations and new REST features, it does not contain yet the necessary performance enhancements foreseen in step 2. This does not represent an issue for data structure definitions that have only a few dimensions and attributes, but those with more dimensions and attributes are now too slow for production mode. These performance enhancements of step 2 are expected to be released very soon. We therefore recommend to not use this release in production mode but to wait for the next .Stat Suite Core (.NET) release, especially if you manage dataflows with many dimensions and attributes.

DISCLAIMER: Starting with this release, the SDMX-RI NSI web service within the .Stat Suite will use the default NSI data plugin instead of the previous .Stat Suite-specific data plugin. This requires the usage of a SDMX-RI MappingSet object (stored in the MappingStore structure database) for each of the dataflows. Except for MappingSets that are manually entered by the user through the SDMX-RI MappingAssistant, all MappingSets must be generated specifically in the context of the .Stat Suite. This is to be done through the following methods:

  • Before you generate the MappingSets (see next bullet), if during the migration/upgrade to version 5.0.0 with the DBUP tool some DSDs/Dataflows migrations fail (check the logs using the transfer service /status/requests method), then you should migrate these dataflows manually using the Transfer service method /init/dataflow. Note that his should happen only extremely rarely, and would be caused by previous inconsistencies in the DB state. In case the manual dataflow migration is still unsuccessful then it is recommended to delete the underlying DSD, cleanup the related DB objects (using the Transfer service method /cleanup/dsd), recreate the data structures and reload the related data.
  • Generate the MappingSets for all already existing dataflows when the .Stat Suite .NET version is migrated to 5.0.0 using the .Stat Suite Transfer service method /init/allMappingsets. This method must be called manually as the very last step of the deployment of the new version (after all components are deployed/updated, and after the DBUP tool has run to update the databases). (Documentation)
  • Generate the MappingSet for any newly added dataflow using the .Stat Suite Transfer service method init/dataflow. This can be done using the Transfer service Swagger UI. (Documentation)
  • Generate the MappingSet for any newly added dataflow by uploading any data (in DLM or with the .Stat Suite Transfer service). In other words, the MappingSet of a newly added dataflow will be automatically generated once you upload data for this dataflow.

Important NOTE: From this release on, any new/additional attribution of an admin permission (AdminRole) to the authorization service must use the new permission id:4095 (see major change below).

major (backward-incompatibility) changes:

  • dotstatsuite-core-sdmxri-nsi-plugin#48 Remove the custom .Stat Suite nsi-plugin (Replace NSI-Plugin).
  • dotstatsuite-core-auth-management#20 AdminRole permission id is changed from ‘2047’ to ‘4095’ in auth DB.
  • The nsiws.net v8.1.2 included a bug correction SDMXRI-1443 Support of observation values with string data type at SDMX-JSON data writers with a major impact in .Stat Suite. Whenever observation values are of String type, then the SDMX-JSON messages generated in SDMX-RI NSI extractions use the (correct) String type. It also means that string data types are now correctly exposed as string types in the SDMX-RI NSI extractions, and the DE also shows strings as strings without altering concepts such as thousand or decimal separators.

minor changes:

patches:

January 21, 2021

Release .Stat Suite JS 7.0.0

This major release includes a new version of the data-lifecycle-manager, data-explorer, data-viewer, sdmx-faceted-search, and share services.
Tip: we also recommend that you upgrade your config and proxy services, even though no changes were made for those services directly resutling in modifications to the .Stat Suite product.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

major (backward-incompatibility) changes:

  • dotstatsuite-data-explorer#472 DE search result download is now optional (New entry in Javascript settings search.downloadableDataflowResults). Since this option is not compatible with the indexation of externally defined/stored dataflows (see related specifications), thus this feature is disabled by default. When upgrading to this release, you must set this new configuration to true if you want to keep the download option on the search result page. (Documentation)
  • dotstatsuite-share#24 New dedicated redis databases for share and sfs services (Documentation). Share and sfs are using a redis service to store objects (REDIS_HOST, REDIS_PORT). Until now, share and sfs were using the same database by default, which is an issue when sfs flushes its data. From now on, share and sfs will have their dedicated databases (REDIS_DB):
    • share will use the existing database by default (0) to keep existing data
    • sfs will use a new database by default (1) and a re-index will restore previous data
      Notes:
    • REDIS_HOST, REDIS_PORT and REDIS_DB are environment variables than can be overridden
    • REDIS_DB is an integer (up to 15 without creating new databases in redis)
    • by default means that without changing anything, restarting sfs and share will apply the update
  • dotstatsuite-data-explorer#456 & dotstatsuite-share#12 Translations keys have changed and been updated (here below with default English examples):
    • 1 key was deleted "vx.no.data.available": "There is no data available." and replaced by 3 new keys: "log.error.sdmx.404": "There is no data available.", "log.error.sdmx.40x": "You are not permitted to see this data. Please try again after logging in with another account.", and "log.error.sdmx.xxx": "Whoops, something went wrong on our side. We are working to solve this. Please try again later."
    • 2 keys were deleted "de.app.viewer.owner": "SIS-CC", "de.app.viewer.term.label": "Terms & Conditions" and replaced by 3 new keys: "de.viewer.copyright.label": "©", "de.viewer.copyright.content.label": "SIS-CC {link}", and "de.viewer.copyright.content.link.label": "Terms & Conditions"
    • 2 new keys were added for the data-viewer in order to better handle error notifications: "data.not.found": "Oops, the view you are looking for doesn't exist. Make sure to use a correct and complete link.", "data.pending": "Oops, this view isn't accessible yet. Make sure to validate the related email address first."

significant & minor changes:

patches:

December 2, 2020

Patch Release .Stat Suite JS 6.1.1

This patch release includes a new minor version of the data-explorer service.

patches:

November 30, 2020

Release .Stat Suite JS 6.1.0

This release includes a new version of the data-explorer, data-viewer, data-lifecycle-manager, sdmx-faceted-search, and share (patch version 6.1.1) services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v7.13.2.

significant (non-major) and minor changes:

patches:

deprecations:

November 24, 2020

Patch Release .Stat Suite .NET 4.4.0

This patch release includes a new version of the core-sdmxri-nsi-ws service. This version resolves a change of default properties in the configuration of the nsiws. This applies ONLY to our custom nsiws plugin (on top of the nsiws.net v7.13.2), and this should be obsolete/by-design in the next versions of both nsiws.net (v8.x) and generic nsi plugin.
Find more about the nsiws.net configurations here.

patches:

October 26, 2020

Release .Stat Suite JS 6.0.0

This release includes a new major version of the data-explorer, data-viewer, data-lifecycle-manager, and sdmx-faceted-search services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v7.13.2

major (backward-incompatibility) changes:

  • dotstatsuite-sdmx-faceted-search#83 Indexing of dataflows in sub-categories misses adding the sub-category to the categoryscheme facet.
    NB: This results from the introduction in the SDMX-JSON format of a change on the link-type property (replaced by a mime-type) that was not parsed by the data-explorer as expected in the previous JS5.4.0 release, causing the disappearance of the indexed categories (loss of the categories in the DE homepage ‘browse by’).

minor changes:

patches:

October 7, 2020

DISCLAIMER for Release .Stat Suite .NET 4.3.0 and Release .Stat Suite JS 5.4.0

Both releases of the .NET and JS services of the .Stat Suite include backward-incompatible changes due to an update of the underlying SDMX-JSON format to its first official version within the SDMX standard. This update was included in ESTAT NSIWS v7.13.1 (deployed along with v7.13.2 here), and implied changes also in the JS services and applications.
It is therefore absolutely necessary, when upgrading your .Stat Suite instances to any future version, to simultaneously upgrade the .NET and JS services at least to versions .NET 4.3.0 and JS 5.4.0, in order to prevent misbehaviour or functional issues due to a backward-incompatibility of either of these services.
If your deployment topology does not include the .Stat Suite CORE (.NET) services (e.g. when only deploying the Data Explorer and its underlying services), you need either to align your NSIWS version to v7.13.1 minimum (v7.13.2 at best), or use an SDMX web service that is compatible with the official SDMX-JSON release version.

.Stat DLM Excel-Addin: The .NET v4.3.0 release also contains a new version of the .Stat DLM Excel-addin, namely v3.0.0.5, including the various changes and patches mentionned in the list below.
Reminder: The .Stat DLM Excel-addin is an Excel-based application (using the ClickOnce deployment technology) to be installed individually on each client machine. Therefore there is no Docker image produced for this component, and you need to retrieve it directly from its GitLab master branch.

Known limitations and/or deprecating behaviors:

  • Inconsistent rendering of DE table views with Unit of Measure information. Some cases described in the functional specifications may not always behave yet as expected. Note that the Unit of Measure feature behaviour can be temporarily ‘switched off’ through the DE configuration settings (by emtpying the UNIT_MEASURE_CONCEPTS annotation).
  • The table/chart views generated from the DE Share “Latest available data for the period defined” option are not always consistent with the original views in the DE browser (e.g. especially in use cases compiling Unit of Measure information).

Release .Stat Suite .NET 4.3.0

This release includes a new version of the core-sdmxri-nsi-plugin and dlm-excel-addin services.

significant (non-major) changes:

  • dotstatsuite-core-sdmxri-nsi-ws#66 Deploy NSIWS v7.13.2, including v7.13.1. Non backwards compatible changes/breaking changes have been introduced in the SDMX-JSON message output following the nsiws v7.13.1 release. Details can be found here.

minor changes:

patches:

Release .Stat Suite JS 5.4.0

This release includes a new version of the data-explorer, data-viewer, sdmx-faceted-search and data-lifecycle-manager services.

significant (non-major) changes:

minor changes:

patches:

September 2, 2020

Release .Stat Suite .NET 4.2.0

This release includes a new minor version of the core-sdmxri-nsi-plugin service and a major 3.0.0.4 version of the dlm-excel-addin.

minor changes:

patches:

Release .Stat Suite JS 5.3.1

This patch release includes a new minor version of the data-explorer service.

minor changes:

patch:

August 25, 2020

Release .Stat Suite JS 5.3.0

This new release includes a new minor version of the data-explorer, data-viewer and data-lifecycle-manager services.

Disclaimer: This new version of the .Stat Suite JavaScript services now uses the Keycloak access_token instead of the id_token. It is therefore required to change the Keycloak configuration accordingly. A good technical procedure is available here.

minor changes:

patches

July 23, 2020

Release .Stat Suite JS 5.2.0

This new release includes a new version of the data-explorer and data-lifecycle-manager services.

Disclaimer: Support and use of the Proof Key for Code Exchange (PKCE) in both DE & DLM by ensuring .Stat Suite using Keycloak v7+ (PKCE is supported from Keycloak v.7).
If Keycloak does not support pkce (below version 6), then everything will work properly (because pkce from .Stat DE and .stat DLM will be ignored).
When Keycloak is configured to enforce pkce, then non-compliant clients will break.

minor changes:

patches:

July 7, 2020

Release .Stat Suite .NET 4.1.0

This new release includes a new version of the core-sdmxri-nsi-plugin service, with the release of the Eurostat’s NSI-WS version 7.13.0.

minor changes:

  • dotstatsuite-core-sdmxri-nsi-ws#53 Deploy NSI version 7.13.0 (incl. 7.12.2). The task also included the creation, upon Eurostat authorization to the SIS-CC, of a new sub-group called Eurostat SDMX-RI components into GitLab for hosting Read-Only copies (manually mirrored) of some of the Eurostat components:

patches:

June 24, 2020

Release .Stat Suite .NET 4.0.0

This new release includes a new major version of the core-transfer service, with an upgrade to the .NET core version 3.1.
It also includes a minor version of the core-auth-management service.

major changes:

minor changes:

patches:

June 23, 2020

Release .Stat Suite JS 5.1.0

This new release includes a new version of the data-explorer, sdmx-faceted-search and data-lifecycle-manager services.

minor changes:

patches:

June 15, 2020

Release .Stat Suite .NET 3.9.0

This release includes a new minor version of the core-sdmxri-nsi-ws service, with the deployment of the Eurostat’s NSIWS v7.12.1 that contains the upgrade of the Mappingstore database to v6.9.
Disclaimer: you will need to upgrade to NSIWS v7.12.1 if you want to benefit from all the latest features/fixes in the .Stat Suite context. I you don’t have access to the Eurostat repository, you can check their full release note from here.

Both source-code and Docker images are concerned by this release.

minor changes:

patches:

May 18, 2020

Release .Stat Suite JS 5.0.0

This new release includes a new major version of the data-explorer, data-viewer, sdmx-faceted-search and data-lifecycle-manager services.

This version does not introduce backward incompatibility, but it brings significant changes in the css API and DE design.
WARNING: this release changes and extends how the Data Explorer Theme is customised and will result in your custom styles being lost (if you previously modified the default one). Please see this updated documentation for how to reapply your styles.

Both source-code and Docker images are concerned by this release.

major changes:

minor changes:

patches:

April 20, 2020

Release .Stat Suite .NET 3.5.0

This new release includes a new minor version of the transfer and auth-management services.
Both source-code and Docker images are concerned by this release.

minor changes:

patches:

April 2, 2020

Release .Stat Suite JS 4.2.0

This new release includes a new minor version of the .Stat Suite sdmx-faceted-search service.
Both source-code and Docker images are concerned by this release.

patch:

  • dotstatsuite-sdmx-faceted-search#67 Use of referencepartial parameter in dataflow queries for indexing.
    This fix resolves the issue that caused the .Stat Data Explorer to return all dataflows in all categories.
    With this solved, dataflows are only retrieved through the CategoryScheme facet value in which they are categorised.
    This feature needs at least nsiws.net v7.11.3 to fully function, and nsiws.net v7.11.0 to not break the request.

DE fixed issue for facet filter

March 31, 2020

Release .Stat Suite .NET 3.4.0

This new release includes a new minor version of the .Stat Suite core-transfer and core-data-access services.
Both source-code and Docker images are concerned by this release.

minor changes:

patches:

March 29, 2020

Release .Stat Suite .NET 3.3.0

This new release includes a new minor version of the .Stat Suite core-sdmxri-nsi-ws and core-sdmxri-nsi-plugin services.
Both source-code and Docker images are concerned by this release.

DISCLAIMER: This release includes an unofficial nsiws.net v7.11.3-hotfix version of NSI webservice, until the resolution of Eurostat’s SDMXRI-1315 issue is made available in the next NSIWS v7.11.5 (see details).

minor changes:

patches:

March 27, 2020

Release .Stat Suite JS 4.1.0

This new release includes a new minor version of the .Stat Suite data-explorer, config and data-lifecycle-manager services.
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.

minor changes:

  • dotstatsuite-data-explorer#210 RTL mode xlsx-populate, this allowing to programmatically switching the layout of a downloaded Excel data table between left-to-right and right-to-left.

patches:

March 05, 2020

Release .Stat Suite JS 4.0.1

This new release includes a new patch version of the .Stat Suite sdmx-faceted-search, config and data-explorer services.
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.

minor changes:

  • dotstatsuite-data-explorer#228 Add Data Explorer authentication. This is the first version of the Data Explorer auhtentication feature. The Data Explorer now works in both anonymous and authenticated modes. The same authenticated session (through Keycloak) is used when logging in DLM and DE. See documentation.

patches:

February 28, 2020

Release .Stat Suite JS 4.0.0 with SFS 2.0

This new MAJOR release includes a new version of the .Stat Suite sdmx-faceted-search, config, data-explorer, dotstatsuite-proxy, dotstatsuite-share, dotstatsuite-data-viewer and data-lifecycle-manager services.
The new version of the sdmx-faceted-search service requires nodejs version 13.3.x (mandatory to upgrade when installing/upgrading .Stat Suite from source-code).
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.

Important DISCLAIMER: There is a known dependency issue that causes the .Stat Data Explorer to return all dataflows in all categories.
This issue is addressed in dotstatsuite-core-sdmxri-nsi-ws#28 and fix was released by Eurostat in the latest NSI WebService v7.11.2 (issue#34 for its integration to .Stat Suite).

From this release on, incremental semantic version is also used in this changelog for categorising changes and bug fixes:
major: incompatible API change was introduced
minor: functionality was added in a backwards-compatible manner
patch: backwards-compatible bug fixes were applied

major changes:

minor changes:

patches:

February 17, 2020

Release .Stat Suite .NET 3.2.0

This new release includes a new version of the .Stat Suite core-sdmxri-nsi-ws, core-sdmxri-nsi-plugin and core-data-access services.
Both source-code and Docker images are concerned by this release.

Changes:

  • dotstatsuite-core-sdmxri-nsi-plugin#36 Deployment of the Eurostat’s NSI web service version 7.11.1 in DevOps (Eurostat’s changelog is included in GitLab’s ticket), now running on .NET Core 3.1.

Bug fixes:

February 04, 2020

Release .Stat Suite .NET v3.1.0

This new release includes a new version of the .Stat Suite core-sdmxri-nsi-ws and core-sdmxri-nsi-plugin services.
Both source-code and Docker images are concerned by this release.

Major changes:

  • dotstatsuite-core-sdmxri-nsi-ws#15 Allow configurable authenticated and/or anonymous access to a single instance of the NSI service. This is enabled by a new configuration parameter of the authentication auth.json with following content:
{
    "auth": {
        ...
        "allowAnonymous": true,
        ...
    }
}

The following kind of user rule is from now on required to be configured using the authorisation web service in order to allow public read access to anonymous users (.Stat DE users):

{
      "userMask": "*",      --> anonymous users
      "isGroup": false,
      "dataSpace": "staging:SIS-CC-stable",     --> data space 
      "artefactType": 0,
      "artefactAgencyId": "*",
      "artefactId": "*",
      "artefactVersion": "*",
      "permission": 3       --> allowed to read structure and data
}

You can check this page of the documentation in order to get the full list of available pemrissions.

January 31, 2020

Release .Stat Suite .NET v3.0.0

Disclaimer: This major version contains breaking changes in the dotstatsuite-core-transfer and dotstatsuite-core-auth-management services with changes to the authentication management (dotstatsuite-core-transfer#66, dotstatsuite-core-transfer#52 & dotstatsuite-core-auth-management#7). All impacted services are: core-transfer, core-auth-management, core-sdmxri-nsi-ws, core-data-access, and core-sdmxri-nsi-plugin.
This release has been tested and is running with the NSI WebService v7.10.8.
Both source-code and Docker images are concerned by this release.

Major changes:

  • dotstatsuite-core-transfer#66 Unify the auth config between the transfer service, the authentication service, and the NSI webservice. the Kubernetes strategy is also updated in order to use the authentication in the NSI WS.
  • dotstatsuite-core-sdmxri-nsi-ws#19 Deploy the NSI version 7.10.8 in DevOps. Including the following details from the Eurostat changelog:
    • Allow admin user to access all mapping store, to avoid having the administrator to logout and login everytime there is a new Mapping Store. OAuth2 caches the user with the mapping store.
    • The keyword latest at version is not allowed when deleting artefacts.
    • Fix SOAP 2.1 structure requests with CodeWhere and Stubs.

Minor changes:

Bug fixes:

January 28, 2020

Release .Stat Suite .NET v2.3.0

Disclaimer: This release contains breaking changes in the dotstatsuite-core-transfer service with a new entry in the dataspaces.private.json and the introduction of localization.json via the Dotstat.config nuget package.
Both source-code and Docker images are concerned by this release.

Major changes:

  • dotstatsuite-data-lifecycle-manager#50 Download selected data in Excel format (with simple layout).
    The original scope of this issue was extended in order to deliver a new component of the .Stat Suite: the .Stat DLM Excel-Addin.
    The DLM Excel-Addin is delivered as a ClickOnce application and is deployed using the common procedure for deployment of ClickOnce applications.
    You can access the installation files from here. A best practice is to install it on a shared network folder.
    The DLM Excel-Addin is made for configuring a single or multiple connection endpoints, in order to retrieve data within Excel tables with simple layout. This first version supports flat, time series down, and time series across layouts.
    You can find more information about deployment, set up and configuration here.

DLM Excel-Addin

DLM Excel-Addin

DLM Excel-Addin

DLM Excel-Addin

DLM Excel-Addin

Minor changes:

Bug fixes:

  • dotstatsuite-core-transfer#67 ERROR Invalid column name ‘DSD_MAX_TEXT_ATTR_LENGTH’.
  • dotstatsuite-core-common#102 Allow uploading data with long textual attribute values for dataflows in first and subsequent uploads.
    A configuration parameter is used as a default setting for MaxTextAttributeLength, and its default value is 150.
    A new DSD annotation of type MAXTEXTATTRIBUTELENGTH can be set by the user for a specific DSD (in its definition).
    At first data upload, the default MaxTextAttributeLength is used if the annotation MAXTEXTATTRIBUTELENGTH is not provided.
    Whenever a DSD-specific MaxTextAttributeLength is increased after a first data upload, then the new length is applied.
    When adding the annotation type MAXTEXTATTRIBUTELENGTH to a DSD definition:
  <structure:DataStructure id="SPS_ATT_TEST_TFWithNoRelationship_DSD" agencyID="OECD" version="11.1" isFinal="false">  
                 <common:Annotations>  
                     <common:Annotation>  
                         <common:AnnotationType>MAXTEXTATTRIBUTELENGTH</common:AnnotationType>  
                         <common:AnnotationText xml:lang="en">500</common:AnnotationText>  
                     </common:Annotation>  
                 </common:Annotations>  
                 <common:Name xml:lang="en">Attribute test dsd scenario with TIME_FORMAT attribute relationship on dimensions ETA and SESSO</common:Name>  
                 <structure:DataStructureComponents>

Then the textual attribute value can be longer in its display than the default length.
The user is also notified through the email of the transfer action in the DLM:

MaxTexAttributeLength message

January 8, 2020

Release .Stat Suite .NET v2.2.0

This new release includes a new version of the .Stat Suite .NET/SQL services.
Both source-code and Docker images are concerned by this release.

Major changes:

E-mail notifications from the transfer service for data transfer activities and made by .Stat DLM users are received with clearer and well-formulated information.

The following two groupings/categories now describe the issue, its origin and it also identifies responsibility or who may be concerned by the type of issue:

  • Type of issues
  • System transaction
  • Database connection
  • Database structure
  • User Import/Transfer
  • Data Modelling
  • Level of issues
  • WARNING
  • ERROR

More details can be found here (descriptive types of known issues, causes and solutions).

Several tasks remain to be done for providing some of the messages with more comprehensive, non-technical descriptions of the cause, and descriptions of the possible remedy solutions in case of failure.

Minor changes:

Bug fixes:

December 5 2019

Release .Stat Suite .NET v2.1.0

This new release includes a new version of the .NET/SQL services .Stat Suite.
This release has been tested with and is using the NSI WebService v7.10.3.
Both source-code and Docker images are concerned by this release.

Minor changes:

Bug fixes:

December 02, 2019

Release .Stat Suite JS milestone 7

This new release concerns the JavaScript JS part of .Stat Suite, exclusively the data-explorer application.

Major changes:

  • dotstatsuite-data-explorer#171 Excel download with styles and colors applied from the original Data Explorer table view.
  • dotstatsuite-data-explorer#128 Additional downloads for external resources can be added at the Dataflow level by using the SDMX annotation type EXT_RESOURCE.
    It is possible to relate a specific dataflow, in the SDMX Dataflow structure definition, to one or several materials (any type of file format) that are stored externally (outside .Stat Suite). The feature is supported by the SDMX annotation EXT_RESOURCE type, that is localised and defined at the dataflow level.

In the following example, the dataflow structure definition contains 3 annotations of EXT_RESOURCE type for localised (in English and French) externally stored files to link to the datalfow. Each hyperlink can be enriched with a link to an image for its display as an icon of the link in the final representation:

Dataflow structure with external resources

Once published in the Data Explorer, the external resources related to the dataflow are available from the visualisation page (table and chart views) under the “Download” option, and in addition to the default data download links:

Dataflow display with external sources

Switching the language of the Data Explorer from English to French will make the localised external files available for download (acccording to the dataflow structure definition).

Bug fixes:

November 19, 2019

Release .Stat Suite JS v3.0.0

This new release concerns the JavaScript JS part of .Stat Suite, especially the data-explorer, data-viewer and config. applications.
This release has been tested with the NSI WebService v7.10.1 and v7.10.3.

Major changes:

  • dotstatsuite-data-explorer#98 Download data in Excel from the .Stat Data Explorer data views.
    This first version of the Excel download feature allows the users to download their current filtered and customised data view into an Excel table (.xlsx extension format). It is triggered from the visualisation page’s drop-down “Download” option (in addition to the .csv formats and in both table and chart views).
    It includes the layout of the table view, but not yet the colors in the table (to be done with #171).
    Attributes in flags and footnotes are also included in the downloaded Excel file, as internal links into a second sheet with exactly the same layout, but without any of the values and with flags in the cells instead of the values.
  • dotstatsuite-data-explorer#8 Attributes in footnotes can be displayed at the always highest possible level in the .Stat DE data table.
    Depending on the scenario and whenever the attribute value is the same in a section of the table, and in order to not repeat an unnecessary same information, then an attribute will be displayed in a footnote at the observation value level, time series level, grouping of dimension members level, or dataflow level.
  • dotstatsuite-data-explorer#35 The NOT_DISPLAYED SDMX annotation allows to hide dimensions, dimension values, attributes, or attribute values from the .Stat Data Explorer table and chart representations, including from the dimension filters when relevant.
  • dotstatsuite-data-explorer#45 Add a UI feature for changing the LastNObservations parameter. See details here and here about the behavior of the LastNObservations feature.
  • dotstatsuite-data-explorer#125 Reintroducing the choropleth map chart to .Stat Data Explorer, including writing the instructions for the setup and configuration of a chart map. Documentation is available here.
  • dotstatsuite-data-explorer#62 Integrate the Time Period selections into the Used Filters area of the .Stat Data Explorer visualisation pages.

Minor changes:

Bug fixes:

November 13, 2019

Release .Stat Suite .NET v2.0.0

This new release includes a new version of the .NET/SQL services the .Stat Suite.

Major changes:

Minor changes:

Bug fixes:

October 11, 2019

Release .Stat Suite CORE services (GitLab milestone Release BE 1)

Major changes:

Minor changes:

Bug fixes:

October 08, 2019

Release .stat suite JS v2.0.0

NSI WS compatible versions: this new release concerns only the JavaScript JS part of the .Stat Suite and is compatible with the NSI WS versions from v7.xx to v7.8.

Major changes:

Minor changes:

Bug fixes:

August 20, 2019

Major changes:

Minor changes:

Bug fixes:

August 13, 2019

Major changes:

  • dotstatsuite-data-explorer#89 Drag&Drop feature is restricted to minimum of 1 dimension on row. The last dimension set to Row cannot be removed.
    RowLimit1
  • dotstatsuite-data-explorer#71 Extend the behavior of the DEFAULT filter selections by applying also items from search ; the sdmx DEFAULT annotation rule is combined with the end-user’s selections from the search result page.

Minor changes:

Bug fixes:

July 11, 2019

Major changes:

Minor changes:

Bug fixes:

June 4, 2019

Major changes:

  • dotstatsuite-data-explorer#23 Time period range selection with annual and non-annual frequencies.
    This new feature of the Data Explorer visualisation page is replacing the time range slider and enables the end-user to select a time frequency and an corresponding range of time period values for a given dataflow view.

    Time Period Selector
  • dotstatsuite-data-explorer#26 Share API and publication workflow.
    The new Share service for table and chart sharing views (including url and embedded codes) relies on a email validation workflow process detailed here.
    The end-user needs to provide a valid email address in order to receive a validation email along with the url and embedded code of the shared table or chart view.

    Share email
  • dotstatsuite-data-explorer#33 LastNObservations default setting allowing to define a default number of latest time series observations per time series returned for table and chart views.
    This is implemented as an sdmx annotation parameter, but it should then be enhanced with a UI feature allowing the end-user to change this default parameter.
  • dotstatsuite-data-explorer#15 .CSV download includes labels/codes option.
    From the search result, .csv downloads include both codes and localised labels of dimensions and dimension members.
    From the visualisation pages, .csv downloads include codes only when “Code” option is set on the UI, and .csv downloads include both codes and localised labels when “Name” or “Both” options are set on the UI.

Minor changes:

Bug fixes:

May 17, 2019

Major changes:

Bug fixes:

April 25, 2019

Major changes:

Minor changes:

Bug fixes:

April 17, 2019

Minor changes:

Bug fixes:

April 11, 2019

Major changes:

  • dotstatsuite-data-lifecycle-manager#37 .Stat DLM list artefacts feature enhancements, including information about data and observation number for a given dataflow, data preview and dataflow categorisation under the same expand/collapse feature.

Minor changes:

Bug fixes:

Release v0.5 - March 2019

Release notes and known limitations in Github

Release v0.4 - February 2019

Release notes and known limitations in Github

Release 14.12.2018

Release notes and known limitations in Github

Release 28.09.2018

Release notes and known limitations in Github

Release 10.07.2018

Release notes and known limitations in Github