.Stat Suite documentation

Changelog

General disclaimer: If you upgrade a .Stat Suite installation from a .NET version below 5.0.0, to .NET v5.0.0 or higher, you must follow the migration procedure explained as follows. All MappingSets must be generated specifically in the context of the .Stat Suite. This is to be done through the following methods:

  • Before you generate the MappingSets (see next bullet), if during the migration/upgrade to version 5.0.0 or higher with the DBUP tool, some DSDs/Dataflows migrations fail (check the logs using the transfer service /status/requests method), then you should migrate these dataflows manually using the Transfer service method /init/dataflow. Note that this should happen only extremely rarely, and would be caused by previous inconsistencies in the DB state. In case the manual dataflow migration is still unsuccessful then it is recommended to delete the underlying DSD, cleanup the related DB objects (using the Transfer service method /cleanup/dsd), recreate the data structures and reload the related data.
  • Generate the MappingSets for all already existing dataflows when the .Stat Suite .NET version is migrated to 5.0.0 or higher, using the .Stat Suite Transfer service method /init/allMappingsets. This method must be called manually as the very last step of the deployment of the new version (after all components are deployed/updated, and after the DBUP tool has run to update the databases). (Documentation)
  • Generate the MappingSet for any newly added dataflow using the .Stat Suite Transfer service method init/dataflow. This can be done using the Transfer service Swagger UI. (Documentation)
  • Generate the MappingSet for any newly added dataflow by uploading any data (in DLM or with the .Stat Suite Transfer service). In other words, the MappingSet of a newly added dataflow will be automatically generated once you upload data for this dataflow.

July 8, 2021

Release .Stat Suite JS 9.0.0

This major release includes a new version of the data-explorer, sdmx-faceted-search, data-viewer, config, and data-lifecycle-manager services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.3.0.

TECHNICAL DISCLAIMERS:
Config folder contents significantly change: tenants.json contains more ; settings.json has no more relation to dataspace or datasource ; datasources.json is removed.

  • New tenant model:
    • tenants.json is bigger with centralized spaces/datasources information
    • separation between spaces and datasources concepts
    • new concept of scope used to bind spaces/datasources to apps
    • references to datasources in settings.json moved in tenants.json
    • routes (when proxy is used) can use tenant slug: <tenantId> or <tenantId>:<scopeId>
  • Major change in the search service (new tenant model adaptation):
    • reminder from previous release, a collection is required for each tenant (Documentation)
    • clean and re-index all data for all tenants to avoid side-effects
  • ROBOTS_POLICY env var in DE and DV to configure search engine indexation (default enabled), don’t forget to override robot.txt file accordingly to the env var value
  • AUTHZ_SERVER_URL env var is now used to bind the DLM to AUTHZ (previously in settings.json)
  • TRANSFER_SERVER_URL env var dropped (was used in DLM) and replaced by transferUrl in tenants.json (at scope level): DLM is not bound to a transfer, it is the spaces used within its scope that are bound to a transfer (which is more flexible because different transfers can be used without deploying several DLM)
  • To override space definition in search service:
    • temporary fix waiting for a better integration in the new tenant model
    • no more datasources.json file to mount at sfs level
    • define searchUrl in spaces definition, sfs will use it if defined in tenants.json
    • define headers in datasources definition in tenants.json

For your migration/upgrade process, think about:

  1. datasources are inside the spaces
  2. Define your datasources in tenants.json
  3. From your settings.json file, retrieve your datasources IDs and add it in scope(s) inside your tenants.json, then once this is done, remove the obsolete datasources IDs from your settings.json file.
  4. Then delete your datasources.json file (becoming useless).
  5. When using proxy, change of root: the tenant identification is made of a tenant and a scope for applications supporting scopes: DLM, DE, but not DV.

major (backward-incompatible) changes:

  • dotstatsuite-config-data#4 New tenant model adaptation. See above disclaimer and Documentation. Also available a video record demoing and explaining the adaptation of this new tenant model.
  • dotstatsuite-sdmx-faceted-search#28 Allow individually adding new dataflows in the search index. The previous sfs query to update an individual dataflow PATCH /admin/dataflow is replaced by the new ‘upsert’ POST /admin/dataflow query (Documentation).
  • dotstatsuite-config#29 (Refactoring) config data i18n follow up. Final migration of app configs from .stat-suite/dotstatsuite-config to .stat-suite/dotstatsuite-config-data

significant and minor changes:

patch changes:

June 16, 2021

Release .Stat Suite .NET 6.4.0

This release includes a new version of the sdmxri-nsi-ws service.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.2.0.

minor changes:

patch changes:

June 1, 2021

Release .Stat Suite .NET 6.3.0

This release includes a new version of the sdmxri-nsi-ws service.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.3.

minor changes:

patch changes:

Release .Stat Suite JS 8.0.1 - Keycloak

This release only includes a new custom image of the keycloak Docker image.
.Stat Suite compatibility to JS 8.0.0 and .NET 6.3.0
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.3.

significant change:

  • Connect Keycloak to Active Directory ADFS. This is a release of the new and ‘custom’ Keycloak Docker image including the new plugin that allows to connect an Active Directory ADFS as a third-party provider to Keycloak:
  • Docker hub image
  • Newly made public Keycloak project
  • ADFS-to-keycloak plugin
  • Documentation on how to configure ADFS as a SAML provider to Keycloak

May 19, 2021

Release .Stat Suite JS 8.0.0

This major release includes a new version of the data-explorer, sdmx-faceted-search, data-viewer, share, config, and data-lifecycle-manager services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

major (backward-incompatible) changes:

  • dotstatsuite-sdmx-faceted-search#40 Solr upgrade to version 8.7. (Updated API documentation)
    Disclaimer:
    • sfs is now multi-tenant, which means that each API call should include a tenant. Providing a tenant can be done in 2 ways:
      • as a query param: &tenant=test
      • as a header: x-tenant=test
    • Existing data can be dropped, it is reindexed (new solr major version, no need to struggle with a migration). No need to create a SOLR Core anymore!
    • Create a Collection in solr named defaulttenant: http://0.0.0.0:8983/solr/admin/collections?action=CREATE&name=defaulttenant&numShards=1&collection.configName=_default
    • Add DEFAULT_TENANT env. variable with the value defaulttenant to sfs
    • Note that sfs API is unchanged when using DEFAULT_TENANT, all calls without a tenant will use the value of DEFAULT_TENANT as a tenant
    • data-explorer will request sfs without a tenant until the upcoming new sfs model is released (see this list of iteration D issues)
    • On-premise installation: http call to solr api locally (since solr should be exposed)
    • Docker installation: http call to solr locally or from sfs container (see latest docker-compose)
  • keycloak#8 & keycloak#11 (DevOps) Keycloak upgrade to version 12.0.4.
    Disclaimer:
    • Migration process to this new major version is quite sensitive even though not backward-incompatible
    • Backup your data before processing with the upgrade!
    • See this tutorial page about Keycloak configuration, and also Kubernetes strategy and historical migrations documented here (as source of inspiration from our DevOps deployment process)
  • dotstatsuite-share#9 (Refactoring) Remove hard-coded dependency on third-party mail service MailGun.
    Disclaimer:
    • This allows the full SMTP setup of the share service: see updated documentation about using SMTP connection.
    • Add the new MAIL_FROM env. variable with the contextual value '"Share" share@your-domain.org' (was previsouly hard-coded)
    • Mailgun API key is kept as a param for an extra period of time (until September 30, 2021). After this date, the Mailgun API key will be definitely removed from the source code.
  • dotstatsuite-data-explorer#525 (Refactoring) Align i18n versions of data-viewer with data-explorer.
    Disclaimer:
    • data-viewer i18n is now using the default translations similarly to the data-explorer, hence no need to create overrides specific to data-viewer.

Additional disclaimer: This release implies to temporarily retrieve any new and default i18n key and translation values in a new public project and from its master branch https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-config-data/-/tree/master/i18n. The fluid process of releasing i18n default translations as part of the open-source .Stat Suite source code will be complete with the next iteration C delivery.

significant and minor changes:

patch changes:

May 11, 2021

Release .Stat Suite .NET 6.2.0

This release includes a new minor version of the core-data-access service.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

minor changes:

April 27, 2021

Release .Stat Suite .NET 6.1.0

This release includes a new version of the core-transfer, core-data-access, core-auth-management, and excel-addin services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

minor changes:

patch changes:

April 8, 2021

Patch Release .Stat Suite JS 7.2.1

This release includes a new patch version of the data-explorer and data-viewer services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

patch changes:

April 1, 2021

Release .Stat Suite JS 7.2.0

This release includes a new version of the data-explorer, data-lifecycle-manager, data-viewer, share, config, and proxy services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

Disclaimer for .Stat Suite Docker setup: as a result of dotstatsuite-config#27, a new Docker image needs to be used for Docker setup: siscc/dotstatsuite-config-prod:master -> siscc/dotstatsuite-config:master. Note that the previous siscc/dotstatsuite-config-prod:master will still work and will be removed at some point in the future.
In addition, the i18n translations were implicitly loaded, now it is done explicitly: folder i18n to mount config/data/prod/i18n -> opt/i18n.

minor changes:

patch changes:

March 15, 2021

Patch Release .Stat Suite JS 7.1.1

This release includes a new patch version of the data-explorer service.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

patch change:

  • dotstatsuite-data-explorer#500 Search results additional downloads for external resources does not work. All use cases for internally/externally stored dataflows, and externally stored materials to download must be handled.

March 10, 2021

Release .Stat Suite JS 7.1.0

This release includes a new version of the data-explorer, data-viewer, sdmx-faceted-search, and data-lifecycle-manager services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

significant and minor changes:

patch changes:

March 5, 2021

Release .Stat Suite .NET 6.0.0

This major release includes a new version of the core-transfer, core-sdmxri-nsi-ws, core-auth-management, and core-data-access services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

Disclaimer on the performance evolutions in this release. Here below is a summary of comparison of the performance before and after using the SDMX-RI NSI web service default data plugin in .Stat Suite (all performance tests processes, definitions and types are documented here). Although a series of improvements can already be noticed with this release, it does not yet meet the expected performance improvements for high-load situations (many parallel data queries). Additionally required improvements (related mainly to the default nsi-ws-plugin) are being addressed with dotstatsuite-core-sdmxri-nsi-ws#111.

  • Comparison of the performance before and after using the SDMX-RI NSI web service default data plugin
    Smoke-test data imports (time increase by 15%) 
    data_import_time................: avg=16.85s --> avg=19.42s  
✓ { datasetSize:extraSmall }....: avg=2.36s --> avg=2.6s  
✓ { datasetSize:small }.........: avg=31.34s --> avg=36.24

    Smoke-test data extractions (time decrease by 20%)

    http_req_duration...............: avg=227.65ms --> avg=182.51ms

    Load-test data extractions (time increase by 605%)

    http_req_duration...............: avg=1.33s --> avg=8.05s  
✓ { datasetSize:extraSmall }....: avg=1.6s --> avg=7.07s  
✓ { datasetSize:small }.........: avg=1.39s --> avg=8.85s

    Stress-test data extractions (time increase by 29%)

    http_req_duration................: avg=1.89s --> avg=2.43s  
✓ { datasetSize:extraSmall }.....: avg=2.27s --> avg=2.3s  
✓ { datasetSize:small }..........: avg=1.99s --> avg=2.63s

    Spike-test data extractions (time decrease by 3%)

    http_req_duration................: avg=3.8s --> avg=3.68s  
✗ { datasetSize:extraSmall }.....: avg=4.25s --> avg=4.11s  
✗ { datasetSize:small }..........: avg=3.88s --> avg=3.76s

    Soak-test data extractions (time increase by 97%)

    http_req_duration..........: avg=670.42ms --> avg=1.32s

major (backward-incompatible) changes:

  • dotstatsuite-core-sdmxri-nsi-plugin#37 Always protect non-public data. Whether openid-connect authentication is turned ON or OFF in the NSI web-service configuration, anonymous users ALWAYS only get data that is especially made public (through appropriate permissions).
    This represents a major change from previous releases, where with NSI openid-connect authentication was turned OFF, all users (necessarily unauthenticated) could get all data. Now, Docker image with authorization is enabled (turned ON) by default.
  • Known limitation: the current DLM feature for viewing artefact content (in xml format) in a new web-browser tab by clicking on the (hyperlinked) name of an artefact does not work anymore with the new nsi-ws authentication in place, unless a generic CanReadStructures permission is granted to all users incl. anonymous users. (Updated documentation)
  • dotstatsuite-core-transfer#124 Allow for non-numeric and coded measures (observation values) in .Stat CORE. Now, according to the SDMX standard, String is the default SDMX data TextType format when it is not specified in the data structure definition (previsouly, Float was the default TextType format). (Documentation)

minor changes:

patch changes:

January 25, 2021

Release .Stat Suite .NET 5.0.0

This major release includes a new version of the core-transfer, core-sdmxri-nsi-ws, core-auth-management, and core-data-access services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

WARNING: While this first .Stat Suite Core (.NET) version using the SDMX-RI NSI web service default data plugin approach (according to the DB model review step 1) includes the related basic features such as PIT features, authorisations and new REST features, it does not contain yet the necessary performance enhancements foreseen in step 2. This does not represent an issue for data structure definitions that have only a few dimensions and attributes, but those with more dimensions and attributes are now too slow for production mode. These performance enhancements of step 2 are expected to be released very soon. We therefore recommend to not use this release in production mode but to wait for the next .Stat Suite Core (.NET) release, especially if you manage dataflows with many dimensions and attributes.

DISCLAIMER: Starting with this release, the SDMX-RI NSI web service within the .Stat Suite will use the default NSI data plugin instead of the previous .Stat Suite-specific data plugin. This requires the usage of a SDMX-RI MappingSet object (stored in the MappingStore structure database) for each of the dataflows. Except for MappingSets that are manually entered by the user through the SDMX-RI MappingAssistant, all MappingSets must be generated specifically in the context of the .Stat Suite. This is to be done through the following methods:

  • Before you generate the MappingSets (see next bullet), if during the migration/upgrade to version 5.0.0 with the DBUP tool some DSDs/Dataflows migrations fail (check the logs using the transfer service /status/requests method), then you should migrate these dataflows manually using the Transfer service method /init/dataflow. Note that his should happen only extremely rarely, and would be caused by previous inconsistencies in the DB state. In case the manual dataflow migration is still unsuccessful then it is recommended to delete the underlying DSD, cleanup the related DB objects (using the Transfer service method /cleanup/dsd), recreate the data structures and reload the related data.
  • Generate the MappingSets for all already existing dataflows when the .Stat Suite .NET version is migrated to 5.0.0 using the .Stat Suite Transfer service method /init/allMappingsets. This method must be called manually as the very last step of the deployment of the new version (after all components are deployed/updated, and after the DBUP tool has run to update the databases). (Documentation)
  • Generate the MappingSet for any newly added dataflow using the .Stat Suite Transfer service method init/dataflow. This can be done using the Transfer service Swagger UI. (Documentation)
  • Generate the MappingSet for any newly added dataflow by uploading any data (in DLM or with the .Stat Suite Transfer service). In other words, the MappingSet of a newly added dataflow will be automatically generated once you upload data for this dataflow.

Important NOTE: From this release on, any new/additional attribution of an admin permission (AdminRole) to the authorization service must use the new permission id:4095 (see major change below).

major (backward-incompatibility) changes:

  • dotstatsuite-core-sdmxri-nsi-plugin#48 Remove the custom .Stat Suite nsi-plugin (Replace NSI-Plugin).
  • dotstatsuite-core-auth-management#20 AdminRole permission id is changed from ‘2047’ to ‘4095’ in auth DB.
  • The nsiws.net v8.1.2 included a bug correction SDMXRI-1443 Support of observation values with string data type at SDMX-JSON data writers with a major impact in .Stat Suite. Whenever observation values are of String type, then the SDMX-JSON messages generated in SDMX-RI NSI extractions use the (correct) String type. It also means that string data types are now correctly exposed as string types in the SDMX-RI NSI extractions, and the DE also shows strings as strings without altering concepts such as thousand or decimal separators.

minor changes:

patches:

January 21, 2021

Release .Stat Suite JS 7.0.0

This major release includes a new version of the data-lifecycle-manager, data-explorer, data-viewer, sdmx-faceted-search, and share services.
Tip: we also recommend that you upgrade your config and proxy services, even though no changes were made for those services directly resutling in modifications to the .Stat Suite product.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.

major (backward-incompatibility) changes:

  • dotstatsuite-data-explorer#472 DE search result download is now optional (New entry in Javascript settings search.downloadableDataflowResults). Since this option is not compatible with the indexation of externally defined/stored dataflows (see related specifications), thus this feature is disabled by default. When upgrading to this release, you must set this new configuration to true if you want to keep the download option on the search result page. (Documentation)
  • dotstatsuite-share#24 New dedicated redis databases for share and sfs services (Documentation). Share and sfs are using a redis service to store objects (REDIS_HOST, REDIS_PORT). Until now, share and sfs were using the same database by default, which is an issue when sfs flushes its data. From now on, share and sfs will have their dedicated databases (REDIS_DB):
    • share will use the existing database by default (0) to keep existing data
    • sfs will use a new database by default (1) and a re-index will restore previous data
      Notes:
    • REDIS_HOST, REDIS_PORT and REDIS_DB are environment variables than can be overridden
    • REDIS_DB is an integer (up to 15 without creating new databases in redis)
    • by default means that without changing anything, restarting sfs and share will apply the update
  • dotstatsuite-data-explorer#456 & dotstatsuite-share#12 Translations keys have changed and been updated (here below with default English examples):
    • 1 key was deleted "vx.no.data.available": "There is no data available." and replaced by 3 new keys: "log.error.sdmx.404": "There is no data available.", "log.error.sdmx.40x": "You are not permitted to see this data. Please try again after logging in with another account.", and "log.error.sdmx.xxx": "Whoops, something went wrong on our side. We are working to solve this. Please try again later."
    • 2 keys were deleted "de.app.viewer.owner": "SIS-CC", "de.app.viewer.term.label": "Terms & Conditions" and replaced by 3 new keys: "de.viewer.copyright.label": "©", "de.viewer.copyright.content.label": "SIS-CC {link}", and "de.viewer.copyright.content.link.label": "Terms & Conditions"
    • 2 new keys were added for the data-viewer in order to better handle error notifications: "data.not.found": "Oops, the view you are looking for doesn't exist. Make sure to use a correct and complete link.", "data.pending": "Oops, this view isn't accessible yet. Make sure to validate the related email address first."

significant & minor changes:

patches:

December 2, 2020

Patch Release .Stat Suite JS 6.1.1

This patch release includes a new minor version of the data-explorer service.

patches:

November 30, 2020

Release .Stat Suite JS 6.1.0

This release includes a new version of the data-explorer, data-viewer, data-lifecycle-manager, sdmx-faceted-search, and share (patch version 6.1.1) services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v7.13.2.

significant (non-major) and minor changes:

patches:

deprecations:

November 24, 2020

Patch Release .Stat Suite .NET 4.4.0

This patch release includes a new version of the core-sdmxri-nsi-ws service. This version resolves a change of default properties in the configuration of the nsiws. This applies ONLY to our custom nsiws plugin (on top of the nsiws.net v7.13.2), and this should be obsolete/by-design in the next versions of both nsiws.net (v8.x) and generic nsi plugin.
Find more about the nsiws.net configurations here.

patches:

October 26, 2020

Release .Stat Suite JS 6.0.0

This release includes a new major version of the data-explorer, data-viewer, data-lifecycle-manager, and sdmx-faceted-search services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v7.13.2

major (backward-incompatibility) changes:

  • dotstatsuite-sdmx-faceted-search#83 Indexing of dataflows in sub-categories misses adding the sub-category to the categoryscheme facet.
    NB: This results from the introduction in the SDMX-JSON format of a change on the link-type property (replaced by a mime-type) that was not parsed by the data-explorer as expected in the previous JS5.4.0 release, causing the disappearance of the indexed categories (loss of the categories in the DE homepage ‘browse by’).

minor changes:

patches:

October 7, 2020

DISCLAIMER for Release .Stat Suite .NET 4.3.0 and Release .Stat Suite JS 5.4.0

Both releases of the .NET and JS services of the .Stat Suite include backward-incompatible changes due to an update of the underlying SDMX-JSON format to its first official version within the SDMX standard. This update was included in ESTAT NSIWS v7.13.1 (deployed along with v7.13.2 here), and implied changes also in the JS services and applications.
It is therefore absolutely necessary, when upgrading your .Stat Suite instances to any future version, to simultaneously upgrade the .NET and JS services at least to versions .NET 4.3.0 and JS 5.4.0, in order to prevent misbehaviour or functional issues due to a backward-incompatibility of either of these services.
If your deployment topology does not include the .Stat Suite CORE (.NET) services (e.g. when only deploying the Data Explorer and its underlying services), you need either to align your NSIWS version to v7.13.1 minimum (v7.13.2 at best), or use an SDMX web service that is compatible with the official SDMX-JSON release version.

.Stat DLM Excel-Addin: The .NET v4.3.0 release also contains a new version of the .Stat DLM Excel-addin, namely v3.0.0.5, including the various changes and patches mentionned in the list below.
Reminder: The .Stat DLM Excel-addin is an Excel-based application (using the ClickOnce deployment technology) to be installed individually on each client machine. Therefore there is no Docker image produced for this component, and you need to retrieve it directly from its GitLab master branch.

Known limitations and/or deprecating behaviors:

  • Inconsistent rendering of DE table views with Unit of Measure information. Some cases described in the functional specifications may not always behave yet as expected. Note that the Unit of Measure feature behaviour can be temporarily ‘switched off’ through the DE configuration settings (by emtpying the UNIT_MEASURE_CONCEPTS annotation).
  • The table/chart views generated from the DE Share “Latest available data for the period defined” option are not always consistent with the original views in the DE browser (e.g. especially in use cases compiling Unit of Measure information).

Release .Stat Suite .NET 4.3.0

This release includes a new version of the core-sdmxri-nsi-plugin and dlm-excel-addin services.

significant (non-major) changes:

  • dotstatsuite-core-sdmxri-nsi-ws#66 Deploy NSIWS v7.13.2, including v7.13.1. Non backwards compatible changes/breaking changes have been introduced in the SDMX-JSON message output following the nsiws v7.13.1 release. Details can be found here.

minor changes:

patches:

Release .Stat Suite JS 5.4.0

This release includes a new version of the data-explorer, data-viewer, sdmx-faceted-search and data-lifecycle-manager services.

significant (non-major) changes:

minor changes:

patches:

September 2, 2020

Release .Stat Suite .NET 4.2.0

This release includes a new minor version of the core-sdmxri-nsi-plugin service and a major 3.0.0.4 version of the dlm-excel-addin.

minor changes:

patches:

Release .Stat Suite JS 5.3.1

This patch release includes a new minor version of the data-explorer service.

minor changes:

patch:

August 25, 2020

Release .Stat Suite JS 5.3.0

This new release includes a new minor version of the data-explorer, data-viewer and data-lifecycle-manager services.

Disclaimer: This new version of the .Stat Suite JavaScript services now uses the Keycloak access_token instead of the id_token. It is therefore required to change the Keycloak configuration accordingly. A good technical procedure is available here.

minor changes:

patches

July 23, 2020

Release .Stat Suite JS 5.2.0

This new release includes a new version of the data-explorer and data-lifecycle-manager services.

Disclaimer: Support and use of the Proof Key for Code Exchange (PKCE) in both DE & DLM by ensuring .Stat Suite using Keycloak v7+ (PKCE is supported from Keycloak v.7).
If Keycloak does not support pkce (below version 6), then everything will work properly (because pkce from .Stat DE and .stat DLM will be ignored).
When Keycloak is configured to enforce pkce, then non-compliant clients will break.

minor changes:

patches:

July 7, 2020

Release .Stat Suite .NET 4.1.0

This new release includes a new version of the core-sdmxri-nsi-plugin service, with the release of the Eurostat’s NSI-WS version 7.13.0.

minor changes:

  • dotstatsuite-core-sdmxri-nsi-ws#53 Deploy NSI version 7.13.0 (incl. 7.12.2). The task also included the creation, upon Eurostat authorization to the SIS-CC, of a new sub-group called Eurostat SDMX-RI components into GitLab for hosting Read-Only copies (manually mirrored) of some of the Eurostat components:

patches:

June 24, 2020

Release .Stat Suite .NET 4.0.0

This new release includes a new major version of the core-transfer service, with an upgrade to the .NET core version 3.1.
It also includes a minor version of the core-auth-management service.

major changes:

minor changes:

patches:

June 23, 2020

Release .Stat Suite JS 5.1.0

This new release includes a new version of the data-explorer, sdmx-faceted-search and data-lifecycle-manager services.

minor changes:

patches:

June 15, 2020

Release .Stat Suite .NET 3.9.0

This release includes a new minor version of the core-sdmxri-nsi-ws service, with the deployment of the Eurostat’s NSIWS v7.12.1 that contains the upgrade of the Mappingstore database to v6.9.
Disclaimer: you will need to upgrade to NSIWS v7.12.1 if you want to benefit from all the latest features/fixes in the .Stat Suite context. I you don’t have access to the Eurostat repository, you can check their full release note from here.

Both source-code and Docker images are concerned by this release.

minor changes:

patches:

May 18, 2020

Release .Stat Suite JS 5.0.0

This new release includes a new major version of the data-explorer, data-viewer, sdmx-faceted-search and data-lifecycle-manager services.

This version does not introduce backward incompatibility, but it brings significant changes in the css API and DE design.
WARNING: this release changes and extends how the Data Explorer Theme is customised and will result in your custom styles being lost (if you previously modified the default one). Please see this updated documentation for how to reapply your styles.

Both source-code and Docker images are concerned by this release.

major changes:

minor changes:

patches:

April 20, 2020

Release .Stat Suite .NET 3.5.0

This new release includes a new minor version of the transfer and auth-management services.
Both source-code and Docker images are concerned by this release.

minor changes:

patches:

April 2, 2020

Release .Stat Suite JS 4.2.0

This new release includes a new minor version of the .Stat Suite sdmx-faceted-search service.
Both source-code and Docker images are concerned by this release.

patch:

  • dotstatsuite-sdmx-faceted-search#67 Use of referencepartial parameter in dataflow queries for indexing.
    This fix resolves the issue that caused the .Stat Data Explorer to return all dataflows in all categories.
    With this solved, dataflows are only retrieved through the CategoryScheme facet value in which they are categorised.
    This feature needs at least nsiws.net v7.11.3 to fully function, and nsiws.net v7.11.0 to not break the request.

DE fixed issue for facet filter

March 31, 2020

Release .Stat Suite .NET 3.4.0

This new release includes a new minor version of the .Stat Suite core-transfer and core-data-access services.
Both source-code and Docker images are concerned by this release.

minor changes:

patches:

March 29, 2020

Release .Stat Suite .NET 3.3.0

This new release includes a new minor version of the .Stat Suite core-sdmxri-nsi-ws and core-sdmxri-nsi-plugin services.
Both source-code and Docker images are concerned by this release.

DISCLAIMER: This release includes an unofficial nsiws.net v7.11.3-hotfix version of NSI webservice, until the resolution of Eurostat’s SDMXRI-1315 issue is made available in the next NSIWS v7.11.5 (see details).

minor changes:

patches:

March 27, 2020

Release .Stat Suite JS 4.1.0

This new release includes a new minor version of the .Stat Suite data-explorer, config and data-lifecycle-manager services.
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.

minor changes:

  • dotstatsuite-data-explorer#210 RTL mode xlsx-populate, this allowing to programmatically switching the layout of a downloaded Excel data table between left-to-right and right-to-left.

patches:

March 05, 2020

Release .Stat Suite JS 4.0.1

This new release includes a new patch version of the .Stat Suite sdmx-faceted-search, config and data-explorer services.
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.

minor changes:

  • dotstatsuite-data-explorer#228 Add Data Explorer authentication. This is the first version of the Data Explorer auhtentication feature. The Data Explorer now works in both anonymous and authenticated modes. The same authenticated session (through Keycloak) is used when logging in DLM and DE. See documentation.

patches:

February 28, 2020

Release .Stat Suite JS 4.0.0 with SFS 2.0

This new MAJOR release includes a new version of the .Stat Suite sdmx-faceted-search, config, data-explorer, dotstatsuite-proxy, dotstatsuite-share, dotstatsuite-data-viewer and data-lifecycle-manager services.
The new version of the sdmx-faceted-search service requires nodejs version 13.3.x (mandatory to upgrade when installing/upgrading .Stat Suite from source-code).
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.

Important DISCLAIMER: There is a known dependency issue that causes the .Stat Data Explorer to return all dataflows in all categories.
This issue is addressed in dotstatsuite-core-sdmxri-nsi-ws#28 and fix was released by Eurostat in the latest NSI WebService v7.11.2 (issue#34 for its integration to .Stat Suite).

From this release on, incremental semantic version is also used in this changelog for categorising changes and bug fixes:
major: incompatible API change was introduced
minor: functionality was added in a backwards-compatible manner
patch: backwards-compatible bug fixes were applied

major changes:

minor changes:

patches:

February 17, 2020

Release .Stat Suite .NET 3.2.0

This new release includes a new version of the .Stat Suite core-sdmxri-nsi-ws, core-sdmxri-nsi-plugin and core-data-access services.
Both source-code and Docker images are concerned by this release.

Changes:

  • dotstatsuite-core-sdmxri-nsi-plugin#36 Deployment of the Eurostat’s NSI web service version 7.11.1 in DevOps (Eurostat’s changelog is included in GitLab’s ticket), now running on .NET Core 3.1.

Bug fixes:

February 04, 2020

Release .Stat Suite .NET v3.1.0

This new release includes a new version of the .Stat Suite core-sdmxri-nsi-ws and core-sdmxri-nsi-plugin services.
Both source-code and Docker images are concerned by this release.

Major changes:

  • dotstatsuite-core-sdmxri-nsi-ws#15 Allow configurable authenticated and/or anonymous access to a single instance of the NSI service. This is enabled by a new configuration parameter of the authentication auth.json with following content:
{
    "auth": {
        ...
        "allowAnonymous": true,
        ...
    }
}

The following kind of user rule is from now on required to be configured using the authorisation web service in order to allow public read access to anonymous users (.Stat DE users):

{
      "userMask": "*",      --> anonymous users
      "isGroup": false,
      "dataSpace": "staging:SIS-CC-stable",     --> data space 
      "artefactType": 0,
      "artefactAgencyId": "*",
      "artefactId": "*",
      "artefactVersion": "*",
      "permission": 3       --> allowed to read structure and data
}

You can check this page of the documentation in order to get the full list of available pemrissions.

January 31, 2020

Release .Stat Suite .NET v3.0.0

Disclaimer: This major version contains breaking changes in the dotstatsuite-core-transfer and dotstatsuite-core-auth-management services with changes to the authentication management (dotstatsuite-core-transfer#66, dotstatsuite-core-transfer#52 & dotstatsuite-core-auth-management#7). All impacted services are: core-transfer, core-auth-management, core-sdmxri-nsi-ws, core-data-access, and core-sdmxri-nsi-plugin.
This release has been tested and is running with the NSI WebService v7.10.8.
Both source-code and Docker images are concerned by this release.

Major changes:

  • dotstatsuite-core-transfer#66 Unify the auth config between the transfer service, the authentication service, and the NSI webservice. the Kubernetes strategy is also updated in order to use the authentication in the NSI WS.
  • dotstatsuite-core-sdmxri-nsi-ws#19 Deploy the NSI version 7.10.8 in DevOps. Including the following details from the Eurostat changelog:
    • Allow admin user to access all mapping store, to avoid having the administrator to logout and login everytime there is a new Mapping Store. OAuth2 caches the user with the mapping store.
    • The keyword latest at version is not allowed when deleting artefacts.
    • Fix SOAP 2.1 structure requests with CodeWhere and Stubs.

Minor changes:

Bug fixes:

January 28, 2020

Release .Stat Suite .NET v2.3.0

Disclaimer: This release contains breaking changes in the dotstatsuite-core-transfer service with a new entry in the dataspaces.private.json and the introduction of localization.json via the Dotstat.config nuget package.
Both source-code and Docker images are concerned by this release.

Major changes:

  • dotstatsuite-data-lifecycle-manager#50 Download selected data in Excel format (with simple layout).
    The original scope of this issue was extended in order to deliver a new component of the .Stat Suite: the .Stat DLM Excel-Addin.
    The DLM Excel-Addin is delivered as a ClickOnce application and is deployed using the common procedure for deployment of ClickOnce applications.
    You can access the installation files from here. A best practice is to install it on a shared network folder.
    The DLM Excel-Addin is made for configuring a single or multiple connection endpoints, in order to retrieve data within Excel tables with simple layout. This first version supports flat, time series down, and time series across layouts.
    You can find more information about deployment, set up and configuration here.

DLM Excel-Addin

DLM Excel-Addin

DLM Excel-Addin

DLM Excel-Addin

DLM Excel-Addin

Minor changes:

Bug fixes:

  • dotstatsuite-core-transfer#67 ERROR Invalid column name ‘DSD_MAX_TEXT_ATTR_LENGTH’.
  • dotstatsuite-core-common#102 Allow uploading data with long textual attribute values for dataflows in first and subsequent uploads.
    A configuration parameter is used as a default setting for MaxTextAttributeLength, and its default value is 150.
    A new DSD annotation of type MAXTEXTATTRIBUTELENGTH can be set by the user for a specific DSD (in its definition).
    At first data upload, the default MaxTextAttributeLength is used if the annotation MAXTEXTATTRIBUTELENGTH is not provided.
    Whenever a DSD-specific MaxTextAttributeLength is increased after a first data upload, then the new length is applied.
    When adding the annotation type MAXTEXTATTRIBUTELENGTH to a DSD definition:
  <structure:DataStructure id="SPS_ATT_TEST_TFWithNoRelationship_DSD" agencyID="OECD" version="11.1" isFinal="false">  
                 <common:Annotations>  
                     <common:Annotation>  
                         <common:AnnotationType>MAXTEXTATTRIBUTELENGTH</common:AnnotationType>  
                         <common:AnnotationText xml:lang="en">500</common:AnnotationText>  
                     </common:Annotation>  
                 </common:Annotations>  
                 <common:Name xml:lang="en">Attribute test dsd scenario with TIME_FORMAT attribute relationship on dimensions ETA and SESSO</common:Name>  
                 <structure:DataStructureComponents>

Then the textual attribute value can be longer in its display than the default length.
The user is also notified through the email of the transfer action in the DLM:

MaxTexAttributeLength message

January 8, 2020

Release .Stat Suite .NET v2.2.0

This new release includes a new version of the .Stat Suite .NET/SQL services.
Both source-code and Docker images are concerned by this release.

Major changes:

E-mail notifications from the transfer service for data transfer activities and made by .Stat DLM users are received with clearer and well-formulated information.

The following two groupings/categories now describe the issue, its origin and it also identifies responsibility or who may be concerned by the type of issue:

  • Type of issues
  • System transaction
  • Database connection
  • Database structure
  • User Import/Transfer
  • Data Modelling
  • Level of issues
  • WARNING
  • ERROR

More details can be found here (descriptive types of known issues, causes and solutions).

Several tasks remain to be done for providing some of the messages with more comprehensive, non-technical descriptions of the cause, and descriptions of the possible remedy solutions in case of failure.

Minor changes:

Bug fixes:

December 5 2019

Release .Stat Suite .NET v2.1.0

This new release includes a new version of the .NET/SQL services .Stat Suite.
This release has been tested with and is using the NSI WebService v7.10.3.
Both source-code and Docker images are concerned by this release.

Minor changes:

Bug fixes:

December 02, 2019

Release .Stat Suite JS milestone 7

This new release concerns the JavaScript JS part of .Stat Suite, exclusively the data-explorer application.

Major changes:

  • dotstatsuite-data-explorer#171 Excel download with styles and colors applied from the original Data Explorer table view.
  • dotstatsuite-data-explorer#128 Additional downloads for external resources can be added at the Dataflow level by using the SDMX annotation type EXT_RESOURCE.
    It is possible to relate a specific dataflow, in the SDMX Dataflow structure definition, to one or several materials (any type of file format) that are stored externally (outside .Stat Suite). The feature is supported by the SDMX annotation EXT_RESOURCE type, that is localised and defined at the dataflow level.

In the following example, the dataflow structure definition contains 3 annotations of EXT_RESOURCE type for localised (in English and French) externally stored files to link to the datalfow. Each hyperlink can be enriched with a link to an image for its display as an icon of the link in the final representation:

Dataflow structure with external resources

Once published in the Data Explorer, the external resources related to the dataflow are available from the visualisation page (table and chart views) under the “Download” option, and in addition to the default data download links:

Dataflow display with external sources

Switching the language of the Data Explorer from English to French will make the localised external files available for download (acccording to the dataflow structure definition).

Bug fixes:

November 19, 2019

Release .Stat Suite JS v3.0.0

This new release concerns the JavaScript JS part of .Stat Suite, especially the data-explorer, data-viewer and config. applications.
This release has been tested with the NSI WebService v7.10.1 and v7.10.3.

Major changes:

  • dotstatsuite-data-explorer#98 Download data in Excel from the .Stat Data Explorer data views.
    This first version of the Excel download feature allows the users to download their current filtered and customised data view into an Excel table (.xlsx extension format). It is triggered from the visualisation page’s drop-down “Download” option (in addition to the .csv formats and in both table and chart views).
    It includes the layout of the table view, but not yet the colors in the table (to be done with #171).
    Attributes in flags and footnotes are also included in the downloaded Excel file, as internal links into a second sheet with exactly the same layout, but without any of the values and with flags in the cells instead of the values.
  • dotstatsuite-data-explorer#8 Attributes in footnotes can be displayed at the always highest possible level in the .Stat DE data table.
    Depending on the scenario and whenever the attribute value is the same in a section of the table, and in order to not repeat an unnecessary same information, then an attribute will be displayed in a footnote at the observation value level, time series level, grouping of dimension members level, or dataflow level.
  • dotstatsuite-data-explorer#35 The NOT_DISPLAYED SDMX annotation allows to hide dimensions, dimension values, attributes, or attribute values from the .Stat Data Explorer table and chart representations, including from the dimension filters when relevant.
  • dotstatsuite-data-explorer#45 Add a UI feature for changing the LastNObservations parameter. See details here and here about the behavior of the LastNObservations feature.
  • dotstatsuite-data-explorer#125 Reintroducing the choropleth map chart to .Stat Data Explorer, including writing the instructions for the setup and configuration of a chart map. Documentation is available here.
  • dotstatsuite-data-explorer#62 Integrate the Time Period selections into the Used Filters area of the .Stat Data Explorer visualisation pages.

Minor changes:

Bug fixes:

November 13, 2019

Release .Stat Suite .NET v2.0.0

This new release includes a new version of the .NET/SQL services the .Stat Suite.

Major changes:

Minor changes:

Bug fixes:

October 11, 2019

Release .Stat Suite CORE services (GitLab milestone Release BE 1)

Major changes:

Minor changes:

Bug fixes:

October 08, 2019

Release .stat suite JS v2.0.0

NSI WS compatible versions: this new release concerns only the JavaScript JS part of the .Stat Suite and is compatible with the NSI WS versions from v7.xx to v7.8.

Major changes:

Minor changes:

Bug fixes:

August 20, 2019

Major changes:

Minor changes:

Bug fixes:

August 13, 2019

Major changes:

  • dotstatsuite-data-explorer#89 Drag&Drop feature is restricted to minimum of 1 dimension on row. The last dimension set to Row cannot be removed.
    RowLimit1
  • dotstatsuite-data-explorer#71 Extend the behavior of the DEFAULT filter selections by applying also items from search ; the sdmx DEFAULT annotation rule is combined with the end-user’s selections from the search result page.

Minor changes:

Bug fixes:

July 11, 2019

Major changes:

Minor changes:

Bug fixes:

June 4, 2019

Major changes:

  • dotstatsuite-data-explorer#23 Time period range selection with annual and non-annual frequencies.
    This new feature of the Data Explorer visualisation page is replacing the time range slider and enables the end-user to select a time frequency and an corresponding range of time period values for a given dataflow view.

    Time Period Selector
  • dotstatsuite-data-explorer#26 Share API and publication workflow.
    The new Share service for table and chart sharing views (including url and embedded codes) relies on a email validation workflow process detailed here.
    The end-user needs to provide a valid email address in order to receive a validation email along with the url and embedded code of the shared table or chart view.

    Share email
  • dotstatsuite-data-explorer#33 LastNObservations default setting allowing to define a default number of latest time series observations per time series returned for table and chart views.
    This is implemented as an sdmx annotation parameter, but it should then be enhanced with a UI feature allowing the end-user to change this default parameter.
  • dotstatsuite-data-explorer#15 .CSV download includes labels/codes option.
    From the search result, .csv downloads include both codes and localised labels of dimensions and dimension members.
    From the visualisation pages, .csv downloads include codes only when “Code” option is set on the UI, and .csv downloads include both codes and localised labels when “Name” or “Both” options are set on the UI.

Minor changes:

Bug fixes:

May 17, 2019

Major changes:

Bug fixes:

April 25, 2019

Major changes:

Minor changes:

Bug fixes:

April 17, 2019

Minor changes:

Bug fixes:

April 11, 2019

Major changes:

  • dotstatsuite-data-lifecycle-manager#37 .Stat DLM list artefacts feature enhancements, including information about data and observation number for a given dataflow, data preview and dataflow categorisation under the same expand/collapse feature.

Minor changes:

Bug fixes:

Release v0.5 - March 2019

Release notes and known limitations in Github

Release v0.4 - February 2019

Release notes and known limitations in Github

Release 14.12.2018

Release notes and known limitations in Github

Release 28.09.2018

Release notes and known limitations in Github

Release 10.07.2018

Release notes and known limitations in Github