.Stat Suite documentation

Changelog

November 24, 2020

Patch Release .Stat Suite .NET 4.4.0

This patch release includes a new version of the core-sdmxri-nsi-ws service. This version resolves a change of default properties in the configuration of the nsiws. This applies ONLY to our custom nsiws plugin (on top of the nsiws.net v7.13.2), and this should be obsolete/by-design in the next versions of both nsiws.net (v8.x) and generic nsi plugin.
Find more about the nsiws.net configurations here.

patches:

October 26, 2020

Release .Stat Suite JS 6.0.0

This release includes a new major version of the data-explorer, data-viewer, data-lifecycle-manager, and sdmx-faceted-search services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v7.13.2

major (backward-incompatibility) changes:

  • dotstatsuite-sdmx-faceted-search#83 Indexing of dataflows in sub-categories misses adding the sub-category to the categoryscheme facet.
    NB: This results from the introduction in the SDMX-JSON format of a change on the link-type property (replaced by a mime-type) that was not parsed by the data-explorer as expected in the previous JS5.4.0 release, causing the disappearance of the indexed categories (loss of the categories in the DE homepage ‘browse by’).

minor changes:

patches:

October 7, 2020

DISCLAIMER for Release .Stat Suite .NET 4.3.0 and Release .Stat Suite JS 5.4.0

Both releases of the .NET and JS services of the .Stat Suite include backward-incompatible changes due to an update of the underlying SDMX-JSON format to its first official version within the SDMX standard. This update was included in ESTAT NSIWS v7.13.1 (deployed along with v7.13.2 here), and implied changes also in the JS services and applications.
It is therefore absolutely necessary, when upgrading your .Stat Suite instances to any future version, to simultaneously upgrade the .NET and JS services at least to versions .NET 4.3.0 and JS 5.4.0, in order to prevent misbehaviour or functional issues due to a backward-incompatibility of either of these services.
If your deployment topology does not include the .Stat Suite CORE (.NET) services (e.g. when only deploying the Data Explorer and its underlying services), you need either to align your NSIWS version to v7.13.1 minimum (v7.13.2 at best), or use an SDMX web service that is compatible with the official SDMX-JSON release version.

.Stat DLM Excel-Addin: The .NET v4.3.0 release also contains a new version of the .Stat DLM Excel-addin, namely v3.0.0.5, including the various changes and patches mentionned in the list below.
Reminder: The .Stat DLM Excel-addin is an Excel-based application (using the ClickOnce deployment technology) to be installed individually on each client machine. Therefore there is no Docker image produced for this component, and you need to retrieve it directly from its GitLab master branch.

Known limitations and/or deprecating behaviors:

  • Inconsistent rendering of DE table views with Unit of Measure information. Some cases described in the functional specifications may not always behave yet as expected. Note that the Unit of Measure feature behaviour can be temporarily ‘switched off’ through the DE configuration settings (by emtpying the UNIT_MEASURE_CONCEPTS annotation).
  • The table/chart views generated from the DE Share “Latest available data for the period defined” option are not always consistent with the original views in the DE browser (e.g. especially in use cases compiling Unit of Measure information).

Release .Stat Suite .NET 4.3.0

This release includes a new version of the core-sdmxri-nsi-plugin and dlm-excel-addin services.

significant (non-major) changes:

  • dotstatsuite-core-sdmxri-nsi-ws#66 Deploy NSIWS v7.13.2, including v7.13.1. Non backwards compatible changes/breaking changes have been introduced in the SDMX-JSON message output following the nsiws v7.13.1 release. Details can be found here.

minor changes:

patches:

Release .Stat Suite JS 5.4.0

This release includes a new version of the data-explorer, data-viewer, sdmx-faceted-search and data-lifecycle-manager services.

significant (non-major) changes:

minor changes:

patches:

September 2, 2020

Release .Stat Suite .NET 4.2.0

This release includes a new minor version of the core-sdmxri-nsi-plugin service and a major 3.0.0.4 version of the dlm-excel-addin.

minor changes:

patches:

Release .Stat Suite JS 5.3.1

This patch release includes a new minor version of the data-explorer service.

minor changes:

patch:

August 25, 2020

Release .Stat Suite JS 5.3.0

This new release includes a new minor version of the data-explorer, data-viewer and data-lifecycle-manager services.

Disclaimer: This new version of the .Stat Suite JavaScript services now uses the Keycloak access_token instead of the id_token. It is therefore required to change the Keycloak configuration accordingly. A good technical procedure is available here.

minor changes:

patches

July 23, 2020

Release .Stat Suite JS 5.2.0

This new release includes a new version of the data-explorer and data-lifecycle-manager services.

Disclaimer: Support and use of the Proof Key for Code Exchange (PKCE) in both DE & DLM by ensuring .Stat Suite using Keycloak v7+ (PKCE is supported from Keycloak v.7).
If Keycloak does not support pkce (below version 6), then everything will work properly (because pkce from .Stat DE and .stat DLM will be ignored).
When Keycloak is configured to enforce pkce, then non-compliant clients will break.

minor changes:

patches:

July 7, 2020

Release .Stat Suite .NET 4.1.0

This new release includes a new version of the core-sdmxri-nsi-plugin service, with the release of the Eurostat’s NSI-WS version 7.13.0.

minor changes:

  • dotstatsuite-core-sdmxri-nsi-ws#53 Deploy NSI version 7.13.0 (incl. 7.12.2). The task also included the creation, upon Eurostat authorization to the SIS-CC, of a new sub-group called Eurostat SDMX-RI components into GitLab for hosting Read-Only copies (manually mirrored) of some of the Eurostat components:

patches:

June 24, 2020

Release .Stat Suite .NET 4.0.0

This new release includes a new major version of the core-transfer service, with an upgrade to the .NET core version 3.1.
It also includes a minor version of the core-auth-management service.

major changes:

minor changes:

patches:

June 23, 2020

Release .Stat Suite JS 5.1.0

This new release includes a new version of the data-explorer, sdmx-faceted-search and data-lifecycle-manager services.

minor changes:

patches:

June 15, 2020

Release .Stat Suite .NET 3.9.0

This release includes a new minor version of the core-sdmxri-nsi-ws service, with the deployment of the Eurostat’s NSIWS v7.12.1 that contains the upgrade of the Mappingstore database to v6.9.
Disclaimer: you will need to upgrade to NSIWS v7.12.1 if you want to benefit from all the latest features/fixes in the .Stat Suite context. I you don’t have access to the Eurostat repository, you can check their full release note from here.

Both source-code and Docker images are concerned by this release.

minor changes:

patches:

May 18, 2020

Release .Stat Suite JS 5.0.0

This new release includes a new major version of the data-explorer, data-viewer, sdmx-faceted-search and data-lifecycle-manager services.

This version does not introduce backward incompatibility, but it brings significant changes in the css API and DE design.
WARNING: this release changes and extends how the Data Explorer Theme is customised and will result in your custom styles being lost (if you previously modified the default one). Please see this updated documentation for how to reapply your styles.

Both source-code and Docker images are concerned by this release.

major changes:

minor changes:

patches:

April 20, 2020

Release .Stat Suite .NET 3.5.0

This new release includes a new minor version of the transfer and auth-management services.
Both source-code and Docker images are concerned by this release.

minor changes:

patches:

April 2, 2020

Release .Stat Suite JS 4.2.0

This new release includes a new minor version of the .Stat Suite sdmx-faceted-search service.
Both source-code and Docker images are concerned by this release.

patch:

  • dotstatsuite-sdmx-faceted-search#67 Use of referencepartial parameter in dataflow queries for indexing.
    This fix resolves the issue that caused the .Stat Data Explorer to return all dataflows in all categories.
    With this solved, dataflows are only retrieved through the CategoryScheme facet value in which they are categorised.
    This feature needs at least nsiws.net v7.11.3 to fully function, and nsiws.net v7.11.0 to not break the request.

DE fixed issue for facet filter

March 31, 2020

Release .Stat Suite .NET 3.4.0

This new release includes a new minor version of the .Stat Suite core-transfer and core-data-access services.
Both source-code and Docker images are concerned by this release.

minor changes:

patches:

March 29, 2020

Release .Stat Suite .NET 3.3.0

This new release includes a new minor version of the .Stat Suite core-sdmxri-nsi-ws and core-sdmxri-nsi-plugin services.
Both source-code and Docker images are concerned by this release.

DISCLAIMER: This release includes an unofficial nsiws.net v7.11.3-hotfix version of NSI webservice, until the resolution of Eurostat’s SDMXRI-1315 issue is made available in the next NSIWS v7.11.5 (see details).

minor changes:

patches:

March 27, 2020

Release .Stat Suite JS 4.1.0

This new release includes a new minor version of the .Stat Suite data-explorer, config and data-lifecycle-manager services.
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.

minor changes:

  • dotstatsuite-data-explorer#210 RTL mode xlsx-populate, this allowing to programmatically switching the layout of a downloaded Excel data table between left-to-right and right-to-left.

patches:

March 05, 2020

Release .Stat Suite JS 4.0.1

This new release includes a new patch version of the .Stat Suite sdmx-faceted-search, config and data-explorer services.
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.

minor changes:

  • dotstatsuite-data-explorer#228 Add Data Explorer authentication. This is the first version of the Data Explorer auhtentication feature. The Data Explorer now works in both anonymous and authenticated modes. The same authenticated session (through Keycloak) is used when logging in DLM and DE. See documentation.

patches:

February 28, 2020

Release .Stat Suite JS 4.0.0 with SFS 2.0

This new MAJOR release includes a new version of the .Stat Suite sdmx-faceted-search, config, data-explorer, dotstatsuite-proxy, dotstatsuite-share, dotstatsuite-data-viewer and data-lifecycle-manager services.
The new version of the sdmx-faceted-search service requires nodejs version 13.3.x (mandatory to upgrade when installing/upgrading .Stat Suite from source-code).
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.

Important DISCLAIMER: There is a known dependency issue that causes the .Stat Data Explorer to return all dataflows in all categories.
This issue is addressed in dotstatsuite-core-sdmxri-nsi-ws#28 and fix was released by Eurostat in the latest NSI WebService v7.11.2 (issue#34 for its integration to .Stat Suite).

From this release on, incremental semantic version is also used in this changelog for categorising changes and bug fixes:
major: incompatible API change was introduced
minor: functionality was added in a backwards-compatible manner
patch: backwards-compatible bug fixes were applied

major changes:

minor changes:

patches:

February 17, 2020

Release .Stat Suite .NET 3.2.0

This new release includes a new version of the .Stat Suite core-sdmxri-nsi-ws, core-sdmxri-nsi-plugin and core-data-access services.
Both source-code and Docker images are concerned by this release.

Changes:

  • dotstatsuite-core-sdmxri-nsi-plugin#36 Deployment of the Eurostat’s NSI web service version 7.11.1 in DevOps (Eurostat’s changelog is included in GitLab’s ticket), now running on .NET Core 3.1.

Bug fixes:

February 04, 2020

Release .Stat Suite .NET v3.1.0

This new release includes a new version of the .Stat Suite core-sdmxri-nsi-ws and core-sdmxri-nsi-plugin services.
Both source-code and Docker images are concerned by this release.

Major changes:

  • dotstatsuite-core-sdmxri-nsi-ws#15 Allow configurable authenticated and/or anonymous access to a single instance of the NSI service. This is enabled by a new configuration parameter of the authentication auth.json with following content:
{
    "auth": {
        ...
        "allowAnonymous": true,
        ...
    }
}

The following kind of user rule is from now on required to be configured using the authorisation web service in order to allow public read access to anonymous users (.Stat DE users):

{
      "userMask": "*",      --> anonymous users
      "isGroup": false,
      "dataSpace": "staging:SIS-CC-stable",     --> data space 
      "artefactType": 0,
      "artefactAgencyId": "*",
      "artefactId": "*",
      "artefactVersion": "*",
      "permission": 3       --> allowed to read structure and data
}

You can check this page of the documentation in order to get the full list of available pemrissions.

January 31, 2020

Release .Stat Suite .NET v3.0.0

Disclaimer: This major version contains breaking changes in the dotstatsuite-core-transfer and dotstatsuite-core-auth-management services with changes to the authentication management (dotstatsuite-core-transfer#66, dotstatsuite-core-transfer#52 & dotstatsuite-core-auth-management#7). All impacted services are: core-transfer, core-auth-management, core-sdmxri-nsi-ws, core-data-access, and core-sdmxri-nsi-plugin.
This release has been tested and is running with the NSI WebService v7.10.8.
Both source-code and Docker images are concerned by this release.

Major changes:

  • dotstatsuite-core-transfer#66 Unify the auth config between the transfer service, the authentication service, and the NSI webservice. the Kubernetes strategy is also updated in order to use the authentication in the NSI WS.
  • dotstatsuite-core-sdmxri-nsi-ws#19 Deploy the NSI version 7.10.8 in DevOps. Including the following details from the Eurostat changelog:
    • Allow admin user to access all mapping store, to avoid having the administrator to logout and login everytime there is a new Mapping Store. OAuth2 caches the user with the mapping store.
    • The keyword latest at version is not allowed when deleting artefacts.
    • Fix SOAP 2.1 structure requests with CodeWhere and Stubs.

Minor changes:

Bug fixes:

January 28, 2020

Release .Stat Suite .NET v2.3.0

Disclaimer: This release contains breaking changes in the dotstatsuite-core-transfer service with a new entry in the dataspaces.private.json and the introduction of localization.json via the Dotstat.config nuget package.
Both source-code and Docker images are concerned by this release.

Major changes:

  • dotstatsuite-data-lifecycle-manager#50 Download selected data in Excel format (with simple layout).
    The original scope of this issue was extended in order to deliver a new component of the .Stat Suite: the .Stat DLM Excel-Addin.
    The DLM Excel-Addin is delivered as a ClickOnce application and is deployed using the common procedure for deployment of ClickOnce applications.
    You can access the installation files from here. A best practice is to install it on a shared network folder.
    The DLM Excel-Addin is made for configuring a single or multiple connection endpoints, in order to retrieve data within Excel tables with simple layout. This first version supports flat, time series down, and time series across layouts.
    You can find more information about deployment, set up and configuration here.

DLM Excel-Addin

DLM Excel-Addin

DLM Excel-Addin

DLM Excel-Addin

DLM Excel-Addin

Minor changes:

Bug fixes:

  • dotstatsuite-core-transfer#67 ERROR Invalid column name ‘DSD_MAX_TEXT_ATTR_LENGTH’.
  • dotstatsuite-core-common#102 Allow uploading data with long textual attribute values for dataflows in first and subsequent uploads.
    A configuration parameter is used as a default setting for MaxTextAttributeLength, and its default value is 150.
    A new DSD annotation of type MAXTEXTATTRIBUTELENGTH can be set by the user for a specific DSD (in its definition).
    At first data upload, the default MaxTextAttributeLength is used if the annotation MAXTEXTATTRIBUTELENGTH is not provided.
    Whenever a DSD-specific MaxTextAttributeLength is increased after a first data upload, then the new length is applied.
    When adding the annotation type MAXTEXTATTRIBUTELENGTH to a DSD definition:
  <structure:DataStructure id="SPS_ATT_TEST_TFWithNoRelationship_DSD" agencyID="OECD" version="11.1" isFinal="false">  
                 <common:Annotations>  
                     <common:Annotation>  
                         <common:AnnotationType>MAXTEXTATTRIBUTELENGTH</common:AnnotationType>  
                         <common:AnnotationText xml:lang="en">500</common:AnnotationText>  
                     </common:Annotation>  
                 </common:Annotations>  
                 <common:Name xml:lang="en">Attribute test dsd scenario with TIME_FORMAT attribute relationship on dimensions ETA and SESSO</common:Name>  
                 <structure:DataStructureComponents>

Then the textual attribute value can be longer in its display than the default length.
The user is also notified through the email of the transfer action in the DLM:

MaxTexAttributeLength message

January 8, 2020

Release .Stat Suite .NET v2.2.0

This new release includes a new version of the .Stat Suite .NET/SQL services.
Both source-code and Docker images are concerned by this release.

Major changes:

E-mail notifications from the transfer service for data transfer activities and made by .Stat DLM users are received with clearer and well-formulated information.

The following two groupings/categories now describe the issue, its origin and it also identifies responsibility or who may be concerned by the type of issue:

  • Type of issues
  • System transaction
  • Database connection
  • Database structure
  • User Import/Transfer
  • Data Modelling
  • Level of issues
  • WARNING
  • ERROR

More details can be found here (descriptive types of known issues, causes and solutions).

Several tasks remain to be done for providing some of the messages with more comprehensive, non-technical descriptions of the cause, and descriptions of the possible remedy solutions in case of failure.

Minor changes:

Bug fixes:

December 5 2019

Release .Stat Suite .NET v2.1.0

This new release includes a new version of the .NET/SQL services .Stat Suite.
This release has been tested with and is using the NSI WebService v7.10.3.
Both source-code and Docker images are concerned by this release.

Minor changes:

Bug fixes:

December 02, 2019

Release .Stat Suite JS milestone 7

This new release concerns the JavaScript JS part of .Stat Suite, exclusively the data-explorer application.

Major changes:

  • dotstatsuite-data-explorer#171 Excel download with styles and colors applied from the original Data Explorer table view.
  • dotstatsuite-data-explorer#128 Additional downloads for external resources can be added at the Dataflow level by using the SDMX annotation type EXT_RESOURCE.
    It is possible to relate a specific dataflow, in the SDMX Dataflow structure definition, to one or several materials (any type of file format) that are stored externally (outside .Stat Suite). The feature is supported by the SDMX annotation EXT_RESOURCE type, that is localised and defined at the dataflow level.

In the following example, the dataflow structure definition contains 3 annotations of EXT_RESOURCE type for localised (in English and French) externally stored files to link to the datalfow. Each hyperlink can be enriched with a link to an image for its display as an icon of the link in the final representation:

Dataflow structure with external resources

Once published in the Data Explorer, the external resources related to the dataflow are available from the visualisation page (table and chart views) under the “Download” option, and in addition to the default data download links:

Dataflow display with external sources

Switching the language of the Data Explorer from English to French will make the localised external files available for download (acccording to the dataflow structure definition).

Bug fixes:

November 19, 2019

Release .Stat Suite JS v3.0.0

This new release concerns the JavaScript JS part of .Stat Suite, especially the data-explorer, data-viewer and config. applications.
This release has been tested with the NSI WebService v7.10.1 and v7.10.3.

Major changes:

  • dotstatsuite-data-explorer#98 Download data in Excel from the .Stat Data Explorer data views.
    This first version of the Excel download feature allows the users to download their current filtered and customised data view into an Excel table (.xlsx extension format). It is triggered from the visualisation page’s drop-down “Download” option (in addition to the .csv formats and in both table and chart views).
    It includes the layout of the table view, but not yet the colors in the table (to be done with #171).
    Attributes in flags and footnotes are also included in the downloaded Excel file, as internal links into a second sheet with exactly the same layout, but without any of the values and with flags in the cells instead of the values.
  • dotstatsuite-data-explorer#8 Attributes in footnotes can be displayed at the always highest possible level in the .Stat DE data table.
    Depending on the scenario and whenever the attribute value is the same in a section of the table, and in order to not repeat an unnecessary same information, then an attribute will be displayed in a footnote at the observation value level, time series level, grouping of dimension members level, or dataflow level.
  • dotstatsuite-data-explorer#35 The NOT_DISPLAYED SDMX annotation allows to hide dimensions, dimension values, attributes, or attribute values from the .Stat Data Explorer table and chart representations, including from the dimension filters when relevant.
  • dotstatsuite-data-explorer#45 Add a UI feature for changing the LastNObservations parameter. See details here and here about the behavior of the LastNObservations feature.
  • dotstatsuite-data-explorer#125 Reintroducing the choropleth map chart to .Stat Data Explorer, including writing the instructions for the setup and configuration of a chart map. Documentation is available here.
  • dotstatsuite-data-explorer#62 Integrate the Time Period selections into the Used Filters area of the .Stat Data Explorer visualisation pages.

Minor changes:

Bug fixes:

November 13, 2019

Release .Stat Suite .NET v2.0.0

This new release includes a new version of the .NET/SQL services the .Stat Suite.

Major changes:

Minor changes:

Bug fixes:

October 11, 2019

Release .Stat Suite CORE services (GitLab milestone Release BE 1)

Major changes:

Minor changes:

Bug fixes:

October 08, 2019

Release .stat suite JS v2.0.0

NSI WS compatible versions: this new release concerns only the JavaScript JS part of the .Stat Suite and is compatible with the NSI WS versions from v7.xx to v7.8.

Major changes:

Minor changes:

Bug fixes:

August 20, 2019

Major changes:

Minor changes:

Bug fixes:

August 13, 2019

Major changes:

  • dotstatsuite-data-explorer#89 Drag&Drop feature is restricted to minimum of 1 dimension on row. The last dimension set to Row cannot be removed.
    RowLimit1
  • dotstatsuite-data-explorer#71 Extend the behavior of the DEFAULT filter selections by applying also items from search ; the sdmx DEFAULT annotation rule is combined with the end-user’s selections from the search result page.

Minor changes:

Bug fixes:

July 11, 2019

Major changes:

Minor changes:

Bug fixes:

June 4, 2019

Major changes:

  • dotstatsuite-data-explorer#23 Time period range selection with annual and non-annual frequencies.
    This new feature of the Data Explorer visualisation page is replacing the time range slider and enables the end-user to select a time frequency and an corresponding range of time period values for a given dataflow view.

    Time Period Selector
  • dotstatsuite-data-explorer#26 Share API and publication workflow.
    The new Share service for table and chart sharing views (including url and embedded codes) relies on a email validation workflow process detailed here.
    The end-user needs to provide a valid email address in order to receive a validation email along with the url and embedded code of the shared table or chart view.

    Share email
  • dotstatsuite-data-explorer#33 LastNObservations default setting allowing to define a default number of latest time series observations per time series returned for table and chart views.
    This is implemented as an sdmx annotation parameter, but it should then be enhanced with a UI feature allowing the end-user to change this default parameter.
  • dotstatsuite-data-explorer#15 .CSV download includes labels/codes option.
    From the search result, .csv downloads include both codes and localised labels of dimensions and dimension members.
    From the visualisation pages, .csv downloads include codes only when “Code” option is set on the UI, and .csv downloads include both codes and localised labels when “Name” or “Both” options are set on the UI.

Minor changes:

Bug fixes:

May 17, 2019

Major changes:

Bug fixes:

April 25, 2019

Major changes:

Minor changes:

Bug fixes:

April 17, 2019

Minor changes:

Bug fixes:

April 11, 2019

Major changes:

  • dotstatsuite-data-lifecycle-manager#37 .Stat DLM list artefacts feature enhancements, including information about data and observation number for a given dataflow, data preview and dataflow categorisation under the same expand/collapse feature.

Minor changes:

Bug fixes:

Release v0.5 - March 2019

Release notes and known limitations in Github

Release v0.4 - February 2019

Release notes and known limitations in Github

Release 14.12.2018

Release notes and known limitations in Github

Release 28.09.2018

Release notes and known limitations in Github

Release 10.07.2018

Release notes and known limitations in Github