Changelog
March 5, 2021
Release .Stat Suite .NET 6.0.0
This major release includes a new version of the core-transfer, core-sdmxri-nsi-ws, core-auth-management, and core-data-access services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.
Disclaimer on the performance evolutions in this release. Here below is a summary of comparison of the performance before and after using the SDMX-RI NSI web service default data plugin in .Stat Suite (all performance tests processes, definitions and types are documented here). Although a series of improvements can already be noticed with this release, it does not yet meet the expected performance improvements for high-load situations (many parallel data queries). Additionally required improvements (related mainly to the default nsi-ws-plugin) are being addressed with dotstatsuite-core-sdmxri-nsi-ws#111.
- Comparison of the performance before and after using the SDMX-RI NSI web service default data plugin
Smoke-test data imports (time increase by 15%)data_import_time................: avg=16.85s --> avg=19.42s ✓ { datasetSize:extraSmall }....: avg=2.36s --> avg=2.6s ✓ { datasetSize:small }.........: avg=31.34s --> avg=36.24
Smoke-test data extractions (time decrease by 20%)
http_req_duration...............: avg=227.65ms --> avg=182.51ms
Load-test data extractions (time increase by 605%)
http_req_duration...............: avg=1.33s --> avg=8.05s ✓ { datasetSize:extraSmall }....: avg=1.6s --> avg=7.07s ✓ { datasetSize:small }.........: avg=1.39s --> avg=8.85s
Stress-test data extractions (time increase by 29%)
http_req_duration................: avg=1.89s --> avg=2.43s ✓ { datasetSize:extraSmall }.....: avg=2.27s --> avg=2.3s ✓ { datasetSize:small }..........: avg=1.99s --> avg=2.63s
Spike-test data extractions (time decrease by 3%)
http_req_duration................: avg=3.8s --> avg=3.68s ✗ { datasetSize:extraSmall }.....: avg=4.25s --> avg=4.11s ✗ { datasetSize:small }..........: avg=3.88s --> avg=3.76s
Soak-test data extractions (time increase by 97%)
http_req_duration..........: avg=670.42ms --> avg=1.32s
major (backward-incompatible) changes:
- dotstatsuite-core-sdmxri-nsi-plugin#37 Always protect non-public data. Whether openid-connect authentication is turned ON or OFF in the NSI web-service configuration, anonymous users ALWAYS only get data that is especially made public (through appropriate permissions).
This represents a major change from previous releases, where with NSI openid-connect authentication was turned OFF, all users (necessarily unauthenticated) could get all data. Now, Docker image with authorization is enabled (turned ON) by default. - Known limitation: the current DLM feature for viewing artefact content (in xml format) in a new web-browser tab by clicking on the (hyperlinked) name of an artefact does not work anymore with the new nsi-ws authentication in place, unless a generic CanReadStructures permission is granted to all users incl. anonymous users. (Updated documentation)
- dotstatsuite-core-transfer#124 Allow for non-numeric and coded measures (observation values) in .Stat CORE. Now, according to the SDMX standard,
String
is the default SDMX data TextType format when it is not specified in the data structure definition (previsouly,Double
was the default TextType format). (Documentation)
minor changes:
- dotstatsuite-core-data-access#16 Data model review of the current codelist-based approach of time dimension. All standard time period formats can now be used, e.g. daily or weekly time periods, without maintaining an internal time period code table.
- dotstatsuite-core-sdmxri-nsi-ws#84 Support for DSD without Time dimension.
- dotstatsuite-core-transfer#159 Allow for attributes at group-level that includes the time dimension (part 1). First partial implementation of this feature, which will be completed once dotstatsuite-core-transfer#189 is released.
- dotstatsuite-core-transfer#167 (DevOps) Validate the data database version used by the transfer-service.
patch changes:
- dotstatsuite-core-data-access#50 (Refactoring) Storage of non-observation attributes at series and observation levels. This improves performance.
- dotstatsuite-core-data-access#57 (Refactoring) Modify the
IObservation
producers to allow SQL bulk insert in a single process. This improves performance. - dotstatsuite-core-data-access#51 (Refactoring) Performance improvement: Remove ROW_ID from DSDs with less than 34 dimensions. This improves performance for data structures with many dimensions.
- dotstatsuite-core-data-access#67 Manage
ExecutionTimeout
with DbUp. - dotstatsuite-core-data-access#69 (DevOps) Run dbup upgrade in a single user mode.
- dotstatsuite-core-auth-management#25 Authorisation management : permissions
ID=0
andID=1
not working as expected. (Updated documentation) - dotstatsuite-core-sdmxri-nsi-ws#90 SdmxSource csv & xml generic readers do not support dataflows without Time dimension.
- dotstatsuite-core-transfer#168 Issues with data upload and data retrieval when
CoreRepresentation
andLocalRepresentation
use different versions of the same codelist. - dotstatsuite-core-transfer#136 Incorrect error when uploading data with attributes (while data is fully ok and also importing same data in slices is ok).
- dotstatsuite-core-transfer#60 Improve error messages for mistakes in the csv layout.
- dotstatsuite-core-sdmxri-nsi-ws#96 Issue with ESTAT:DEMOGRAPHY(2.3).
- dotstatsuite-core-data-access#68 Fix datetime format issue in
UpdateMappingSet
. - dotstatsuite-quality-assurance#1 (DevOps) Migrate performance tests out of the dotstatsuite-core-sdmxri-nsi-plugin repository.
- dotstatsuite-data-lifecycle-manager#156 (Support) Attribute attached to time dimension : Dimension not found in management db.
January 25, 2021
Release .Stat Suite .NET 5.0.0
This major release includes a new version of the core-transfer, core-sdmxri-nsi-ws, core-auth-management, and core-data-access services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.
WARNING: While this first .Stat Suite Core (.NET) version using the SDMX-RI NSI web service default data plugin approach (according to the DB model review step 1) includes the related basic features such as PIT features, authorisations and new REST features, it does not contain yet the necessary performance enhancements foreseen in step 2. This does not represent an issue for data structure definitions that have only a few dimensions and attributes, but those with more dimensions and attributes are now too slow for production mode. These performance enhancements of step 2 are expected to be released very soon. We therefore recommend to not use this release in production mode but to wait for the next .Stat Suite Core (.NET) release, especially if you manage dataflows with many dimensions and attributes.
DISCLAIMER: Starting with this release, the SDMX-RI NSI web service within the .Stat Suite will use the default NSI data plugin instead of the previous .Stat Suite-specific data plugin. This requires the usage of a SDMX-RI MappingSet object (stored in the MappingStore structure database) for each of the dataflows. Except for MappingSets that are manually entered by the user through the SDMX-RI MappingAssistant, all MappingSets must be generated specifically in the context of the .Stat Suite. This is to be done through the following methods:
- Before you generate the MappingSets (see next bullet), if during the migration/upgrade to version 5.0.0 with the DBUP tool some DSDs/Dataflows migrations fail (check the logs using the transfer service
/status/requests
method), then you should migrate these dataflows manually using the Transfer service method/init/dataflow
. Note that his should happen only extremely rarely, and would be caused by previous inconsistencies in the DB state. In case the manual dataflow migration is still unsuccessful then it is recommended to delete the underlying DSD, cleanup the related DB objects (using the Transfer service method/cleanup/dsd
), recreate the data structures and reload the related data.- Generate the MappingSets for all already existing dataflows when the .Stat Suite .NET version is migrated to 5.0.0 using the .Stat Suite Transfer service method
/init/allMappingsets
. This method must be called manually as the very last step of the deployment of the new version (after all components are deployed/updated, and after the DBUP tool has run to update the databases). (Documentation)- Generate the MappingSet for any newly added dataflow using the .Stat Suite Transfer service method
init/dataflow
. This can be done using the Transfer service Swagger UI. (Documentation)- Generate the MappingSet for any newly added dataflow by uploading any data (in DLM or with the .Stat Suite Transfer service). In other words, the MappingSet of a newly added dataflow will be automatically generated once you upload data for this dataflow.
Important NOTE: From this release on, any new/additional attribution of an admin permission (AdminRole) to the authorization service must use the new permission id:4095 (see major change below).
major (backward-incompatibility) changes:
- dotstatsuite-core-sdmxri-nsi-plugin#48 Remove the custom .Stat Suite nsi-plugin (Replace NSI-Plugin).
- dotstatsuite-core-auth-management#20 AdminRole permission id is changed from ‘2047’ to ‘4095’ in auth DB.
- The nsiws.net v8.1.2 included a bug correction SDMXRI-1443 Support of observation values with string data type at SDMX-JSON data writers with a major impact in .Stat Suite. Whenever observation values are of
String
type, then the SDMX-JSON messages generated in SDMX-RI NSI extractions use the (correct)String
type. It also means that string data types are now correctly exposed as string types in the SDMX-RI NSI extractions, and the DE also shows strings as strings without altering concepts such as thousand or decimal separators.
minor changes:
- dotstatsuite-core-sdmxri-nsi-ws#103 Deploy NSI version 8.1.2.
- dotstatsuite-core-sdmxri-nsi-ws#91 Deploy NSI version 8.1.1.
- dotstatsuite-core-sdmxri-nsi-ws#92 Support of allowed content constraints in the NSI WS. (Documentation)
- dotstatsuite-core-transfer#173 Allow listing logs of
init-type
transactions via the Swagger UI of the transfer service. - dotstatsuite-core-sdmxri-nsi-ws#54 Implement the PIT feature in the nsiws (Replace NSI-Plugin). (Documentation)
- dotstatsuite-core-transfer#120 Feature to consult the status of the data imports/transactions.
- dotstatsuite-core-data-access#21 Add “lastNObservations” and “firstNObservations” query parameters. (Documentation)
- dotstatsuite-core-sdmxri-nsi-ws#69 Allow referencing non-final codelists in a non-final hierarchical codelist.
- dotstatsuite-core-sdmxri-nsi-ws#8 Implement observer interface for structure updates.
- dotstatsuite-core-transfer#154 For transfers, check existence of target dataflow and necessary permissions before responding with transaction ID.
- dotstatsuite-core-transfer#152 Ability to transfer data from Live to PIT data version.
- dotstatsuite-kube-core-rp#12 (DevOps) Automate performance tests for Transfer data uploads and for NSI data retrievals.
- dotstatsuite-core-data-access#48 (Refactoring) Automatically create the database views for DSDs and dataflows (Replace NSI-Plugin).
- dotstatsuite-core-data-access#49 (Refactoring) Automatically create mapping sets in the mapping store database (Replace NSI-Plugin).
- dotstatsuite-core-common#108 (Refactoring) Use ESTAT PermissionType (includes one new permission type
id:2048 CanReadPitData
: Can read Point-in-Time (PiT) Data). (Documentation) - dotstatsuite-core-transfer#161 (Refactoring) Fix the data views for Non-mandatory attributes at DSD level.
- dotstatsuite-core-auth-management#23 (Refactoring) Authorization Management & Transfer : update ESTAT NuGet references to nsiws v8.1.1.
- dotstatsuite-core-auth-management#17 (Refactoring) Authorization Management : update ESTAT NuGet references to v8.1.
- keycloak#7 (Refactoring) Disable implicit code flow for Keycloak in the transfer service.
- dotstatsuite-core-transfer#149 (Support) Upload issue of uncoded dataset attributes using the EDD mechanism.
- dotstatsuite-core-sdmxri-nsi-ws#67 (Support) Possibility to delete a code from a non-final codelist.
- dotstatsuite-core-sdmxri-nsi-ws#68 (Support) Issues related to the management of Metadata Structure Definition.
- dotstatsuite-core-data-access#58 (Support) Incorrect Content-Range response header in the ESTAT nsiws.
- dotstatsuite-core-transfer#142 (Support) Update Transfer-service references to NSI version 7.13.0 (incl. 7.12.2).
patches:
- dotstatsuite-core-data-access#65 Add dbup functionality to clear orphan records in Artefact table.
- dotstatsuite-core-data-access#66 Fix of bug in
SqlManagementRepository.CleanUpDsd
deleting extra data tables. - dotstatsuite-core-sdmxri-nsi-ws#17 Deletion of several artefacts at once fails.
- dotstatsuite-core-sdmxri-nsi-ws#75 Annotation update issue & wrong HTTP status code for structure requests.
- dotstatsuite-core-common#110 Validation of the allowed content constraint for coded-attributes in upload requests.
- dotstatsuite-core-sdmxri-nsi-ws#88 Fix bug with select statements having reserved SQL words.
- dotstatsuite-core-sdmxri-nsi-ws#63 Data query returns “Error executing generated SQL and populating SDMX model”.
- dotstatsuite-core-sdmxri-nsi-ws#62 Foreign Key constraint violation exception when creating SDMX artefacts.
- dotstatsuite-core-sdmxri-nsi-ws#82 Mappingsets with “valid to” as max datetime fails in the context of the Point in Time release feature.
- dotstatsuite-core-transfer#165 Transfer fix/allDataflows fails with a Timed-Out error.
- dotstatsuite-core-transfer#163 Re-creation of dataflow view fails for dataflows with no dataset-level attributes.
- dotstatsuite-core-sdmxri-nsi-ws#48 Querying actual constraint or allowed constraint by version fails.
- dotstatsuite-core-data-access#63 Fix issues with Mappingsets initialisation in transfer service.
- dotstatsuite-core-data-access#59 Automatically created mapping sets do not work for dataflows with full ID longer than 30 characters.
- dotstatsuite-core-transfer#166 “Error while trying to create the MappingSet” error in the data transfer e-mail (when data upload is successful).
- dotstatsuite-core-data-access#12 Authenticated / role based restrictions in default NSI plugin for both Live and Point-in-time data.
- dotstatsuite-core-transfer#157 Mappingsets of Point in Time data fails when no release time is provided.
- dotstatsuite-core-transfer#133 Failure in data upload for some dataflows due to auto-generated categories (with same ID but different level).
- dotstatsuite-core-sdmxri-nsi-ws#70 When StartPeriod is ‘March’ in monthly data query, data for January and February are also returned.
- dotstatsuite-core-auth-management#20 Update Admin permission id from ‘2047’ to ‘4095’ in auth DB.
- dotstatsuite-core-transfer#61 Some CSV load behaviours are incorrect and do not respect the SDMX-CSV standard.
- dotstatsuite-core-transfer#113 Free-text attributes with special caracters (?) cut during the SDMX-CSV data upload.
- dotstatsuite-core-transfer#148 Missing rights for dostatwriter db user to create views.
- dotstatsuite-core-transfer#105 SDMX-CSV upload from remote URL is not working.
- dotstatsuite-core-transfer#140 Loading XML data file returns a misleading successful e-mail with 0 observation processed (instead of numerous observations values).
- dotstatsuite-core-transfer#99 Observation values disappear when uploading SDMX-ML data with
dimensionAtObservation=AllDimensions
format.
January 21, 2021
This major release includes a new version of the data-lifecycle-manager, data-explorer, data-viewer, sdmx-faceted-search, and share services.
Tip: we also recommend that you upgrade your config and proxy services, even though no changes were made for those services directly resutling in modifications to the .Stat Suite product.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v8.1.2.
major (backward-incompatibility) changes:
- dotstatsuite-data-explorer#472 DE search result download is now optional (New entry in Javascript settings
search.downloadableDataflowResults
). Since this option is not compatible with the indexation of externally defined/stored dataflows (see related specifications), thus this feature is disabled by default. When upgrading to this release, you must set this new configuration totrue
if you want to keep the download option on the search result page. (Documentation) - dotstatsuite-share#24 New dedicated redis databases for share and sfs services (Documentation). Share and sfs are using a redis service to store objects (
REDIS_HOST
,REDIS_PORT
). Until now, share and sfs were using the same database by default, which is an issue when sfs flushes its data. From now on, share and sfs will have their dedicated databases (REDIS_DB
):- share will use the existing database by default (0) to keep existing data
- sfs will use a new database by default (1) and a re-index will restore previous data
Notes: REDIS_HOST
,REDIS_PORT
andREDIS_DB
are environment variables than can be overriddenREDIS_DB
is an integer (up to 15 without creating new databases in redis)- by default means that without changing anything, restarting sfs and share will apply the update
- dotstatsuite-data-explorer#456 & dotstatsuite-share#12 Translations keys have changed and been updated (here below with default English examples):
- 1 key was deleted
"vx.no.data.available": "There is no data available."
and replaced by 3 new keys:"log.error.sdmx.404": "There is no data available."
,"log.error.sdmx.40x": "You are not permitted to see this data. Please try again after logging in with another account."
, and"log.error.sdmx.xxx": "Whoops, something went wrong on our side. We are working to solve this. Please try again later."
- 2 keys were deleted
"de.app.viewer.owner": "SIS-CC"
,"de.app.viewer.term.label": "Terms & Conditions"
and replaced by 3 new keys:"de.viewer.copyright.label": "©"
,"de.viewer.copyright.content.label": "SIS-CC {link}"
, and"de.viewer.copyright.content.link.label": "Terms & Conditions"
- 2 new keys were added for the data-viewer in order to better handle error notifications:
"data.not.found": "Oops, the view you are looking for doesn't exist. Make sure to use a correct and complete link."
,"data.pending": "Oops, this view isn't accessible yet. Make sure to validate the related email address first."
- 1 key was deleted
significant & minor changes:
- dotstatsuite-data-explorer#456 Update of settings: table/chart footer logo (asset) is localisable. (Documentation)
- dotstatsuite-data-lifecycle-manager#180 DLM new feature for a tree visualisation of related artefacts. (Documentation)
- dotstatsuite-data-lifecycle-manager#163 Display a proper error message when a user does not have the rights to upload data to a dataflow.
- dotstatsuite-data-explorer#454 Define, by configuration, one homepage facet to be expanded by default. (Documentation)
- dotstatsuite-data-explorer#131 DE filters with advanced selection features on scopelists (visualisation page only). (Documentation)
- dotstatsuite-data-explorer#462 DE filters with keyboard selection options on scopelists (visualisation page only). (Documentation)
- dotstatsuite-data-explorer#463 Bulk selections (level 1, level 2, etc.) shown in the ‘Used filters’ area.
- dotstatsuite-data-explorer#285 Implement the new design of the scopelist component.
- dotstatsuite-data-explorer#76 Ease usage and understanding of search filters through hiding impact-less filter options. (Documentation)
- dotstatsuite-data-explorer#458 The ‘used filters’ component shows path(s) in a tooltip.
- dotstatsuite-data-explorer#256 Display better error messages on DE visualisation page when the nsi-ws requests fail. (Documentation)
- dotstatsuite-share#12 Share service to return clear error messages.
- dotstatsuite-data-explorer#453 Enhance DE labels (including slogan).
- dotstatsuite-core-data-access#60 (DevOps) Limit the amount of RAM used by the MSSQL. (Updated documentation on Infrastructure recommendations here)
- dotstatsuite-config#21 (Support) Repair/Improve API queries tab on the DE, by moving ‘Developer API’ to the main menu level & displaying (again) the API documentation link.
- dotstatsuite-data-explorer#456 (Refactoring) Migrate table/chart footer to Visions & apply UI doc/Charte Graphique styles.
- dotstatsuite-d3-charts#3 (Refactoring) Errors in map rendering.
- dotstatsuite-data-lifecycle-manager#175 (Refactoring) Breaking change in nsiws.net v8.1.1 in ErrorMessage format.
patches:
- dotstatsuite-data-explorer#476 GoogleAnalytics fails to return PNG download & shared objects' name.
- dotstatsuite-data-explorer#472 Gix GoogleAnalytics troubleshoot and misc.
- dotstatsuite-data-explorer#468 Decimals in table views are not applied.
- dotstatsuite-data-explorer#464 Default
TimePeriod
andDEFAULT
annotations forLastNObs
are not applied. - dotstatsuite-share#24 Shared objects become invalid after a release update.
- dotstatsuite-data-explorer#471 Constraints from the search result make the visualisation page empty.
- dotstatsuite-data-lifecycle-manager#181 Categories are not always applied on DLM artefact request.
- dotstatsuite-data-lifecycle-manager#164 Translated French labels are not (always) displayed in DLM and in DE when switching locale to French.
- dotstatsuite-data-explorer#477 fix
JS
rendering error. - dotstatsuite-visions#23 Tooltip in the data-header freezes the web-browser for shared views.
December 2, 2020
Patch Release .Stat Suite JS 6.1.1
This patch release includes a new minor version of the data-explorer service.
patches:
- dotstatsuite-data-explorer#457 Fix choropleth map translations.
November 30, 2020
This release includes a new version of the data-explorer, data-viewer, data-lifecycle-manager, sdmx-faceted-search, and share (patch version 6.1.1) services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v7.13.2.
significant (non-major) and minor changes:
- dotstatsuite-data-explorer#320 Make DE authentication optional. (Documentation)
- dotstatsuite-share#4 Cleanup feature for outdated share db objects. (Documentation)
- dotstatsuite-sdmx-faceted-search#38 Hub feature allowing to index externally defined dataflows. (Documentation)
- dotstatsuite-data-explorer#96 Hub feature allowing to visualize and download externally defined dataflows. (Documentation)
- dotstatsuite-data-explorer#433 Shortening the DE browser URL. (Updated documentation)
- dotstatsuite-data-lifecycle-manager#170 (Refactoring) Migrate i18n localised elements of the DLM to the tenantless level (same as DE). (Documentation)
- dotstatsuite-sdmx-faceted-search#84 SDMX Faceted Search supports API Keys. (Documentation)
- dotstatsuite-data-explorer#400 Enhance the default chart’s width field and ease chart size/axis editing options.
- dotstatsuite-data-explorer#434 Quarterly end time periods should always be included in pre-selections.
- dotstatsuite-data-explorer#319 Add
BOM
to support the Excel display of accentuated characters in the CSV exports. - dotstatsuite-data-explorer#432 Remaining chart enhancements based on the original business rules and Charte Graphique.
- dotstatsuite-data-explorer#314 Enhance DE header options for accessibility, localisation and login.
- dotstatsuite-data-viewer#9 Chart responsiveness for small chart sizes where chart title should be replaced by the (i) icon.
- dotstatsuite-data-lifecycle-manager#166 DLM filters reordering and increase of width.
- dotstatsuite-data-explorer#404 Style enhancements of the data table’s row height when including footnotes/flags.
- dotstatsuite-visions#20 Correct the default styles for dataflow title and subtitle.
- dotstatsuite-data-explorer#446 (Refactoring) Remove unused translations.
- dotstatsuite-sdmx-faceted-search#87 (Support) Remove default ILO’s excludedCategorySchemeFacets.
- dotstatsuite-data-explorer#419 (Support) Change to footer text ‘Built by SIS-CC using .Stat Suite’.
- dotstatsuite-config#19 (Support) Complement topic icons and facet configs.
patches:
- dotstatsuite-data-explorer#439 GoogleAnalytics rejects the @ character from the dataflow names or ID. (Updated documentation)
- dotstatsuite-data-explorer#241 Data Explorer does not open in Microsoft Edge.
- dotstatsuite-data-explorer#393 WCAG AA contrast level is incorrect at column header placeholder for footnotes.
- dotstatsuite-data-explorer#408 When downloading csv using the ‘Unfiltered data in tabular format’ option, the
Labels=Identifier
selection is not taken into account. - dotstatsuite-data-lifecycle-manager#159 Data transfer is indicated in DLM as ‘successful’ when it is not necessarily the case.
- dotstatsuite-data-explorer#418 Email share content misses the DE logo.
- dotstatsuite-data-explorer#437 ‘Pinned’ filters are not always correctly ordered.
- dotstatsuite-data-explorer#441 DE full csv download fails (generating wrong content).
- dotstatsuite-data-explorer#443 Broken frequency pre-selection in default views when the frequency is not Annual.
- dotstatsuite-data-explorer#445 Encodage Search issue returns the wrong highlighted information from a free text search.
- dotstatsuite-data-explorer#447 The URL of a data visualisation page is wrongly generated when triggered as new tab from the search result.
- dotstatsuite-data-explorer#450 Period invalid dates and empty dates in some use cases with content constraints.
- dotstatsuite-share#21 ‘Share latest data’ view is generated with broken names (header and footer).
- dotstatsuite-data-lifecycle-manager#174 Data availability option is wrongly set to false when opening a data preview from the DLM.
- dotstatsuite-data-explorer#455 Filter ‘data availability’ occasionally appears in DE.
deprecations:
- dotstatsuite-config#17 Make remaining UI labels translatable by deprecating 2 tenants' configurations. (Updated documentation here and here)
November 24, 2020
Patch Release .Stat Suite .NET 4.4.0
This patch release includes a new version of the core-sdmxri-nsi-ws service. This version resolves a change of default properties in the configuration of the nsiws. This applies ONLY to our custom nsiws plugin (on top of the nsiws.net v7.13.2), and this should be obsolete/by-design in the next versions of both nsiws.net (v8.x) and generic nsi plugin.
Find more about the nsiws.net configurations here.
patches:
- dotstatsuite-core-sdmxri-nsi-ws#72 Set the nsiws configurations
createStubCategory
to “false” andStructureUsage
to “dataflow”. - dotstatsuite-core-sdmxri-nsi-ws#73
structureUsage.structureType
nsiws config. is not applied.
October 26, 2020
This release includes a new major version of the data-explorer, data-viewer, data-lifecycle-manager, and sdmx-faceted-search services.
nsiws compatibility: tested and released in compatibility with the Eurostat nsiws.net v7.13.2
major (backward-incompatibility) changes:
- dotstatsuite-sdmx-faceted-search#83 Indexing of dataflows in sub-categories misses adding the sub-category to the categoryscheme facet.
NB: This results from the introduction in the SDMX-JSON format of a change on thelink-type
property (replaced by amime-type
) that was not parsed by the data-explorer as expected in the previous JS5.4.0 release, causing the disappearance of the indexed categories (loss of the categories in the DE homepage ‘browse by’).
minor changes:
- dotstatsuite-data-explorer#288 Visualisation url to include table layout customisation. (Updated documentation)
- dotstatsuite-data-explorer#226 New design of the DE chart customisations.
- dotstatsuite-data-explorer#386 Chart customisations allowing authenticated users to change the header and footer. (Documentation)
- dotstatsuite-chart-generator-legacy#8 Timeline charts: enhance axis label step configuration. (Documentation)
- dotstatsuite-chart-generator-legacy#6 Improve automated and manual chart time axis scale management.
- dotstatsuite-data-lifecycle-manager#146 Remove dataflow selection in the DLM SDMX data upload feature.
- dotstatsuite-data-explorer#435 Add German locale to .Stat DE.
- dotstatsuite-config#18 (Support) Cleanup obsolete DE i18n overwrites.
- dotstatsuite-data-explorer#218 (Support) Create aliases for staging environments.
- dotstatsuite-data-explorer#396 (Support) Request for BFS tenanted deploy of DE.
- dotstatsuite-data-explorer#423 (Support) Reindex STATEC tenant of DE.
- dotstatsuite-data-lifecycle-manager#165 (Support) DLM (in QA) fails to return information about data from (some) external sources.
patches:
- dotstatsuite-data-explorer#421 Units of Measure: header regression. (Updated documentation)
- dotstatsuite-data-explorer#409 Attributes returned as DE flags are displayed in upper case when the actual ID is in lower case.
- dotstatsuite-data-lifecycle-manager#157 Text attributes truncated to the first colon character.
- dotstatsuite-chart-generator-legacy#5 Repair and improve automated and manual chart axis scale management for time and numeric scales.
October 7, 2020
DISCLAIMER for Release .Stat Suite .NET 4.3.0 and Release .Stat Suite JS 5.4.0
Both releases of the .NET and JS services of the .Stat Suite include backward-incompatible changes due to an update of the underlying SDMX-JSON format to its first official version within the SDMX standard. This update was included in ESTAT NSIWS v7.13.1 (deployed along with v7.13.2 here), and implied changes also in the JS services and applications.
It is therefore absolutely necessary, when upgrading your .Stat Suite instances to any future version, to simultaneously upgrade the .NET and JS services at least to versions .NET 4.3.0 and JS 5.4.0, in order to prevent misbehaviour or functional issues due to a backward-incompatibility of either of these services.
If your deployment topology does not include the .Stat Suite CORE (.NET) services (e.g. when only deploying the Data Explorer and its underlying services), you need either to align your NSIWS version to v7.13.1 minimum (v7.13.2 at best), or use an SDMX web service that is compatible with the official SDMX-JSON release version.
.Stat DLM Excel-Addin: The .NET v4.3.0 release also contains a new version of the .Stat DLM Excel-addin, namely v3.0.0.5, including the various changes and patches mentionned in the list below.
Reminder: The .Stat DLM Excel-addin is an Excel-based application (using the ClickOnce deployment technology) to be installed individually on each client machine. Therefore there is no Docker image produced for this component, and you need to retrieve it directly from its GitLab master branch.
Known limitations and/or deprecating behaviors:
- Inconsistent rendering of DE table views with Unit of Measure information. Some cases described in the functional specifications may not always behave yet as expected. Note that the Unit of Measure feature behaviour can be temporarily ‘switched off’ through the DE configuration settings (by emtpying the
UNIT_MEASURE_CONCEPTS
annotation).- The table/chart views generated from the DE Share “Latest available data for the period defined” option are not always consistent with the original views in the DE browser (e.g. especially in use cases compiling Unit of Measure information).
Release .Stat Suite .NET 4.3.0
This release includes a new version of the core-sdmxri-nsi-plugin and dlm-excel-addin services.
significant (non-major) changes:
- dotstatsuite-core-sdmxri-nsi-ws#66 Deploy NSIWS v7.13.2, including v7.13.1. Non backwards compatible changes/breaking changes have been introduced in the SDMX-JSON message output following the nsiws v7.13.1 release. Details can be found here.
minor changes:
- dotstatsuite-core-sdmxri-nsi-ws#33 (Refactoring) Remove logs per observation in the SDMX-JSON data export.
- dotstatsuite-core-sdmxri-nsi-ws#38 Zip file download, allowing obtaining extractions compressed as zip file format attachment.
- dotstatsuite-docker-compose#4 Review of the Docker Compose installation documentation.
- dotstatsuite-excel-addin#13 (Refactoring) Update keycloak server url in DLM Excel-addin.
patches:
- dotstatsuite-core-sdmxri-nsi-ws#46 Correct the SDMX-JSON message format.
- dotstatsuite-core-sdmxri-nsi-ws#59 Error “Only maintainable are supported by this implementation” when submitting data structures.
- dotstatsuite-data-lifecycle-manager#142 Hierarchical codelist successfully uploaded does not show in the .Stat DLM list.
- dotstatsuite-excel-addin#12 Correct DLM Excel-addin menu label and help menu with link to documentation.
- dotstatsuite-excel-addin#15 Error messages/inappropriate info in DLM Excel-addin.
- dotstatsuite-excel-addin#10 Allowed content constraints are not applied in the DLM Excel-Addin filters of the selector.
This release includes a new version of the data-explorer, data-viewer, sdmx-faceted-search and data-lifecycle-manager services.
significant (non-major) changes:
- dotstatsuite-sdmxjs#4 JS parsers update according to the SDMX-JSON format changes (see related .NET issue dotstatsuite-core-sdmxri-nsi-ws#46).
minor changes:
- dotstatsuite-config#8 and dotstatsuite-data-explorer#342 Ease management of localisations. (Documentation)
- dotstatsuite-data-explorer#354 Configure visibility of DE Share option, allowing to hide the ‘Share’ feature from the end-user. (Documentation)
- dotstatsuite-data-explorer#308 Complement the Unit-of-Measure (UoM) feature with additional use cases (with still knwon limitations, see the disclaimer above). (Documentation)
- dotstatsuite-data-lifecycle-manager#47 Export all structures and data from a DLM dataspace in dump mode.
- dotstatsuite-chart-generator-legacy#7 Show bar/row/stacked-bar labels not only for huge charts.
- dotstatsuite-data-lifecycle-manager#133 Subsequently run transfer cleanup method when deleting a dsd in the DLM.
- dotstatsuite-data-explorer#297 Complement i18n DE for Dutch, Arabic, and Khmer languages.
- dotstatsuite-data-lifecycle-manager#150 (Refactofing) DLM should not send token to external datasources.
- dotstatsuite-data-explorer#394 (Refactoring) Migrate react-intl to latest.
patches:
- dotstatsuite-data-explorer#406 Issues found following the recent ‘SDMX-JSON format changes’.
- dotstatsuite-share#19 Shared ‘Latest Data’ table misses subtitle footnotes and the ‘Unit of Measure’ information is incorrectly displayed.
- dotstatsuite-data-explorer#397 Header information not correctly displayed in DE Excel exports.
- dotstatsuite-data-explorer#413 Fix the ‘used filters’ behavior in the search result page.
- dotstatsuite-data-lifecycle-manager#153 Information about dataflow categorisation(s) is sometimes incorrect in DLM.
September 2, 2020
Release .Stat Suite .NET 4.2.0
This release includes a new minor version of the core-sdmxri-nsi-plugin service and a major 3.0.0.4 version of the dlm-excel-addin.
minor changes:
- dotstatsuite-excel-addin#3 DLM Excel-Addin: save back edited observation and attribute values in a flat or time-series oriented table view. (Documentation)
- dotstatsuite-excel-addin#4 Add a compiled executable (.zip) version of the DLM Excel-Addin, with a self-signed certificate for test and demo purposes. (ready-made file)
- dotstatsuite-excel-addin#6 Add a refresh button to clean-up the cache in the DLM Excel-Addin. (Documentation)
- dotstatsuite-excel-addin#7 DLM Excel-Addin icons.
patches:
- dotstatsuite-core-sdmxri-nsi-ws#65 NSIWS no longer creates log files in the log folder.
- dotstatsuite-excel-addin#8 CPU usage too high when auto-refresh formulae is on.
This patch release includes a new minor version of the data-explorer service.
minor changes:
- dotstatsuite-data-explorer#328 How to customize the DE layout theme with pre-built packages. (Updated documentation and new following sections)
- dotstatsuite-data-explorer#337 Missing label for
datasource
filter.
patch:
- dotstatsuite-data-explorer#388 Units of measure is displayed 2 times in the DE table view (issue resulting from the last JS v.5.3.0 release).
August 25, 2020
This new release includes a new minor version of the data-explorer, data-viewer and data-lifecycle-manager services.
Disclaimer: This new version of the .Stat Suite JavaScript services now uses the Keycloak access_token instead of the id_token. It is therefore required to change the Keycloak configuration accordingly. A good technical procedure is available here.
minor changes:
- dotstatsuite-data-explorer#265 Enhance the attribute display configuration logic. The footnotes parameter is removed from the DE configuration. (Documentation)
- dotstatsuite-data-explorer#327 Make the second-level home facet values selectable. This new feature is enabled by a new configuration parameter (see the related configuration documentation and the full functional specifications of the homepage facets).
- dotstatsuite-data-explorer#307 Adjust the theme and some components to reach the WCAG AA contrast compliancy level.
- dotstatsuite-data-explorer#211 (Refactoring) Refactoring and documentation update necessary for GoogleAnalytics plugged to .Stat DE. (Documentation)
- dotstatsuite-data-explorer#237 Empty observation with and without attributes are shown differently.
- dotstatsuite-data-explorer#278 Update the UI Design of the pinned facets.
- dotstatsuite-data-explorer#302 Remove buttons ripple.
- dotstatsuite-data-explorer #376 Additional style corrections.
- dotstatsuite-data-explorer#383 Enable Data Explorer focus only in accessible mode.
- dotstatsuite-visions#18 (Refactoring) .Stat DE Theme moving into
mixins
what is not in thedefault
theme of material UI. (Documentation) - dotstatsuite-data-explorer#309 (Support) UI improvements following the release JS v5.0.0.
- dotstatsuite-data-explorer#323 (Support) .Stat DE becomes non-responsive when viewing footnotes (in specific cases, after several mouse hovers).
- dotstatsuite-data-explorer#352 (Support) In the .Stat DE viz page, the green filter selection’s count for Time Period should display 0 when no time period is selected.
- dotstatsuite-visions#19 (Support) Flags hover (quick correction).
patches
- dotstatsuite-data-explorer#368 Style issues in the ‘External resources’ link of the ‘Download’ menu.
- dotstatsuite-data-explorer#366 Wrong legend fonts in chart(s).
- dotstatsuite-data-explorer#359 Download chart incompatibility with Firefox and Microsoft EDGE.
- dotstatsuite-data-explorer#358 .Stat DE becomes non-responsive when viewing a Vertical Symbol chart.
- dotstatsuite-data-explorer#252 The DECIMALS setting is not applied correctly: the 0-padding is missing.
July 23, 2020
This new release includes a new version of the data-explorer and data-lifecycle-manager services.
Disclaimer: Support and use of the Proof Key for Code Exchange (PKCE) in both DE & DLM by ensuring .Stat Suite using Keycloak v7+ (PKCE is supported from Keycloak v.7).
If Keycloak does not support pkce (below version 6), then everything will work properly (because pkce from .Stat DE and .stat DLM will be ignored).
When Keycloak is configured to enforce pkce, then non-compliant clients will break.
minor changes:
- dotstatsuite-data-explorer#116 Download charts in .png image format (allowing for high-quality resolution). (Documentation)
- dotstatsuite-data-explorer#326 Add a link to the .Stat DE top-left logo. (Documentation)
- dotstatsuite-data-lifecycle-manager#48 View artifacts per category (topic/domain) in .Stat DLM.
- dotstatsuite-data-lifecycle-manager#141 (Refactoring) DLM refactoring including code base conception cleaning, and
SDMX
parsing tosdmxjs
. - dotstatsuite-data-lifecycle-manager#149 (Refactoring) Remove in .Stat DLM ‘references=all’ from query for category filter when no Category[Scheme] is selected.
- dotstatsuite-visions#16 Blueprint migration & integration.
- .stat-suite/keycloak#6 (DevOps) Update to Keycloak version 7 in qa/staging environments.
- dotstatsuite-data-lifecycle-manager#119 (Confidential)
- dotstatsuite-data-explorer#331 (DevOps) Update pipeline with Docker tag.
- dotstatsuite-config#13 and dotstatsuite-config#12 (Support) Update all i18n dev/prod, including new keys.
- dotstatsuite-config#14 (Support) .Stat DE configuration changes for OECD staging.
patches:
- dotstatsuite-data-explorer#341 Navigation in filter with hierarchy: no root (/parent?) values displayed.
- dotstatsuite-data-explorer#299 When applying DECIMALS=0 on integers, thousands separator wrongly disappears.
- dotstatsuite-data-explorer#351 Crash for a specific dataflow (because it has only one observation).
- dotstatsuite-data-explorer#324 Dataset-level attribute is not displayed next to the title (Dataflow header).
- dotstatsuite-data-explorer#349 Reversed number for right to left languages when using thousand separator.
- dotstatsuite-data-explorer#339 wcag wrongly automatically enabled on .Stat DE viz. page.
- dotstatsuite-data-lifecycle-manager#85 Show the correct status of the artifact deletion in .Stat DLM.
July 7, 2020
Release .Stat Suite .NET 4.1.0
This new release includes a new version of the core-sdmxri-nsi-plugin service, with the release of the Eurostat’s NSI-WS version 7.13.0.
minor changes:
- dotstatsuite-core-sdmxri-nsi-ws#53 Deploy NSI version 7.13.0 (incl. 7.12.2). The task also included the creation, upon Eurostat authorization to the SIS-CC, of a new sub-group called Eurostat SDMX-RI components into GitLab for hosting Read-Only copies (manually mirrored) of some of the Eurostat components:
- dotstatsuite-core-sdmxri-nsi-plugin#8 Load of the Concept Scheme CoreRepresentation.
- dotstatsuite-core-sdmxri-nsi-ws#49 Add the database version number to the NSI-WS health page.
- dotstatsuite-core-sdmxri-nsi-ws#50 (DevOps) Update default MSDB variable and use default settings in
kube
&docker-compose
.
patches:
- dotstatsuite-core-sdmxri-nsi-ws#29 Query for dataflows as references to a particular category wrongly returns also other dataflows. This fix enables to properly filter dataflows in the DLM by category dotstatsuite-data-lifecycle-manager#48.
June 24, 2020
Release .Stat Suite .NET 4.0.0
This new release includes a new major version of the core-transfer service, with an upgrade to the .NET core version 3.1.
It also includes a minor version of the core-auth-management service.
major changes:
- dotstatsuite-core-transfer#93 Upgrade the remaining
.statsuite-core
components to .NET core 3.1.
minor changes:
- dotstatsuite-core-transfer#101 Harmonise the construction of coordinates in user messages.
- dotstatsuite-core-transfer#94 Make the API cleanup feature more robust.
- dotstatsuite-core-transfer#48 Review the text of all current localised transfer error messages.
- dotstatsuite-core-transfer#78 Add API feature to initialise Data DB objects when a dataflow was created. (Documentation)
- dotstatsuite-core-transfer#95 (Refactoring) Remove the feature and usage of external spaces by ID in the transfer service.
- dotstatsuite-core-transfer#100 (Refactoring) Error messages are missing in the
config/localization.json
file. - dotstatsuite-core-transfer#108 (Refactoring) Remove the “See details” feature from the current implementation of the transfer service log messages.
- dotstatsuite-core-transfer#112 (Refactoring) Remove the unit test examples of xml files with no dataflow reference.
- dotstatsuite-core-transfer#117 (Refactoring) Transfer service breaks with latest version of data access nuget.
- dotstatsuite-core-transfer#118 (DevOps) Add database version numbers to the Transfer service healthcheck page.
patches:
- dotstatsuite-core-transfer#111 Dataflow not found in the mapping store.
- dotstatsuite-core-transfer#107 After uploading data for a free-text dataset attribute, the value provided in exports is wrongly wrapped in single quotes.
- dotstatsuite-core-transfer#106 DLM data import with Excel+EDD fails because it expects a mandatory
LanguageCode
parameter. - dotstatsuite-core-transfer#102 Add API feature to cleanup the Data DB from all orphan content (garbage collection). (Documentation)
- dotstatsuite-core-transfer#96 Calling
PITInfo
onPITController
falsely claims that times are UTC. - dotstatsuite-core-transfer#83 No
MIME-Type
validation of file uploads. - dotstatsuite-core-transfer#79 Database failure during first import leaves the data database in inconsistent state.
- dotstatsuite-core-transfer#29 Remove request target dataflow from SDMX import request (because the destination is defined already in the SDMX import file).
- dotstatsuite-core-data-access#47 Add reference to
dotstat-common
dependency. - dotstatsuite-core-auth-management#12 (Confidential)
June 23, 2020
This new release includes a new version of the data-explorer, sdmx-faceted-search and data-lifecycle-manager services.
minor changes:
- dotstatsuite-data-explorer#148 Data Explorer web accessibility support according to WCAG 2.1 level AA and dotstatsuite-data-explorer#209 Web-accessible alternative for table layout drag&drop. (Documentation)
- dotstatsuite-sdmx-faceted-search#30 Restrict search facets to items with data, using to the current data availability. (Documentation)
- dotstatsuite-sdmx-faceted-search#75 Adding (again) hierarchical parents without data to search facets when their children have data - to align with approach in viz page filters, see below).
- dotstatsuite-data-explorer#305 Correct display of hierarchical parents without data to viz page filters when their children have data.
- dotstatsuite-sdmx-faceted-search#73 Exclude dimensions of a dataflow from the indexing that have more values than a new configurable limit. (Documentation is here and here)
- dotstatsuite-sdmx-faceted-search#77 Exclude dataflows from the indexing when its Actual Content Constraint is empty. (Documentation)
- dotstatsuite-data-lifecycle-manager#145 When calling the DE preview in the DLM, use
dataAvailability=on
instead ofoff
. (Documentation)
patches:
- dotstatsuite-data-explorer#312 Clicking on the scopelist parent arrow refreshes the table instead of displaying children items.
- dotstatsuite-data-explorer#270
NAMAIN_FLASH_Q
Dataflow with Q freq. data gets a wrong Time-Period selector set on Annual (not possible to switch to Quarterly). - dotstatsuite-data-explorer#313 DE fullscreen error page appears on click.
- dotstatsuite-data-explorer#276 DE fullscreen feature looses chart customisations.
- dotstatsuite-data-explorer#336 Missing label pagination.
- dotstatsuite-data-explorer#322 Repair: Hide filter with one value.
- dotstatsuite-data-explorer#321 data-explorer link in story crash.
- dotstatsuite-core-sdmxri-nsi-ws#36 In DLM, clarify the text for ‘number of observations’ when data request returns
404
. - dotstatsuite-data-lifecycle-manager#107 In DLM, display the correct structure upload status based uniquely on
NSI HTTP
return code (e.g. “200”, “201”, …) and always display the full status message.
June 15, 2020
Release .Stat Suite .NET 3.9.0
This release includes a new minor version of the core-sdmxri-nsi-ws service, with the deployment of the Eurostat’s NSIWS v7.12.1 that contains the upgrade of the Mappingstore database to v6.9.
Disclaimer: you will need to upgrade to NSIWS v7.12.1 if you want to benefit from all the latest features/fixes in the .Stat Suite context. I you don’t have access to the Eurostat repository, you can check their full release note from here.
Both source-code and Docker images are concerned by this release.
minor changes:
- dotstatsuite-core-sdmxri-nsi-ws#41 Deploy NSI version 7.12.1 in DevOps.
patches:
- dotstatsuite-core-sdmxri-nsi-ws#11 Default GenericData message response is defined with DSD while the data query is for Dataflow.
- dotstatsuite-core-sdmxri-nsi-plugin#19 NSI-Web-Service recognises attributes attached to TIME_PERIOD as observation level attributes.
- dotstatsuite-core-sdmxri-nsi-ws#45
Access-Control-Allow-Origin
header is missing in the 404 data response. - dotstatsuite-core-sdmxri-nsi-plugin#47 Performance issue when downloading/retrieving data.
- dotstatsuite-core-sdmxri-nsi-ws#24 AnnotationTitle is (still) fixed to 70 characters.
May 18, 2020
This new release includes a new major version of the data-explorer, data-viewer, sdmx-faceted-search and data-lifecycle-manager services.
This version does not introduce backward incompatibility, but it brings significant changes in the
css API
and DE design.
WARNING: this release changes and extends how the Data Explorer Theme is customised and will result in your custom styles being lost (if you previously modified the default one). Please see this updated documentation for how to reapply your styles.
Both source-code and Docker images are concerned by this release.
major changes:
- dotstatsuite-data-explorer#263 Adjust some UI design in Data Explorer (Material UI v4 migration).
- dotstatsuite-data-explorer#245 Data-explorer Visions component’s integration (part of MUI v4 migration).
- dotstatsuite-data-explorer#150 Apply new Design to DE (based on usability tests performed across the SIS-CCommunity). See the UI reference documentation.
- dotstatsuite-data-explorer#264 Last N periods feature display should be configurable. This new configuration parameter allows, per datasource, to hide or display the LastNperiod feature of the application (from the filter area of the visualisation page). Documentation
minor changes:
- dotstatsuite-data-explorer#117 DE data views: Limit the displayed filter values according to the current data availability. Documentation
- dotstatsuite-data-explorer#113 Full Unit of measure display in data table views. Documentation
- dotstatsuite-data-explorer#261 DE Image background on March 2020 release. This resulted in improving the customisation method. Documentation
- dotstatsuite-data-explorer#82 Time range selector: current period selected in drop-down but not visible.
- dotstatsuite-visions#17 DE toolbar with icon loader for the download options.
- dotstatsuite-data-explorer#281 Efficient handling of big codelists in DE search results facets and visualisation page filters.
- dotstatsuite-data-explorer#160 Add support of Khmer language in DE.
- dotstatsuite-data-explorer#251 and dotstatsuite-data-explorer#254 Add support of Dutch language in DE and DLM.
- dotstatsuite-data-explorer#75 More clearly indicate the “Filter by” features in the facets of the search and the filters in the visualisation.
- dotstatsuite-data-explorer#259 Healthcheck in DE.
- dotstatsuite-data-explorer#300 Inconsistent behavior of ascending/descending custom option for time period table view.
- dotstatsuite-data-explorer#277 Update labels according to new design, and aligning all supported localised version of the DE.
- dotstatsuite-sdmx-faceted-search#63 Handling facet names in solr containing special characters.
- dotstatsuite-chart-generator-legacy#4 Add time period to data tooltip.
- dotstatsuite-visions#13 (Refactoring) Final migration to Material UI v4.
- dotstatsuite-data-viewer#16 Viewer table updates as part of the Material UI migration.
- dotstatsuite-data-lifecycle-manager#117 (Refactoring) Remove feature and usage of external spaces by ID in transfer service.
- dotstatsuite-data-explorer#273 (Refactoring) Lint Data-explorer.
- dotstatsuite-visions#12 (Refactoring) Backport allow configuring icons associated to facet values into Visions.
- dotstatsuite-docker-compose#2 JS Docker-Compose uses config URLs that will not work.
- dotstatsuite-data-explorer#306 Change the default thousand separator to non-breaking space. Documentation
patches:
- dotstatsuite-data-explorer#246 Filtered CSV download must not be restricted by http range header.
- dotstatsuite-data-explorer#262 No reaction on Excel download.
- dotstatsuite-data-explorer#268 Time Zone: invalid time period boundaries.
- dotstatsuite-data-explorer Full page error on search in filter in table view.
- dotstatsuite-data-explorer#274 Reapply the behaviour of the
DEFAULT
filter selections by also applying items from search (part 1: use of search facet selections). - dotstatsuite-data-explorer#279 Table cell limit warning continues to be shown while refreshing the table even for a different dataflow.
- dotstatsuite-data-explorer#284 Error in DE on Entering Table/Chart View “Whoops something went wrong on our end”.
- dotstatsuite-data-lifecycle-manage#121 DLM filter by
AgencyScheme
or byCategorisation
does not return any result. - dotstatsuite-data-lifecycle-manager#84 Display the correct message in the DLM after a data upload if any issue.
- dotstatsuite-data-explorer#298 Wrong frequency when visiting table view is permanent.
April 20, 2020
Release .Stat Suite .NET 3.5.0
This new release includes a new minor version of the transfer and auth-management services.
Both source-code and Docker images are concerned by this release.
minor changes:
- dotstatsuite-core-transfer#65 Add API feature to cleanup the Data DB when a DSD was deleted.
- dotstatsuite-core-transfer#62 Add source file path/name to notification message content sent to user via e-mail.
- dotstatsuite-core-transfer#97 (Refactoring) Increase Transfer response consistency by applying
lowerCamelCase
across all messages. - dotstatsuite-core-transfer#84 (Refactoring) Actual Content Constraint with
ReleaseVersion=PointInTime
generated during the first data upload. - dotstatsuite-core-auth-management#13 (Confidential)
- dotstatsuite-core-transfer#58 (Documentation) Improved documentation for Restoration service (see in README.md file).
- dotstatsuite-core-common#103 (Documentation) Add a README explaining the purpose of the repository.
patches:
- dotstatsuite-core-transfer#3 Not possibe to upload data after the DSD is re-uploaded with a different structure (this is resolved by implementing dotstatsuite-core-transfer#65).
- dotstatsuite-core-transfer#47 Wrong row id in the data transfer error log.
April 2, 2020
This new release includes a new minor version of the .Stat Suite sdmx-faceted-search service.
Both source-code and Docker images are concerned by this release.
patch:
- dotstatsuite-sdmx-faceted-search#67 Use of
referencepartial
parameter in dataflow queries for indexing.
This fix resolves the issue that caused the .Stat Data Explorer to return all dataflows in all categories.
With this solved, dataflows are only retrieved through the CategoryScheme facet value in which they are categorised.
This feature needs at least nsiws.net v7.11.3 to fully function, and nsiws.net v7.11.0 to not break the request.
March 31, 2020
Release .Stat Suite .NET 3.4.0
This new release includes a new minor version of the .Stat Suite core-transfer and core-data-access services.
Both source-code and Docker images are concerned by this release.
minor changes:
- dotstatsuite-core-transfer#91 (Refactoring) Replace usage of AnnotationText by Title in MAXTEXTATTRIBUTELENGTH annotation. Documentation
- dotstatsuite-core-transfer#92 (Refactoring) Refactor actual constraints generation method.
- dotstatsuite-core-transfer#80 (Refactoring) Update project with the latest data-access nuget package.
- dotstatsuite-core-transfer#77 (Refactoring) Transfer Service should not need write permission on import files.
- dotstatsuite-core-transfer#54 (DevOps) Test docker images created in pipelines.
- dotstatsuite-core-sdmxri-nsi-plugin#42 Race condition in plugin.
- dotstatsuite-core-transfer#82 (Refactoring) (Confidential)
patches:
- dotstatsuite-core-transfer#88 Loading PIT data without having done a live load fails.
- dotstatsuite-core-transfer#86
isInclusive
parameter andEndPeriod
value in generated actual content constraints. - dotstatsuite-core-transfer#85 Generated actual content constraints do not take into account empty observation values with existing attributes.
- dotstatsuite-core-transfer#76 Wrong
TimeRange
in actual content constraint if the TIME_PERIOD is not a last dimension in a DSD. - dotstatsuite-core-transfer#75 Transfer fails and no email notifications are received.
- dotstatsuite-core-transfer#71 PITController does not work with authentication switched on.
March 29, 2020
Release .Stat Suite .NET 3.3.0
This new release includes a new minor version of the .Stat Suite core-sdmxri-nsi-ws and core-sdmxri-nsi-plugin services.
Both source-code and Docker images are concerned by this release.
DISCLAIMER: This release includes an unofficial nsiws.net v7.11.3-hotfix version of NSI webservice, until the resolution of Eurostat’s SDMXRI-1315 issue is made available in the next NSIWS v7.11.5 (see details).
minor changes:
- dotstatsuite-core-sdmxri-nsi-ws#34 Deploy NSI version 7.11.3 (incl. 7.11.2) in DevOps.
- dotstatsuite-core-sdmxri-nsi-ws#28 Implement partial CategorySchemes through detail=referencepartial parameter.
- dotstatsuite-core-sdmxri-nsi-ws#23 ShowPII set to True in NSI OpenId Middleware.
- dotstatsuite-core-sdmxri-nsi-plugin#34 Add commit hash in /health.
- dotstatsuite-core-sdmxri-nsi-ws#32 (Refactoring) Change default logging level to WARN.
- dotstatsuite-core-sdmxri-nsi-ws#25 (Refactoring) Make mapping.store
InsertNewItems
parameter configurable for one specific NSI instance.
patches:
- dotstatsuite-core-sdmxri-nsi-ws#42 Apply hotfix on NSI WS version 7.11.3. See documentation.
- dotstatsuite-core-sdmxri-nsi-ws#30 .Stat DLM filter by types does not always return the expected list of artifacts (Note that an knwon issue on JavaScript side still causes a failure for the DLM end-user: dotstatsuite-data-lifecycle-manager#121).
- dotstatsuite-core-sdmxri-nsi-ws#21 Not possible to update labels/annotations in the artifacts that are referenced by other artifacts.
- dotstatsuite-core-sdmxri-nsi-plugin#45 Data query response wrongly contains time periods that should be excluded when applying Allowed Content Constraint with
TimeRange
.
March 27, 2020
This new release includes a new minor version of the .Stat Suite data-explorer, config and data-lifecycle-manager services.
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.
minor changes:
- dotstatsuite-data-explorer#210 RTL mode xlsx-populate, this allowing to programmatically switching the layout of a downloaded Excel data table between left-to-right and right-to-left.
patches:
- dotstatsuite-data-explorer#240 Broken RTL style table.
- dotstatsuite-data-explorer#255 Empty filters in the .Stat DE visualisation page.
- dotstatsuite-data-lifecycle-manager#123 .Stat DLM Preview icon is displayed even if a DE Url is not provided.
- dotstatsuite-data-lifecycle-manager#124 .Stat DLM unauthenticated requests.
March 05, 2020
This new release includes a new patch version of the .Stat Suite sdmx-faceted-search, config and data-explorer services.
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.
minor changes:
- dotstatsuite-data-explorer#228 Add Data Explorer authentication. This is the first version of the Data Explorer auhtentication feature. The Data Explorer now works in both anonymous and authenticated modes. The same authenticated session (through Keycloak) is used when logging in DLM and DE. See documentation.
patches:
- dotstatsuite-data-explorer#230 Decimals set in the configuration are not applied in the Data Explorer.
- dotstatsuite-data-explorer#243 Repeating dataflows in the Data Explorer search results.
February 28, 2020
Release .Stat Suite JS 4.0.0 with SFS 2.0
This new MAJOR release includes a new version of the .Stat Suite sdmx-faceted-search, config, data-explorer, dotstatsuite-proxy, dotstatsuite-share, dotstatsuite-data-viewer and data-lifecycle-manager services.
The new version of the sdmx-faceted-search service requires nodejs version 13.3.x (mandatory to upgrade when installing/upgrading .Stat Suite from source-code).
This release has been tested and is running with the NSI WebService v7.11.1.
Both source-code and Docker images are concerned by this release.
Important DISCLAIMER: There is a known dependency issue that causes the .Stat Data Explorer to return all dataflows in all categories.
This issue is addressed in dotstatsuite-core-sdmxri-nsi-ws#28 and fix was released by Eurostat in the latest NSI WebService v7.11.2 (issue#34 for its integration to .Stat Suite).
From this release on, incremental semantic version is also used in this changelog for categorising changes and bug fixes:
major: incompatible API change was introduced
minor: functionality was added in a backwards-compatible manner
patch: backwards-compatible bug fixes were applied
major changes:
- dotstatsuite-sdmx-faceted-search#41 (Sdmx Faceted Search) Redefine Document ID and Datasource definitions.
- dotstatsuite-sdmx-faceted-search#47 (Sdmx Faceted Search) Index facets' names instead of ID and remove i18n.
- dotstatsuite-sdmx-faceted-search#48 (Sdmx Faceted Search) All facets are hierarchical.
- See README.md, SFS changelog and config.documentation for additional details on major impacts.
minor changes:
- dotstatsuite-sdmx-faceted-search#44 Highlight facet values in DE search results.
- dotstatsuite-sdmx-faceted-search#42 Create facets for all CategorySchemes of a dataflow. DISCLAIMER: There is a known dependency issue that causes the .Stat Data Explorer to return all dataflows in all categories. This issue is addressed in dotstatsuite-core-sdmxri-nsi-ws#28 and fix was released by Eurostat in the latest NSI WebService v7.11.2 (issue#34 for its integration to .Stat Suite).
- dotstatsuite-sdmx-faceted-search#21 Search terms with space should act like AND operator.
- dotstatsuite-sdmx-faceted-search#37 Allow updating or deleting individual dataflows in the search index. See documentation.
- dotstatsuite-data-explorer#111 Allow configuring icons associated to facet values. See documentation.
- dotstatsuite-data-explorer#153 .Stat DE with the right-to-left (RTL) mode for the table view.
- dotstatsuite-data-explorer#101 Indicate in the table/chart subtitle when displayed data are incomplete. See documentation.
- dotstatsuite-data-explorer#170 In visualisation page, display the dataflow’s title and table/chart footer even when there is no data available.
- dotstatsuite-data-explorer#188 Column widths, alignment and wrapped texts for row/column header cells in the Excel download.
- dotstatsuite-data-explorer#199 Excel download with data including comma separators.
- dotstatsuite-data-explorer#197 Excel download includes the right-to-left (RTL) mode. The downloaded table including Arabic localised data from the .Stat DE is automatically displayed in RTL mode only if the targetted environement (for instance the end-user’s PC) has Windows Excel configured by default with the Right-to-left option (reference).
- dotstatsuite-data-lifecycle-manager#101 Remove the .Stat DLM “upload data” shortcut feature from the list view (not working and confusing).
- dotstatsuite-data-lifecycle-manager#95 Add a logout feature to .Stat DLM. See documentation.
- dotstatsuite-data-lifecycle-manager#80 Improve the text of user instructions on .Stat DLM “upload data” page for Excel option.
- dotstatsuite-sdmx-faceted-search#51 (support) Solr issue with JS milestone 7 and Core 2.1.0 update. this issue led to update Docker-Compose to target
master
versions of the Docker images instead ofdevelop
. - dotstatsuite-sdmx-faceted-search#58 (support) HomefacetId for French locale is not applied.
- dotstatsuite-data-explorer#223 (refactoring) Code quality updated (Pretify src/web).
- dotstatsuite-data-lifecycle-manager#111 (refactoring) .Stat DLM Codebase cleaning.
- dotstatsuite-data-lifecycle-manager#110 (refactoring) When getting the number of observations, only request for the first observation.
- dotstatsuite-data-explorer#206 (refactoring) Frequency and Time Period engine.
- dotstatsuite-data-lifecycle-manager#92 (refactoring) Refactor datasources after sfs2.0 update in all applications (DE+DLM) that require those from the config. service.
- dotstatsuite-sdmx-faceted-search#26 (refactoring) Report is sometimes unreadable and buggy.
- dotstatsuite-documentation#47 (documentation) .Stat DLM installation guidelines from source code need to be updated. See updated documentation including .Stat DLM and Keycloak in all source code approahces.
- dotstatsuite-documentation#19 (documentation) Guidance on Sdmx-Faceted-Search Index Job. See documentation and README.md.
- dotstatsuite-visions#11 (documentation) Alter README to Explain Purpose of Repository.
- dotstatsuite-documentation#50 (documentation) Update copyright date to 2020 in licence file.
- dotstatsuite-data-explorer#203 (documentation) Update README to describe Data Explorer.
- dotstatsuite-data-explorer#212 (test) Add tests for the Excel download feature.
- dotstatsuite-ui-components#8 (test) Add tests for the Scopelist feature.
- dotstatsuite-sdmx-faceted-search#50 (refactoring) (confidential)
- dotstatsuite-share#15 (refactoring) (confidential)
- dotstatsuite-d3-charts#1 (refactoring) (confidential)
patches:
- dotstatsuite-data-explorer#187 Missing behaviours for the data table footnote attributes displayed at the highest possible level.
- dotstatsuite-sdmx-faceted-search#23 Indexing dataflows that are categorised in Categories at lower levels of the CategoryScheme. DISCLAIMER: There is a known dependency issue that causes the .Stat Data Explorer to return all dataflows in all categories. This issue is addressed in dotstatsuite-core-sdmxri-nsi-ws#28 and fix was released by Eurostat in the latest NSI WebService v7.11.2 (issue#34 for its integration to .Stat Suite).
- dotstatsuite-sdmx-faceted-search#27 Duplicates of dataflow in the search index. Resolving this issue now implies:
- One dataflow categorised in two different categories is returned only once in the .Stat DE search result;
- Two dataflows with the same Agency/ID/version but from different sources are both returned in the .Stat DE search result and differentiated with their
sourceID
.
- dotstatsuite-data-explorer#20 Concepts should be merged into one single facet if their localised name are the same.
- dotstatsuite-data-explorer#17 Text search does not return 2 distinct dataflows from 2 different endpoints when they have the same identity (ID, Agency, Version).
- dotstatsuite-data-explorer#59 Free-text search hits within codes of codelists are not returned as highlights in the search results.
- dotstatsuite-sdmx-faceted-search#56 Localisation fails in DE search result page.
- dotstatsuite-data-explorer#167 We lost the defaults for layout (when there are no or only partial related SDMX annotations or selections from search).
- dotstatsuite-data-viewer#4 No horizontal scroll in the Arabic table layout.
- dotstatsuite-data-explorer#222 Right-to-left (RTL) view of the DE table returns a wrong data display mode (when displaying in Arabic language).
- dotstatsuite-data-explorer#21 Search result filtered on hierarchical codelist returns an error in the visualisation page.
- dotstatsuite-data-explorer#235 Search filter selections not applied in visualisation page anymore.
- dotstatsuite-data-explorer#216 Broken hamburger menu of the visualisation page (in narrow mode).
- dotstatsuite-data-explorer#217 Broken choropleth map view in .Stat DE.
- dotstatsuite-data-lifecycle-manager#112 .Stat DLM Avoid unnecessary reload of the application.
- dotstatsuite-data-lifecycle-manager#87 .Stat DLM Missing .csv format in the “custom filter” settings of the SDMX file upload.
February 17, 2020
Release .Stat Suite .NET 3.2.0
This new release includes a new version of the .Stat Suite core-sdmxri-nsi-ws, core-sdmxri-nsi-plugin and core-data-access services.
Both source-code and Docker images are concerned by this release.
Changes:
- dotstatsuite-core-sdmxri-nsi-plugin#36 Deployment of the Eurostat’s NSI web service version 7.11.1 in DevOps (Eurostat’s changelog is included in GitLab’s ticket), now running on .NET Core 3.1.
Bug fixes:
- dotstatsuite-core-sdmxri-nsi-plugin#40 (original ticket: dotstatsuite-core-common#27) Bug in the NSI web service with referencepartial parameter “String or binary data would be truncated”.
- dotstatsuite-core-sdmxri-nsi-plugin#39 Load of Hierarchical Codelist results in timeout.
- dotstatsuite-core-data-access#41 Dataflows with allowed Content Constraints applied return ‘restricted’ data if a user queries it directly.
- dotstatsuite-core-data-access#40 Internal error when invalid dimension code is provided in the query filter.
- dotstatsuite-core-sdmxri-nsi-plugin#32 Data retrieval in .Stat DE fails for some dataflows: “Error during writing response”.
- dotstatsuite-core-sdmxri-nsi-ws#14 Referencepartial parameter for request with references does not work anymore.
- dotstatsuite-core-sdmxri-nsi-ws#13 “Semantic Error - Duplicate language
it
for TextType” when using referencepartial. - dotstatsuite-core-sdmxri-nsi-plugin#28 Calling the Content Constraint with “children” or “descendants” references does not return the related artifacts (for the dataflow that it is constraining).
- dotstatsuite-core-sdmxri-nsi-ws#5 Not possible to delete or update allowed Content Constraint (found in NSI WS 7.8.0.0).
- dotstatsuite-core-sdmxri-nsi-ws#22 Increase the UploadSizeLimit to 50MB for structures in the Transfer service.
- dotstatsuite-core-sdmxri-nsi-ws#7 Mechanism to set the NSI ws URL in returned stub artefacts (stored in full in MASTORE).
February 04, 2020
Release .Stat Suite .NET v3.1.0
This new release includes a new version of the .Stat Suite core-sdmxri-nsi-ws and core-sdmxri-nsi-plugin services.
Both source-code and Docker images are concerned by this release.
Major changes:
- dotstatsuite-core-sdmxri-nsi-ws#15 Allow configurable authenticated and/or anonymous access to a single instance of the NSI service. This is enabled by a new configuration parameter of the authentication
auth.json
with following content:
{
"auth": {
...
"allowAnonymous": true,
...
}
}
The following kind of user rule is from now on required to be configured using the authorisation web service in order to allow public read access to anonymous users (.Stat DE users):
{
"userMask": "*", --> anonymous users
"isGroup": false,
"dataSpace": "staging:SIS-CC-stable", --> data space
"artefactType": 0,
"artefactAgencyId": "*",
"artefactId": "*",
"artefactVersion": "*",
"permission": 3 --> allowed to read structure and data
}
You can check this page of the documentation in order to get the full list of available pemrissions.
- dotstatsuite-core-sdmxri-nsi-plugin#35 Update nsi-plugin for NSI v7.10.10 as intermediate step before 7.11.
January 31, 2020
Release .Stat Suite .NET v3.0.0
Disclaimer: This major version contains breaking changes in the dotstatsuite-core-transfer and dotstatsuite-core-auth-management services with changes to the authentication management (dotstatsuite-core-transfer#66, dotstatsuite-core-transfer#52 & dotstatsuite-core-auth-management#7). All impacted services are: core-transfer, core-auth-management, core-sdmxri-nsi-ws, core-data-access, and core-sdmxri-nsi-plugin.
This release has been tested and is running with the NSI WebService v7.10.8.
Both source-code and Docker images are concerned by this release.
Major changes:
- dotstatsuite-core-transfer#66 Unify the auth config between the transfer service, the authentication service, and the NSI webservice. the Kubernetes strategy is also updated in order to use the authentication in the NSI WS.
- dotstatsuite-core-sdmxri-nsi-ws#19 Deploy the NSI version 7.10.8 in DevOps. Including the following details from the Eurostat changelog:
- Allow admin user to access all mapping store, to avoid having the administrator to logout and login everytime there is a new Mapping Store. OAuth2 caches the user with the mapping store.
- The keyword
latest
at version is not allowed when deleting artefacts. - Fix SOAP 2.1 structure requests with CodeWhere and Stubs.
Minor changes:
- dotstatsuite-core-sdmxri-nsi-plugin#25 (documentation) Enhance documentation of the source code installation for NSI web service and Structure Database.
- dotstatsuite-core-sdmxri-nsi-plugin#22 Suggested in-lined user-defined function in structure database.
Bug fixes:
- dotstatsuite-core-auth-management#7 Authorisation rules do not respect the defined space.
- dotstatsuite-core-transfer#52 Not possible to delete observation values from a DSD with mandatory group/series attributes.
- dotstatsuite-core-sdmxri-nsi-ws#12 Invalid child element
Value
in TimeRange of Actual Content Constraint. - dotstatsuite-core-data-access#31 Actual Content Constraints with TimePeriod constrained cause exception to be thrown from
FillIdsFromDisseminationDb
. - dotstatsuite-core-sdmxri-nsi-plugin#27 Deleting artefact using version of latest (Trying to delete an artefact ‘latest’ version targets v1.0 instead of the actual latest).
- dotstatsuite-core-sdmxri-nsi-plugin#24 Enable authorisation check on data download.
- dotstatsuite-core-sdmxri-nsi-plugin#12 Bug for not supporting multi-datasets in SDMX-JSON data.
January 28, 2020
Release .Stat Suite .NET v2.3.0
Disclaimer: This release contains breaking changes in the dotstatsuite-core-transfer service with a new entry in the
dataspaces.private.json
and the introduction oflocalization.json
via the Dotstat.config nuget package.
Both source-code and Docker images are concerned by this release.
Major changes:
- dotstatsuite-data-lifecycle-manager#50 Download selected data in Excel format (with simple layout).
The original scope of this issue was extended in order to deliver a new component of the .Stat Suite: the .Stat DLM Excel-Addin.
The DLM Excel-Addin is delivered as a ClickOnce application and is deployed using the common procedure for deployment of ClickOnce applications.
You can access the installation files from here. A best practice is to install it on a shared network folder.
The DLM Excel-Addin is made for configuring a single or multiple connection endpoints, in order to retrieve data within Excel tables with simple layout. This first version supports flat, time series down, and time series across layouts.
You can find more information about deployment, set up and configuration here.
Minor changes:
- dotstatsuite-docker-compose#1 (DevOps) Update docker-compose for core services after localization changes.
Bug fixes:
- dotstatsuite-core-transfer#67 ERROR Invalid column name ‘DSD_MAX_TEXT_ATTR_LENGTH’.
- dotstatsuite-core-common#102 Allow uploading data with long textual attribute values for dataflows in first and subsequent uploads.
A configuration parameter is used as a default setting for MaxTextAttributeLength, and its default value is 150.
A new DSD annotation of typeMAXTEXTATTRIBUTELENGTH
can be set by the user for a specific DSD (in its definition).
At first data upload, the default MaxTextAttributeLength is used if the annotation MAXTEXTATTRIBUTELENGTH is not provided.
Whenever a DSD-specific MaxTextAttributeLength is increased after a first data upload, then the new length is applied.
When adding the annotation typeMAXTEXTATTRIBUTELENGTH
to a DSD definition:
<structure:DataStructure id="SPS_ATT_TEST_TFWithNoRelationship_DSD" agencyID="OECD" version="11.1" isFinal="false">
<common:Annotations>
<common:Annotation>
<common:AnnotationType>MAXTEXTATTRIBUTELENGTH</common:AnnotationType>
<common:AnnotationText xml:lang="en">500</common:AnnotationText>
</common:Annotation>
</common:Annotations>
<common:Name xml:lang="en">Attribute test dsd scenario with TIME_FORMAT attribute relationship on dimensions ETA and SESSO</common:Name>
<structure:DataStructureComponents>
Then the textual attribute value can be longer in its display than the default length.
The user is also notified through the email of the transfer action in the DLM:
January 8, 2020
Release .Stat Suite .NET v2.2.0
This new release includes a new version of the .Stat Suite .NET/SQL services.
Both source-code and Docker images are concerned by this release.
Major changes:
- dotstatsuite-core-transfer#21 Improved error message content sent to user via e-mail.
E-mail notifications from the transfer service for data transfer activities and made by .Stat DLM users are received with clearer and well-formulated information.
The following two groupings/categories now describe the issue, its origin and it also identifies responsibility or who may be concerned by the type of issue:
- Type of issues
- System transaction
- Database connection
- Database structure
- User Import/Transfer
- Data Modelling
- Level of issues
- WARNING
- ERROR
More details can be found here (descriptive types of known issues, causes and solutions).
Several tasks remain to be done for providing some of the messages with more comprehensive, non-technical descriptions of the cause, and descriptions of the possible remedy solutions in case of failure.
- dotstatsuite-core-data-access#15 Stress testing of NSI and Transfer for upload and download. See the test report here.
Minor changes:
- dotstatsuite-core-data-access#36 (Test) Create a nuget for shared test classes and unit test sample files.
- dotstatsuite-core-transfer#55 (DevOps) Latest docker image misses the localization.json in a config folder.
- dotstatsuite-core-data-access#33 (Refactoring) Refactor SdmxObservationReader & ObservationValidator.
Bug fixes:
- dotstatsuite-core-transfer#49 Import from URL (of SDMX web service) doesn’t work anymore.
- dotstatsuite-core-transfer#47 Wrong row id in the data transfer error log.
- dotstatsuite-core-data-access#32 SdmxObservationReader writes dimensions into db with wrong order when order of dimensions in serieskey do not match structure.
- dotstatsuite-core-transfer#4 Issues when importing data from SDMX-ML files (Eurostat issue ID SDMXRI-1093).
December 5 2019
Release .Stat Suite .NET v2.1.0
This new release includes a new version of the .NET/SQL services .Stat Suite.
This release has been tested with and is using the NSI WebService v7.10.3.
Both source-code and Docker images are concerned by this release.
Minor changes:
- dotstatsuite-core-common#98 (Refactoring) Common location of localization.json.
- dotstatsuite-core-transfer#43 Enhance logging mechanism with option for stdout.
Bug fixes:
- dotstatsuite-core-transfer#42 Cannot upload data when dataflow is not final.
- dotstatsuite-data-lifecycle-manager#90 “Retrieve the number of observations” is not working correctly.
- dotstatsuite-core-transfer#44 Transfer projects appear to allow old version of Common Nuget.
- dotstatsuite-core-common#60 Generic solution for “An invalid column length from the bcp client for colid” error.
December 02, 2019
Release .Stat Suite JS milestone 7
This new release concerns the JavaScript JS part of .Stat Suite, exclusively the data-explorer application.
Major changes:
- dotstatsuite-data-explorer#171 Excel download with styles and colors applied from the original Data Explorer table view.
- dotstatsuite-data-explorer#128 Additional downloads for external resources can be added at the Dataflow level by using the SDMX annotation type EXT_RESOURCE.
It is possible to relate a specific dataflow, in the SDMX Dataflow structure definition, to one or several materials (any type of file format) that are stored externally (outside .Stat Suite). The feature is supported by the SDMX annotation EXT_RESOURCE type, that is localised and defined at the dataflow level.
In the following example, the dataflow structure definition contains 3 annotations of EXT_RESOURCE type for localised (in English and French) externally stored files to link to the datalfow. Each hyperlink can be enriched with a link to an image for its display as an icon of the link in the final representation:
Once published in the Data Explorer, the external resources related to the dataflow are available from the visualisation page (table and chart views) under the “Download” option, and in addition to the default data download links:
Switching the language of the Data Explorer from English to French will make the localised external files available for download (acccording to the dataflow structure definition).
Bug fixes:
- dotstatsuite-data-explorer#190 Time Period displayed in table view is incorrect (not matching with SDMX response) when time period values are non-ordered.
- dotstatsuite-data-explorer#189 NOT_DISPLAYED annotation is not applied to Attribute when it is defined at the dataflow level.
- dotstatsuite-data-explorer#180 NOT_DISPLAYED annotation is not applied to Dimension when it is defined at the dimension level.
- dotstatsuite-data-explorer#181 NOT_DISPLAYED annotation is not applied to Dimension values when it is defined in the codelist.
- dotstatsuite-data-explorer#178 Frequency missing case (issue found when a dataflow has no Frequency dimension).
- dotstatsuite-data-explorer#179 First default selection is not applied in .Stat DE visualisation page.
- dotstatsuite-data-explorer#184 Hierarchical codelists are displayed as flat facets in the search result page.
- dotstatsuite-data-explorer#154 and dotstatsuite-data-explorer#183 Tooltips in .Stat DE table views are not completely shown in some edge cases.
November 19, 2019
This new release concerns the JavaScript JS part of .Stat Suite, especially the data-explorer, data-viewer and config. applications.
This release has been tested with the NSI WebService v7.10.1 and v7.10.3.
Major changes:
- dotstatsuite-data-explorer#98 Download data in Excel from the .Stat Data Explorer data views.
This first version of the Excel download feature allows the users to download their current filtered and customised data view into an Excel table (.xlsx extension format). It is triggered from the visualisation page’s drop-down “Download” option (in addition to the .csv formats and in both table and chart views).
It includes the layout of the table view, but not yet the colors in the table (to be done with #171).
Attributes in flags and footnotes are also included in the downloaded Excel file, as internal links into a second sheet with exactly the same layout, but without any of the values and with flags in the cells instead of the values. - dotstatsuite-data-explorer#8 Attributes in footnotes can be displayed at the always highest possible level in the .Stat DE data table.
Depending on the scenario and whenever the attribute value is the same in a section of the table, and in order to not repeat an unnecessary same information, then an attribute will be displayed in a footnote at the observation value level, time series level, grouping of dimension members level, or dataflow level. - dotstatsuite-data-explorer#35 The NOT_DISPLAYED SDMX annotation allows to hide dimensions, dimension values, attributes, or attribute values from the .Stat Data Explorer table and chart representations, including from the dimension filters when relevant.
- dotstatsuite-data-explorer#45 Add a UI feature for changing the LastNObservations parameter. See details here and here about the behavior of the LastNObservations feature.
- dotstatsuite-data-explorer#125 Reintroducing the choropleth map chart to .Stat Data Explorer, including writing the instructions for the setup and configuration of a chart map. Documentation is available here.
- dotstatsuite-data-explorer#62 Integrate the Time Period selections into the Used Filters area of the .Stat Data Explorer visualisation pages.
Minor changes:
- dotstatsuite-data-explorer#165 (Refactoring) Modify accept header to still allow retrieving urn in structure requests.
- dotstatsuite-data-explorer#127 Dimension labels on COLUMN is not the same style as the labels on ROW. All dimension labels are now displayed in bold in the table views.
Bug fixes:
- dotstatsuite-data-explorer#168 Bug in the time range filter where frequency, start year and end year are not shown.
- dotstatsuite-data-viewer#7 Viewer Google Analytics undefined.
- dotstatsuite-ui-components#7 Text color in the Frequency & Time Period filters is not coherent with the other components.
- dotstatsuite-data-explorer#162 .Stat Data Explorer CSV download does not include the labels/codes option anymore.
- dotstatsuite-data-explorer#156 Subsequent new filter selections are not taken into account while the data retrieval from the previous selection is still executing.
- dotstatsuite-data-explorer#154 Footnotes are not shown completely especially towards end of table.
November 13, 2019
Release .Stat Suite .NET v2.0.0
This new release includes a new version of the .NET/SQL services the .Stat Suite.
Major changes:
- dotstatsuite-core-common#50 Generic OpenID Connect authentication mechanism in the NSI WebService withint .Stat Suite.
- dotstatsuite-core-sdmxri-nsi-ws#4 and dotstatsuite-core-common#96 Add configuration to authentication to disable SSL and token issuer (required for OpenID Connect #50).
Minor changes:
- dotstatsuite-core-common#46 Refactoring the logging mechanism for the .Stat instances of SDMX-RI web service and Transfer service.
- dotstatsuite-core-sdmxri-nsi-ws#2 DevOps Create a pipeline to build on-demand “vanilla” NSI WS Docker image directly from the Eurostat repository.
- dotstatsuite-core-transfer#34 DevOps Add SMTP server to the Docker image of the Transfer service in order to enable the email mechanism for .Stat DLM.
- dotstatsuite-core-auth-management#4 and dotstatsuite-core-transfer#24 DevOps Build the pipelines and Docker files for the Auth-management and Transfer services.
- dotstatsuite-core-transfer#39, dotstatsuite-core-auth-management#5 and dotstatsuite-core-data-access#29 DevOps Auth-management, Transfer and Data-access services: Update pipelines to push images with “latest” for master branch and “develop” for develop branch.
- dotstatsuite-core-common#74 Test Increase of Unit Test coverage.
- dotstatsuite-core-data-access#27 Test Integration tests with database inited from dbup scripts, in order to increase test coverage to at least 50%.
Bug fixes:
- dotstatsuite-core-transfer#17 InvalidCastException in SqlToSqlTransferManager.
- dotstatsuite-core-transfer#5 Dimension/Group attribute issues depending on the order of series keys reported in SDMX-ML and CSV files.
- dotstatsuite-core-transfer#40 SqlBulkCopy error.
- dotstatsuite-core-data-access#28 Missing rights for service account(s).
- dotstatsuite-core-data-access#19 Log table, Include update scripts.
- dotstatsuite-core-transfer#38 Transfer from external source feature is missing from .Stat DLM.
October 11, 2019
Release .Stat Suite CORE services (GitLab milestone Release BE 1)
Major changes:
- dotstatsuite-core-common#10 Implementation of the Point In Time release (see the PIT methods documentation)
- dotstatsuite-core-data-access#13 Point In Time: configuration option to remove the ability to access PIT data on a per-service basis
- dotstatsuite-core-common#6 Porting of code from .NET Framework to .NET Core
- dotstatsuite-core-transfer#2 Database structure performance improvement
Minor changes:
- dotstatsuite-kube-core-rp#1 DevOps (confidential) Manage ports, logins and passwords
- dotstatsuite-core-common#36 Health check call in Transfer Service
- dotstatsuite-core-transfer#25 DevOps Create docker-compose files for all services (files of two topologies + variables are available here)
Bug fixes:
- dotstatsuite-core-sdmxri-nsi-plugin#16 DotStatRetrieverFactory Constructor does not work with newer NSI Services
- dotstatsuite-core-transfer#12 & dotstatsuite-core-data-access#5 (confidential) Security fixes
- dotstatsuite-core-data-access#25 Moderate Data-Retrieval load causes “Error writing response” problems
October 08, 2019
NSI WS compatible versions: this new release concerns only the JavaScript JS part of the .Stat Suite and is compatible with the NSI WS versions from v7.xx to v7.8.
Major changes:
- dotstatsuite-sdmx-faceted-search#32 Adapt the search service to use the new format of localised elements in SDMX-JSON formats
- dotstatsuite-data-explorer#13 Table view with indented children for hierarchies in rows
- dotstatsuite-data-explorer#9 DataExplorer table preview in the Drag&Drop feature
- dotstatsuite-data-explorer#4 Set a limit of cells in the DataExplorer table view
- dotstatsuite-data-explorer#50 Time Dimension labels in DataExplorer for monthly frequency (see the new configuration rules here)
Minor changes:
- dotstatsuite-data-explorer#151 Add Italian localised language of static DataExplorer elements
- dotstatsuite-data-explorer#109 In DataExplorer visualisation pages, display the dataflow title and table/chart footer already before the data query is executed
- dotstatsuite-visions#3 DataExplorer-splash: migration to Material UI
- dotstatsuite-visions#2 DataExplorer-search-facet: migration to Material UI
- dotstatsuite-visions#1 Import header and footer to [Vision]
- dotstatsuite-data-explorer#124 DataExplorer unit tests
- dotstatsuite-data-explorer#44 DevOps Use “qa” env for “qa” step (dev branch) and “staging” env for “release” step (master branch)
Bug fixes:
- dotstatsuite-data-explorer#147 Source of charts could not be undefined
- dotstatsuite-data-explorer#146 Last N observation should not be undefined
- dotstatsuite-ui-components#6 Table view with a render issue when layout is changed
- dotstatsuite-data-explorer#129 Customise dimensions in rows is not easy nor possible sometimes when dragging and ordering dimensions
- dotstatsuite-data-explorer#123 Arabic visualisation page layout with insufficient RTL features
- dotstatsuite-data-explorer#122 DataExplorer Time Range selector and Share dialog with missing translations
- dotstatsuite-data-explorer#119 Change DataExplorer localised-text-elements
August 20, 2019
Major changes:
- dotstatsuite-core-common#13 and dotstatsuite-data-explorer#67 SDMX-JSON structure and data messages retrieve all localised elements (names, descriptions, texts) according to the latest publicly released SDMX-JSON message format version with the new localised text elements (see sdmx references of both structure and data messages).
Minor changes:
- dotstatsuite-core-common#37 Adding script for the scheduled update of table statistics and rebuilding indexes; the related technical documentation is updated accordingly.
Bug fixes:
- dotstatsuite-core-common#1 .Stat DLM Wrong response when querying for an empty dataflow
August 13, 2019
Major changes:
- dotstatsuite-data-explorer#89 Drag&Drop feature is restricted to minimum of 1 dimension on row. The last dimension set to Row cannot be removed.
- dotstatsuite-data-explorer#71 Extend the behavior of the DEFAULT filter selections by applying also items from search ; the sdmx DEFAULT annotation rule is combined with the end-user’s selections from the search result page.
Minor changes:
- dotstatsuite-data-explorer#81 Time-Range selector missing filter title and navigation bar
- dotstatsuite-core-transfer#9 Performance test scaling resources for Transfer and NSI webservices
- dotstatsuite-data-explorer#88 The number of selected items of the Time-Range selector filter is displayed in the visualisation page.
- dotstatsuite-share#2 Style, content and layout enhancements of the content of the Share email validation request
- dotstatsuite-data-explorer#73 Change “Code” option in “Label format” to “Identifier”
Bug fixes:
- dotstatsuite-core-common#5 Services do not return JSON but string and add a BOM as first character of the body
- dotstatsuite-data-explorer#97 Incorrect count number in the used filter area
- dotstatsuite-core-common#53 Security fixes for Heap Inspection and Improper Resource Shutdown or Release
- dotstatsuite-data-explorer#103 dataquery with several frequencies is not possible (e.g. select Annual AND Monthly from search result and display related dataflow result)
- dotstatsuite-data-explorer#104 LAYOUT_ROW and LAYOUT_COLUMN are not respected for some dataflow cases
- dotstatsuite-data-explorer#78 In the data table view, do not scroll the table header row containing the row dimension labels
- dotstatsuite-data-explorer#92 Dataflow description is not displayed in the search result when using the ‘browse by’ feature from the homepage
- dotstatsuite-data-explorer#19 Not possible to open a visualisation page in a new tab from the search result
- dotstatsuite-data-explorer#106 API queries are incorrect
- dotstatsuite-data-explorer#91 Wrong data retieval queries in visualisation page
- dotstatsuite-data-explorer#107 Filename of CSV download is too long in some cases and fails to open in Excel ; solution is implemented to cut and limit the downloaded filenames to 184 characters max (before the Excel limit of 185 characters).
- dotstatsuite-data-explorer#100 Issue in range header configuration (from the settings.json file)
July 11, 2019
Major changes:
- dotstatsuite-core-common#33 Convert authorization service to .NET Core
- dotstatsuite-data-explorer#48 When displaying footnote popup in data views, also show attribute ID:name (according to the code/label/both option)
- dotstatsuite-core-sdmxri-nsi-plugin#3 SDMX-RI implementation for .Stat Suite
- dotstatsuite-data-explorer#60 Empty “homeFacetIds” should return zero homepage facet.
This configuration (in the settings.json file of the DataExplorer config.) allows to configure an instance of the Data Explorer where the homepage has no facets to browse by, and only a free text search box. - dotstatsuite-data-explorer#9 Table preview in the Drag&Drop feature
Minor changes:
- dotstatsuite-data-explorer#43 Remove “Search in” dropdown feature from the search result header
- dotstatsuite-core-common#13 Change of localised text in the SDMX-JSON data and structures messages (in SDMX source, not yet applied to front-end application such as .Stat DE)
- dotstatsuite-data-explorer#89 Drag&Drop is restricted to a minimum of 1 dimension on row axis
- dotstatsuite-share#3 Security review: Unsafe use of target blank
Bug fixes:
- dotstatsuite-core-common#5 Services do not return JSON but string instead, and add a BOM as first character of the body
- dotstatsuite-data-explorer#80 Time range selector for multiple frequencies: the selected frequency is not apply to data query
- dotstatsuite-data-explorer#79 Bug in the time range selector with Monthly data
- dotstatsuite-data-explorer#81 Time range selector with missing filter title and navigation bar
June 4, 2019
Major changes:
- dotstatsuite-data-explorer#23 Time period range selection with annual and non-annual frequencies.
This new feature of the Data Explorer visualisation page is replacing the time range slider and enables the end-user to select a time frequency and an corresponding range of time period values for a given dataflow view.
- dotstatsuite-data-explorer#26 Share API and publication workflow.
The new Share service for table and chart sharing views (including url and embedded codes) relies on a email validation workflow process detailed here.
The end-user needs to provide a valid email address in order to receive a validation email along with the url and embedded code of the shared table or chart view.
- dotstatsuite-data-explorer#33 LastNObservations default setting allowing to define a default number of latest time series observations per time series returned for table and chart views.
This is implemented as an sdmx annotation parameter, but it should then be enhanced with a UI feature allowing the end-user to change this default parameter. - dotstatsuite-data-explorer#15 .CSV download includes labels/codes option.
From the search result, .csv downloads include both codes and localised labels of dimensions and dimension members.
From the visualisation pages, .csv downloads include codes only when “Code” option is set on the UI, and .csv downloads include both codes and localised labels when “Name” or “Both” options are set on the UI.
Minor changes:
- dotstatsuite-data-explorer#34 Allow sdmx Attributes with 4 letter codes to be displayed as Flags
- dotstatsuite-data-explorer#66Theme SETTINGS. The new configuration file settings.json can include configuration the for Data Explorer layout, search header or splash logo.
If these are added to the settings.json file, then it supersedes the default configuration.
See more details in the FAQ note. - dotstatsuite-data-explorer#38 Chart configs (new configs + chnages in the Rules Driver API)
- dotstatsuite-data-explorer#27 Legacy Fix apiqueries links
- dotstatsuite-data-explorer#25 Legacy Fix visualisation reset state
Bug fixes:
- dotstatsuite-data-explorer#22 Issues and remaining adaptations in the new search, configuration services and architecture
- dotstatsuite-data-explorer#68 Update label in scope list
May 17, 2019
Major changes:
- dotstatsuite-data-explorer#24 Internet Explorer 10 & 11 compatible DataExplorer, with gracefully downgraded compatible features
- dotstatsuite-sdmx-faceted-search#18 Pagination of the search result pages. You can also configure the number of results per pages (see how from here)
Bug fixes:
- dotstatsuite-data-explorer#53 When no data in visualisation page, re-request returns an error
- dotstatsuite-data-explorer#61 Incoherent state in the visualisation breadcrumb
- Various fixes on layout, annotations, charts configs and attributes
April 25, 2019
Major changes:
- dotstatsuite-data-explorer#42 Sdmx Faceted Search (sfs) UI adaptation, including configuration of home facets and reject and/or handle special facets in search results
- dotstatsuite-data-explorer#38 Chart Configs
- dotstatsuite-data-explorer#3 Table layout variations relative to user’s actions
Minor changes:
- dotstatsuite-data-explorer#28 Legacy Sort by order / by start all dimensions values
- dotstatsuite-data-explorer#25 Legacy Fix visualization reset state
- dotstatsuite-data-explorer#49 Setup “ILO” tenant on new front DevOps chain
- dotstatsuite-data-explorer#30 Legacy Get default layout from sdmx data
Bug fixes:
- dotstatsuite-data-explorer#32 The dataflow count is not visible for too long search filter values
- dotstatsuite-data-explorer#27 Legacy Fix apiqueries links
April 17, 2019
Minor changes:
- dotstatsuite-data-explorer#29 Legacy Feature cascading sdmx requests
- dotstatsuite-ui-components#2 scope list search adaptation
Bug fixes:
- dotstatsuite-data-explorer#39 unexpected characters in the DE URL
April 11, 2019
Major changes:
- dotstatsuite-data-lifecycle-manager#37 .Stat DLM list artefacts feature enhancements, including information about data and observation number for a given dataflow, data preview and dataflow categorisation under the same expand/collapse feature.
Minor changes:
- dotstatsuite-data-explorer#1 Update translation files
Bug fixes:
- dotstatsuite-data-explorer#6 Impossible to add dimension to Section when default codes
Release v0.5 - March 2019
Release notes and known limitations in Github
Release v0.4 - February 2019
Release notes and known limitations in Github
Release 14.12.2018
Release notes and known limitations in Github
Release 28.09.2018
Release notes and known limitations in Github
Release 10.07.2018
Release notes and known limitations in Github