Data synchronisation between data spaces
Table of Content
This section describes the implementation of the SDMX
updatedAfter feature for data space synchronisations in the .Stat Suite.
updatedAfterfeature introduced with July 4, 2023 Release .Stat Suite dragonfruit
SDMX updatedAfter feature
This feature covers the use case when the data and referential metadata content of a target data space is to be kept synchronised in an efficient manner with the content of a source data space. To avoid regularly retransferring the complete content, the
updatedAfter feature allows transferring only the content that has been changed (inserted, updated, deleted) since the previous synchronisation execution, and so to minimise the content to be extracted from the source data space and re-uploaded in the target data space. The content transferred with the
updatedAfter feature thus also includes the information about the deleted content in the source data space. This content may be specified differently from the way the data has been first uploaded into the source data space. However, uploading this content on the target data space assures a fully synchronised state.
updatedAfter feature is supported in the SDMX (NSI) web service for content downloads from a source data space and by the Transfer service for the copy of content between data spaces.
The response message of the SDMX (NSI) web service to a query with the
updatedAfter parameter contains one or more SDMX datasets with the actions
Replace. Like for other content downloads, also the content download using the
updatedAfter feature needs separate calls for measure and attribute values and for referential metadata values.
See here the SDMX definition of the
updatedAfter URL parameter in the SDMX restful web service. The parameter takes an xs:dateTime value that should correspond to the last time the synchronisation query was performed by the client. It is strongly recommended to specify the time zone of the client, e.g.,
+01:00, in the
updatedAfter parameter value. If the time zone offset is not specified, then the NSI SDMX web service uses its local time instead.
Developers who update their local databases should make use of the
updatedAfterparameter as it is likely to significantly improve performance. Instead of systematically downloading data that may not have changed, only the consolidated changes to be made in your database since the last time your client performed the same query would be received.
updatedAfterfeature only fully synchronises the content that has been uploaded in the source data space for a specific dataflow, even if other dataflows overlap in their scope. This applies especially for deleted content. In order to assure a full synchronisation between two data spaces for a Data Structure Definition always use a dataflow that has no allowed content constraint. Alternatively, execute the synchronisation for all of the related dataflows.
URL parsers, as the one used in the NSI SDMX web service, may need the time zone prefix
+to be provided in encoded format as
For full details about URL encoding, see the Uniform Resource Locators (URL) Specification at the following location: http://www.rfc-editor.org/rfc/rfc1738.txt