.Stat Suite documentation

Embargo management

Table of Content

This section describes the main features of the .Stat Suite data APIs for the management of embargoed data through a “point in time” (PIT) release.


Point in time release feature

This feature covers the use case when an organisation has to prepare a new revision, but for legal reasons, this data cannot be published before a given date time. The aim is to disseminate updated data at the point in time release date without waiting for a data loading delay. Once the point in time release date is reached, the updated data and attributes are immediately available at once in the disseminate space.

Scope
The PIT only applies to the data values and associated attribute values, it does not modify the Data Structure Definition (DSD). For a given DSD, the data producer/owner submits PIT revisions of the data values and attributes, which are not visible for the data consumers until a given datetime in the future. Thus the PIT feature allows for only one embargo version for the entire DSD, but it can use any dataflow (or the whole DSD) for the API call.

Basic characteristics

  • The Point in Time feature is not based on SDMX features.
  • The PIT is done at DSD level, meaning that all dataflows that are part of a DSD will contain a PIT version, when it is created (even if their data might not be changed).
  • Currently the Data Lifecycle Manager (DLM) does not support the PIT features. It is planned to be added later.
  • There can be DSDs with:
    • only live data, no PIT version
    • both live data and PIT version
    • no live data, only PIT version

Storage

The storage of both the LIVE and PIT versions of the data and attributes is done by using a pair of A and B fact tables (for data and attribute) per DSD. A separate table stores the information to manage the PIT and restoration versions. This table stores the release date, restoration date and the mapping of A/B tables with Live/PIT versions.
Tables involved in the PIT feature:

Data database diagram


Using the API to manage the PIT version

Prepare a (new) PIT version

The Point in Time release options can be directly specified in all the Transfer service data upload/copy functions.

  • The import will allow to set the release date at some point in the future. If not set, then the release date is undetermined and the release is never executed. The release data can be set or changed subsequently.
  • Multiple data imports are allowed for the PIT version, which will modify the PIT data values and/or the release date.
  • When there is an existing PIT version, the Live version of the data cannot be updated.

Currently, the following parameters are supported:

Name Type Description Requirement
targetVersion integer Indicates which version of the tables should be targeted (Live or PIT). In case there is already a PIT release defined on the DSD, all transaction on this DSD will target the PIT release irrespectively to the value of this parameter.
Available values: 0 (Live), 1 (PIT) - Default: 0 (Live)
When target version is LIVE version, PITReleaseDate is set to null and restorationOptionRequired is set to false
Optional
sourceVersion integer Only in /transfer/dataflow method. Indicates which version of the tables should be used as source (Live or PIT).
Available values: 0 (Live), 1 (PIT) - Default: 0 (Live)
Optional
PITReleaseDate string When provided, this date will be the date and time when the PIT release is activated (e.g. is made available for data queries). When not provided, the PIT release will not be activated automatically. If its value refers to a date and time in the past, then the PIT release is immediately activated right at the end of the import/copy transaction.
Format dd-MM-yyyy HH:mm:ss e.g. 17-05-2019 12:30:00
Optional
restorationOptionRequired boolean Optional parameter. Indicates if the current LIVE version should be kept for restoration purposes when the PIT release becomes active. Optional

In case PIT data should be released immediately, then a new import is to be submitted with the release date time as the current date-time or a past date-time.

IMPORTANT NOTE: since September 03, 2021 Release .Stat Suite .NET 7.0.0, for the release date-time, the PITReleaseDate parameter is a datetime value with the time zone designator, according to technical standards (see: https://www.w3.org/TR/NOTE-datetime). This time zone designator allows specifying a precise target release datetime independently from the current time zone of the server where the NSI web service is running.
Example: 1997-07-16T19:20:30.45+01:00.

When transfering data between internal data spaces using the Transfer service /transfer/dataflow method, then the following scenarios are supported through the targetVersion and sourceVersion parameters:

  • Source space –> Target space
    • Live data –> Live data
    • Live data –> PIT data
    • PIT data –> Live data
    • PIT data –> PIT data

However, the PIT release date must always be set explicitly for the target. It is not copied from the source.

Check the current PIT version status

Information about the current Point-In-Time release can be obtained with the Transfer web service method /pointintime​/PITInfo:

  • Indicates if currently there is a PIT release defined on the artefact
  • In case there is a PIT release defined, provides the release date
  • In case there is a PIT release defined, indicates if the restoration option was requested, to keep a copy of the current live data

Example:
POST /1.2/pointintime/PITinfo

More information about the Point-In-Time release status check

Automated release of the PIT version

The release of the PIT version happens at the exact date-time as specified in the relase date-time. Ongoing data retrieval requests are not interrupted by a PIT release event and continue to serve the previous Live data.
There is no physical movement of data as part of release event in order to respect the strict timeliness of the release. Therefor, PIT releases do not affect performance. This is important because at the release date time, there is an expected increase of data requests for the new revision.

The release of a PIT version is fully transparent for data consumers. They only need to reexecute the query. The actual content constraints (data availability) are available for both versions of the data.

At a later point, both PIT and Live versions will support the Last updated and Updated after features.

Rollback and Restoration

To Rollback a still awaiting PIT release that has not yet been released, use the Transfer web service method /pointintime​/rollback. In other words, if a data import request creates a PIT with a release date null (undefined) or some time in the future, then this rollback functionality will allow the user to fully cancel and delete the scheduled PIT.

A rollback will:

  • not affect the current Live version, it will only cancel/delete the scheduled PIT version.
  • only be available if there is currently a PIT version and its release date is null (undefined) or scheduled for some time in the future.
  • not be possible once the PIT release date has passed. For this purpose see the Restore function below.

Example:
POST /1.2/pointintime/rollback

More information about Rollback of Point-In-Time release

To Restore previous Live data to an already released Point-In-Time version, use the Transfer web service method /pointintime​/restoration. A precondition for this feature to be available is to indicate during the PIT data import/transfer using the restorationOptionRequired option to keep a restoration copy of the data. This option is available whenever a PIT is newly defined or updated.
This feature cannot restore the data to any state in the past, but only to one previous version.

Scenarios:

    1. PIT Data import/transfer with restorationOptionRequired set to false:
      After the PIT data has been released, the previous Live data cannot be restored anymore.
    1. PIT Data import/transfer with restorationOptionRequired set to true:
      After the PIT data has been released, the previous Live data can be restored as long as there is no new import for dataflows that are part of the same DSD (once a new import is requested, the restoration copy is lost). This scenario can be used to restore the previous Live data once a PIT has been released and the Rollback is not possible anymore.

Example:
POST /1.2/pointintime/restoration

More information about Restoration of an already released PIT version


Data retrieval

The SDMX (NSI) web service supports the retrieval of the PIT and Live versions of the data. By default, the Live data version is returned.

To request a current non-released PIT version, the following additional http header is to be used the data request:

  • Header name: X-Release
  • Header values: PIT

If this header is not specified or it is empty then the Live data are returned.

When querying Actual content constraints, and if there is a PIT version then 2 constraint versions (each with proper validity dates) are returned: one for the Live version, and one for the PIT version.


PIT access permissions

Only a user with the following access permissions valid for that underlying data can read PIT data versions:

  • User with permissions to manage data (insert/update/delete): DataImporterRole (1315) or their respective underlying basic permissions
  • User with permissions to see PIT data: CanReadPitData (2048)
  • User with full admin permissions: AdminRole (4095)

Only a user with the following access permissions valid for that underlying data can impot/copy PIT data versions:

  • User with permissions to manage data (insert/update/delete): DataImporterRole (1315) or their respective underlying basic permissions
  • User with full admin permissions: AdminRole (4095)

How to fully disable PIT data access
Specific SDMX (NSI) web service instances can be configured to never return any PIT data versions, which disables the PIT header mentioned above.
The configuration of whether access to Point-in-Time data should be allowed, is to be injected into the file /config/dataspaces.private.json as a json node SpacesInternal/AllowPITTargetDataVersion of type boolean. If the value is false, then attempts to retrieve PIT data results in a HTTP 401 error.

Configuration example:

"spacesInternal": [
    {
      "Id": "SDMXwsWithoutPITAccess",
      "AllowPITTargetDataVersion": false
    }
]