Upload referential metadata from a csv file
Introduced in February 21, 2022 Release .Stat Suite JS 12.1.0
Table of Content
Referential metadata can be uploaded when the related SDMX Data Structure Defintion (DSD) has already been uploaded in a given dataspace, and when this DSD contains an annotation of type
METADATA indicating the URN of the related Metadata Structure Definition (MSD), e.g.
<structure:DataStructure agencyID=“XXX“ id=“XXX" version=“1.0"> <common:Annotations> <common:Annotation> <common:AnnotationTitle>urn:sdmx:org.sdmx.infomodel.metadatastructure.MetadataStructure=AGENCY:MSD_ID(1.0)</common:AnnotationTitle> <common:AnnotationType>METADATA</common:AnnotationType> </common:Annotation> </common:Annotations> <common:Name xml:lang="en">XXX</common:Name> <structure:DataStructureComponents> ... </structure:DataStructureComponents> </structure:DataStructure>
As for the DSD, the concepts used in the MSD to name the metadata attributes are previously defined in a Concept Scheme. The concepts and/or the MSD also define the types of the metadata attributes (html, string, number, coded value, etc.).
Here attached is an example of a full SDMX structure file allowing to upload referential metadata, and containing
- a dataflow
- a list of codelists
- a list of concept schemes
- a metadata structure definition (MSD)
- a data strcuture definition (MSD) with the
.Stat Suite uses a specific SDMX-CSV data message to upload (and retrieve) referential metadata values, which is separate from the message for data and attributes. The main reason is that the attachment of referential metadata values to specific combinations of dimension is defined in the data message itself and must be preserved. Note that in contrary to referential metadata, the attachments for normal attributes are defined in the Data Structure Definition (DSD) and that normal attributes are always presented at observation level in SDMX-CSV data messages for observations.
The message for referential metadata values does not allow omitting dimension columns for referential metadata. All dimension columns must be present even if empty. Columns for referential metadata attributes that are not being transmitted can be omitted.
Referential metadata can be uploaded by using the main menu feature “Upload data” available from the top banner of the DLM, and similar to upload data.
The format of the .csv file for referential metadata must comply to the SDMX-CSV 2.0 format, which is as follow:
- the first column: header row containing ‘STRUCTURE’ and each other row containing ‘dataflow’
- the second column: header row containing ‘STRUCTURE_ID’ and each other row containing the full dataflow identification ‘agencyID:dataflowId(version)’ e.g. ‘AGENCY:DF_ID(1.0)’
- the third column (optional): header row containing ‘ACTION’ and each other row containing either ‘I’ for Information or ‘A’ for Append. Actions are however (not yet implemented).
- one column for each dimension of the dataflow: header row containing the dimension IDs and each other row containing the dimension value IDs to which the attribute values of this row are attached. It is left empty in rows where the attribute doesn’t attach to that dimension.
- one column for each submitted referential metadata: header row containing the attribute ID and each other row containing the corresponding attribute values
STRUCTURE,STRUCTURE_ID,ACTION,DIM_1,DIM_2,DIM_3,METADATAATTR_1,METADATAATTR_2 dataflow,AGENCY:DF_ID(1.0.0),I,A,B,,N, dataflow,AGENCY:DF_ID(1.0.0),A,A,B,,Y, dataflow,AGENCY:DF_ID(1.0.0),R,,,2014-01,,”Value X” dataflow,AGENCY:DF_ID(1.0.0),D,,,2014-02,,-
In spreadsheet format:
All textual values containing commas need to be encapsulated within double-quotes. All double-quotes inside these values are to be escaped using doubled double-quotes. In addition, the doubled double-quotes inside localised values using JSON syntax are escaped using a preceeding backslash ‘', e.g.:
"en: ""<a href=\""mailto:firstname.lastname@example.org\"">email@example.com</a>"",fr: ""<a href=\""mailto:firstname.lastname@example.org\"">email@example.com</a>"""
As can be seen in this example, the language codes in localised JSON snippets do not need to be put into (doubled) double quotes.
For more detailed information see the functional specification of the SDMX-CSV version 2.0 format.
For me details about the referential metadata types supported by the data explorer display, see this topic.
Here attached is an example of a .csv file containing referential medata.