.Stat Suite documentation

Source code Windows installation example of .Stat Core services

Table of Content

The following example contains the list of steps required to deploy a specific topology of the dotstatsuite-core components. The configuration of the components has been predefined with default values to ease the installation process. The installation process is based on Git Bash commands as a way to standardize and reduce the installation steps.


Disclaimer

This example SHOULD NOT be used as is for production deployments.

In this installation example all sensitive information is set to use default values, including connection strings, SQL users and passwords, database names, application folders, among others.
This information is publically available, making your deployment vulnerable.


Pre-requisites

Microsoft .NET Core 6.0.* required for source code installation since June 28, 2022 Release

Make sure that the windows machine which will be used in this installation process, has all the following components already installed, and that you have all the required information.

  • IIS Web server

    • IIS server 7.5 or later
    • ASP.Net application roles/features enabled
    • Microsoft Visual C++ 2015 Redistributable download
  • SQL server or MariaDb server

    • SQL server 2017 or higher
      • SQL authentication enabled
      • Named pipes enabled
      • SQL browser service running
      • SQL Server Agent running
      • User and password with sysadmin role
    • MariaDb versions 11 , (note that .Stat Suite has been tested and validated with MariaDb version 11.4.3.)
      • Root user configured
      • Lower case table names enabled (see details here)
  • Microsoft .NET

    • Microsoft .NET Core SDK 6.0.* (x64 version) download
    • Microsoft .NET Core 6.0.* - Windows Server Hosting Bundle download (IIS must be restarted after installation)
    • (Optional) Visual Studio 2022 is to compile solutions with .net core 6.0
  • Git for windows (x64 version) download.

WARNING! - This installation example will fail if any of the pre-requisites is missing.

As a general rule, we suggest to limit the amount of memory allocated to the MSSQL server; A maximum of 75% of the total memory should be allocated to MSSQL.


Topology

.Stat Core topology two spaces


Installation overview

  1. Download the source code
  2. Compile the source code
  3. Initialize the databases
  4. Deploy the Transfer service
  5. Deploy the Design NSI web service
  6. Deploy the Disseminate NSI web service

1 Download the source code

In this section we’ll download the source code for the databases and for each of the applications.

A word on versions

The versions of the source code we download here need to work together. This means that we need to download compatible versions of all the various components, which is why none of the clone statements refer to branches such as “master” or “develop”, but instead refer to tags we know to be compatible.

1 . Open Git Bash with admin rights from the windows start menu

2 . Create a new folder C:/git to store the source code

mkdir /c/git

3 . Move to the new folder

cd /c/git

4 . Clone the dotstatsuite-core-data-access repository.- This repository contains the dotstatsuite-core-dbup tool, which will be used to create and initialize the common and data databases.

git clone -b dragonfruit --single-branch https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-core-data-access.git dotstatsuite-core-dbup

5 . Clone the maapi.net tool repository from the SIS-CC’s mirror of Eurostat repository - This tool will be used to initialize the structure databases.


git clone -b 8.17.0 --single-branch https://gitlab.com/sis-cc/eurostat-sdmx-ri/maapi.net.mirrored.git maapi.net

WARNING! - This repository has a git submodule (authdb.sql) that points to the original ESTAT’s repository in the mirror repository. To change the url of the submodule and to clone it manually (from the SIS-CC’s mirror of Eurostat repository) use the following commands:

cd maapi.net
git submodule add https://gitlab.com/sis-cc/eurostat-sdmx-ri/authdb.sql.mirrored.git authdb
cd ..

6 . Clone the NSI web service repository from the SIS-CC’s mirror of Eurostat repository.

git clone -b 8.17.0 --single-branch https://gitlab.com/sis-cc/eurostat-sdmx-ri/nsiws.net.mirrored.git nsiws.net

7 . Clone the authorization.net repository from the SIS-CC’s mirror of Eurostat repository.- For authorization plugin.

git clone -b 8.17.0 --single-branch https://gitlab.com/sis-cc/eurostat-sdmx-ri/authorization.net.mirrored.git authorization.net

8 . Clone the dotstatsuite-core-transfer repository

git clone -b dragonfruit --single-branch https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-core-transfer.git

2 Compile the source code

1 . Compile the dotstatsuite-core-dbup tool

dotnet publish /c/git/dotstatsuite-core-dbup

2 . Compile the maapi.net tool

dotnet publish /c/git/maapi.net/src/Estat.Sri.Mapping.Tool/Estat.Sri.Mapping.Tool.csproj

3 . Compile the NSI web service

WARNING: The nsi webservice requires a long list of libraries that are downloaded at build time. Sometimes a single build is not enough to download all these libraries, therefore make sure all resources are downloaded at build time, otherwise run the build command multiple times

dotnet build /c/git/nsiws.net/NSIWebServices.sln
dotnet publish /c/git/nsiws.net/NSIWebServices.sln

4 . Compile the authorization plugin

dotnet publish /c/git/authorization.net/src/estat.sri.ws.auth.dotstat/estat.sri.ws.auth.dotstat.csproj

5 . Compile the dotstatsuite-core-transfer

dotnet publish /c/git/dotstatsuite-core-transfer

3 Initialize the databases

For this step you will need the Microsoft SQL sysadmin user and password (server admin when using Azure SQL Database or Azure SQL Managed Instance) or root user and password for MariaDB.

Azure SQL Database pricing tier requirements

Please note that for .Stat data databases (e.g. DesignDataDb and DisseminateDataDb below) on Azure SQL Database the minimum pricing tier required is Standard S3 because of the usage of columnstore indexes.

.Stat DbUp tool creates all databases at Standard S0 pricing on Azure SQL Database and the pricing tiers of data databases are upgraded to Standard S3. Pricing tiers of common and structure databases remain at Standard S0. At later updates DbUp will not change the pricing tier of any of the databases.

If the databases are not created by DbUp tool then the pricing tier of a data database is changed only to Standard S3 if the already existing database is at Basic, Standard S0, Standard S1 or Standard S2 tier. This upgrade of pricing tier is done only at the first run of DbUp tool, at later updates DbUp will not change the pricing tier of any of the databases.

After successful initialization of the databases the pricing tiers may be changed according to the needs, keeping in mind that the minimum required tier is Standard S3 for .Stat data databases.

Important note for Azure SQL

Please ensure the server role loginmanager is attached to user running DbUp. This role provides enough permission level to query sys.sys_login table. For more information, please refer to official documentation.

Initialize the DotStatSuiteCore_Common database

.Stat Core topology Common

Execute the DbUp tool (DotStat.DbUp.dll) with the parameters to create and initialize the DotStatSuiteCore_Common database.

Replace SA_USER and SA_PASSWORD with the Microsoft SQL sysadmin credentials (root user credentials for MariaDb).

Add parameter --useMariaDb to the command when using MariaDb server.

dotnet /c/git/dotstatsuite-core-dbup/DotStat.DbUp/bin/Debug/net6.0/publish/DotStat.DbUp.dll upgrade --connectionString "Server=localhost;Database=CommonDb;User=SA_USER;Password=SA_PASSWORD;" --commonDb --loginName testLoginCommon --loginPwd "testLogin(\!)Password" --force

Initialize one Design DotStatSuiteCore_Data database

.Stat Core topology designData

Execute the Dbup tool (DotStat.DbUp.dll) with the parameters to create and initialize Design DotStatSuiteCore_Data database.

Replace SA_USER and SA_PASSWORD with the Microsoft SQL sysadmin credentials (root user credentials for MariaDb).

Add parameter --useMariaDb to the command when using MariaDb server.

dotnet /c/git/dotstatsuite-core-dbup/DotStat.DbUp/bin/Debug/net6.0/publish/DotStat.DbUp.dll upgrade --connectionString "Server=localhost;Database=DesignDataDb;User=SA_USER;Password=SA_PASSWORD;" --dataDb --loginName testLoginDesignData --loginPwd "testLogin(\!)Password" --force

Initialize one Disseminate DotStatSuiteCore_Data database

.Stat Core topology disseminateData

Execute the DbUp tool (DotStat.DbUp.dll) with the parameters to create and initialize the Disseminate DotStatSuiteCore_Data database.

Replace SA_USER and SA_PASSWORD with the Microsoft SQL sysadmin credentials (root user credentials for MariaDb).

Add parameter --useMariaDb to the command when using MariaDb server.

dotnet /c/git/dotstatsuite-core-dbup/DotStat.DbUp/bin/Debug/net6.0/publish/DotStat.DbUp.dll upgrade --connectionString "Server=localhost;Database=DisseminateDataDb;User=SA_USER;Password=SA_PASSWORD;" --dataDb  --loginName testLoginDisseminateData --loginPwd "testLogin(\!)Password" --force

Configure the maapi.net tool

The maapi.net tool is used to initialize DotStatSuiteCore_Struct (mappingStore) databases. In this example we will use it to initialize two DotStatSuiteCore_Struct databases (design and disseminate).

The tool requires a pre-configured list of connection strings of each of the databases that will be created and/or updated. This information should be added to the Estat.Sri.Mapping.Tool.dll.config file, under the section “<connectionStrings>”.

...
  <!--app.config -->
  <connectionStrings>
    <clear/>
    <add name="sqlserver" connectionString="Data Source=sodi-test;Initial Catalog=msdb_scratch;User Id=mauser;Password=123"
        providerName="System.Data.SqlClient"/>
    <add name="odp" connectionString="Data Source=sodi-test/xe;User ID=msdb_scratch;Password=123" providerName="Oracle.ManagedDataAccess.Client"/>
    <add name="mysql" connectionString="server=sodi-test;user id=mauser;password=123;database=msdb_scratch;default command timeout=120"
        providerName="MySql.Data.MySqlClient"/>
  </connectionStrings>
...

To simplify the process, we will use the configuration example maapi-app.config file from dotstatsuite-core-sdmxri-nsi-ws repository.

  • Move to the maapi.net folder
cd /c/git/maapi.net/src/Estat.Sri.Mapping.Tool/bin/Debug/net6.0/publish/
  • From the dotstatsuite-core-sdmxri-nsi-ws repository, download the example configuration file maapi-app.config to the Estat.Sri.Mapping.Tool.dll.config
curl https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-core-sdmxri-nsi-ws/-/raw/master/docs/installation/config-examples/maapi-app.config?inline=false >Estat.Sri.Mapping.Tool.dll.config

The example file is set up for SQL Server, to configure MariaDb you will need to:

  • Replace the connection string with your MariaDb connection string
  • Change the provider to MySqlClient
...
  <!--app.config -->
  <connectionStrings>
    <clear/>
    <add name="DesignStructDb" connectionString="server=localhost;user id=mauser;password=123;database=DesignStructDb"
        providerName="MySql.Data.MySqlClient"/>
    <add name="DisseminateStructDb" connectionString="Server=localhost;User=mauser;Password=123;Database=DisseminateStructDb"
providerName="MySql.Data.MySqlClient"/>
  </connectionStrings>
...

Initialize one Design DotStatSuiteCore_Struct database (MappingStore db)

.Stat Core topology designStruct

The initialization of MappingStore databases is done in two steps, first the dotstatsuite-core-dbup tool is used to create an empty database with the user and its credentials, and finally the maapi.net tool will generate all the remaining database artifacts.

Step 1. Create the empty database using the dotstatsuite-core-dbup tool.

Execute the Dbup tool (DotStat.DbUp.dll) with the parameters to create and initialize the Disseminate DotStatSuiteCore_Struct database.

Replace SA_USER and SA_PASSWORD with the Microsoft SQL sysadmin credentials (root user credentials for MariaDb).

Add parameter --useMariaDb to the command when using MariaDb server.

dotnet /c/git/dotstatsuite-core-dbup/DotStat.DbUp/bin/Debug/net6.0/publish/DotStat.DbUp.dll upgrade --connectionString "Server=localhost;Database=DesignStructDb;User=SA_USER;Password=SA_PASSWORD;" --mappingStoreDb --loginName testLoginDesignStruct --loginPwd "testLogin(\!)Password" --force

Step 2. Initialize the database using the maapi.net tool

dotnet Estat.Sri.Mapping.Tool.dll init -m DesignStructDb -f 

Initialize one Disseminate DotStatSuiteCore_Struct database (MappingStore db)

.Stat Core topology disseminateStruct

The initialization of MappingStore databases is done in two steps, first the dotstatsuite-core-dbup tool is used to create an empty database with the user and its credentials, and finally the maapi.net tool will generate all the remaining database artifacts.

Step 1. Create the empty database using the dotstatsuite-core-dbup tool.

Execute the Dbup tool (DotStat.DbUp.dll) with the parameters to create and initialize the Disseminate DotStatSuiteCore_Struct database.

Replace SA_USER and SA_PASSWORD with the Microsoft SQL sysadmin credentials (root user credentials for MariaDb).

Add parameter --useMariaDb to the command when using MariaDb server.

dotnet /c/git/dotstatsuite-core-dbup/DotStat.DbUp/bin/Debug/net6.0/publish/DotStat.DbUp.dll upgrade --connectionString "Server=localhost;Database=DisseminateStructDb;User=SA_USER;Password=SA_PASSWORD;" --mappingStoreDb --loginName testLoginDisseminateStruct --loginPwd "testLogin(\!)Password" --force

Step 2. Initialize the database using the maapi.net tool

dotnet Estat.Sri.Mapping.Tool.dll init -m DisseminateStructDb -f 

4 Deploy the Transfer service

.Stat Core topology transfer

Step 1. Create a new folder to create the web service

mkdir -p /c/dotstatsuite-website/transfer-service/

Step 2. Add full access rights to the users “IUSR” and “IIS_IUSRS” to the folder

icacls "C:\dotstatsuite-website\transfer-service" /grant:r "IUSR":"(OI)(CI)F"
icacls "C:\dotstatsuite-website\transfer-service" /grant:r "IIS_IUSRS":"(OI)(CI)F"

Step 3. Copy the compiled binaries to the new folder

cp -r /c/git/dotstatsuite-core-transfer/DotStatServices.Transfer/bin/Debug/net6.0/publish/* /c/dotstatsuite-website/transfer-service/

Step 4. Create a new IIS application called transfer-service in port 93, using appcmd command

Make sure git bash is running in admin mode.

/c/Windows/System32/inetsrv/appcmd add site /name:transfer-service /physicalPath:C:\\dotstatsuite-website\\transfer-service /bindings:http/*:93:

Step 5. Create a new IIS application pool (TransferServiceAppPool)

/c/Windows/System32/inetsrv/appcmd add apppool /name:TransferServiceAppPool /managedRuntimeVersion:"" /managedPipelineMode:Integrated
  • Add the transfer-service application to the newly created application pool
/c/Windows/System32/inetsrv/appcmd set app "transfer-service/" /applicationPool:TransferServiceAppPool

Step 6. Configure the transfer service

There are two options to configure the transfer service:

  1. Json config files, examples are based on SQL Server.- NOT RECOMMENDED

Copy the sample file dataspaces.private.json from C:\git\dotstatsuite-core-transfer\docs\installation\config-examples to the deployment folder (C:\dotstatsuite-website\transfer-service\config).

  1. Saving the configuration setting as environment variables for the IIS site (transfer-service). Recommended

See more about the configuration settings.

For this example we will use the second option:

  • Set the common database connection string:
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='DotStatSuiteCoreCommonDbConnectionString',value='Server=localhost;Database=CommonDb;User=testLoginCommon;Password=testLogin(\!)Password']" /commit:apphost
  • Set the common database type, supported values SqlServer(default) and MariaDb:
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='DbType',value='SqlServer']" /commit:apphost
  • Set the default language code:
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='DefaultLanguageCode',value='en']" /commit:apphost
  • Disable authentication:
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__enabled',value='false']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__claimsMapping__email',value='null']" /commit:apphost

NOTE that the transfer service will not send emails (for actions performed on data from the .Stat DLM) until authentication is enabled and properly configured. To enable authentication in the transfer-service, see .Stat authentication configuration.

  • Set the design dataspace values, make sure to change DbType value when using MariaDb server (supported values SqlServer(default) and MariaDb):
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__0__Id',value='design']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__0__DotStatSuiteCoreStructDbConnectionString',value='Server=localhost;Database=DesignStructDb;User=testLoginDesignStruct;Password=testLogin(\!)Password']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__0__DotStatSuiteCoreDataDbConnectionString',value='Server=localhost;Database=DesignDataDb;User=testLoginDesignData;Password=testLogin(\!)Password']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__0__DataImportTimeOutInMinutes',value='60']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__0__DatabaseCommandTimeoutInSec',value='360']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__0__AutoLog2DB',value='true']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__0__DbType',value='SqlServer']" /commit:apphost
  • Set the disseminate dataspace values, make sure to change DbType value when using MariaDb server (supported values SqlServer(default) and MariaDb):
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__1__Id',value='disseminate']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__1__DotStatSuiteCoreStructDbConnectionString',value='Server=localhost;Database=DisseminateStructDb;User=testLoginDisseminateStruct;Password=testLogin(\!)Password']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__1__DotStatSuiteCoreDataDbConnectionString',value='Server=localhost;Database=DisseminateDataDb;User=testLoginDisseminateData;Password=testLogin(\!)Password']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__1__DataImportTimeOutInMinutes',value='60']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__1__DatabaseCommandTimeoutInSec',value='360']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__1__AutoLog2DB',value='true']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "transfer-service" -section:system.webServer/aspNetCore /+"environmentVariables.[name='spacesInternal__1__DbType',value='SqlServer']" /commit:apphost

NOTE AutoLog2DB is enabled for both dataspaces. This configuration allows the transfer service to store logs in the database. When this setting is enabled, the functions /status/request and /status/requests can be used to retrieve logs.

Step 7. Start the new application

/c/Windows/System32/inetsrv/appcmd start site /site.name:transfer-service

Step 8. Test that the application is up and running

  • Using curl
curl localhost:93/health
  • Using a web browser
    Open a web browser and open the url localhost:93/health

You should see similar result:

{
	"service": {
		"status": "Healthy",
		"details": {
			"version": "12.0.0+83c10b048d",
			"auth_enabled": true,
			"tz": "(UTC) Coordinated Universal Time",
			"time": "10.07.2023 07:55:47"
		},
		"responseTime": 19.9126
	},
	"database": {
		"status": "Healthy",
		"details": {
			"design": {
				"structureDb": {
					"version": "6.20",
					"logs": "72.00 MB",
					"data": "72.00 MB"
				},
				"dataDb": {
					"version": "7.4",
					"logs": "8.00 MB",
					"data": "72.00 MB"
				},
				"supportedDbVersion": "7.4",
				"isDataDbVersionCompatible": true
			},
			"disseminate": {
				"structureDb": {
					"version": "6.20",
					"logs": "72.00 MB",
					"data": "72.00 MB"
				},
				"dataDb": {
					"version": "7.4",
					"logs": "8.00 MB",
					"data": "72.00 MB"
				},
				"supportedDbVersion": "7.4",
				"isDataDbVersionCompatible": true
			}
		},
		"responseTime": 232.1524
	},
	"memory": {
		"status": "Healthy",
		"details": {
			"allocatedMb": 16.4213,
			"gen0Collections": 0,
			"gen1Collections": 0,
			"gen2Collections": 0
		},
		"responseTime": 3.6066
	},
	"disk": {
		"status": "Healthy",
		"details": {
			"tempPath": "/tmp/TransferService",
			"totalSize": "250.92 GB",
			"totalFreeSpace": "231.25 GB",
			"availableFreeSpace": "218.44 GB",
			"free": "87.1%",
			"filesCount": 0
		},
		"responseTime": 6.8963
	},
	"queue": {
		"status": "Healthy",
		"details": {
			"uptime": "00:00:55.6645698",
			"tasksInQueue": 0,
			"runningTasks": 0,
			"takenTasks": 0
		},
		"responseTime": 3.6176
	},
	"totalResponseTime": 237.3245
}

Note: By default all the logs will be stored at C:\dotstatsuite-website\transfer-service\logs\


5 Deploy the Design NSI web service in port 81

.Stat Core topology nsiwsDesign

Step 1. Create a new folder to create the web service

mkdir -p /c/dotstatsuite-website/nsiws-design/

Step 2. Add full access rights to the users “IUSR” and “IIS_IUSRS” to the folder

icacls "C:\dotstatsuite-website\nsiws-design" /grant:r "IUSR":"(OI)(CI)F"
icacls "C:\dotstatsuite-website\nsiws-design" /grant:r "IIS_IUSRS":"(OI)(CI)F"

Step 3. Copy the compiled binaries to the new folder

cp -r /c/git/nsiws.net/src/NSIWebServiceCore/bin/Debug/net6.0/publish/* /c/dotstatsuite-website/nsiws-design/

Step 4. Copy the following binaries from authorization.net to the Plugins folder. This will allow the NSI web service to retrieve authorization rules from .Stat common database when the authorization is enabled.

  • DotStat.Common.dll
  • DotStat.DB.dll
  • DotStat.Domain.dll
  • DotStat.MappingStore.dll
  • estat.sri.ws.auth.dotstat.dll
  • estat.sri.ws.auth.dotstat.deps.json
cp -r /c/git/authorization.net/src/estat.sri.ws.auth.dotstat/bin/Debug/net6.0/publish/{DotStat.Common.dll,DotStat.DB.dll,DotStat.Domain.dll,DotStat.MappingStore.dll,estat.sri.ws.auth.dotstat.dll,estat.sri.ws.auth.dotstat.deps.json} /c/dotstatsuite-website/nsiws-design/Plugins

Step 5. Configure the nsi web service From the dotstatsuite-core-sdmxri-nsi-ws repository, download the following sample configuration files to the deployment folder:

This sample configuration file has been set to use the SQL Server databases and users that were previously created in this guide. See more about the configuration settings.

curl https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-core-sdmxri-nsi-ws/-/raw/master/docs/installation/config-examples/nsiws-design-app.config?inline=false >/c/dotstatsuite-website/nsiws-design/config/app.config
  • When using MariaDb
  • Change connection string to match your MariaDb conenction string to the database created previously
  • Change providerName from System.Data.SqlClient to MySql.Data.MySqlClient
  • NEW - Set data database connection string Modify the configuration file /config/Properties.json - Add the value for the disseminationDbConnection connectionString:

    • Change From:
  "disseminationDbConnection": {
    "dbType": "SqlServer",
    "connectionString": ""
  }
  • To
  "disseminationDbConnection": {
    "dbType": "SqlServer",
    "connectionString": "Server=localhost;Database=DesignDataDb;User=testLoginDesignData;Password=testLogin(\!)Password"
  }
  • When using MariaDb change dbType to:
  "disseminationDbConnection": {
    "dbType": "MySql",
    "connectionString": "Server=localhost;Database=DesignDataDb;User=testLoginDesignData;Password=testLogin(\!)Password"
  }

Logging configuration.- By default, the service has been configured to log all activity. This causes performance issues during data extractions. To avoid performance issues on production envirements, please make the following changes to the file /config/log4net.config

In line 80 modify the xml block FROM:

...
  <root>
    <level value="ALL"/>
    <appender-ref ref="SystemActivityAppender"/>
    <appender-ref ref="UserActivityAppender"/>
  </root>
...

TO:

...
  <root>
    <level value="WARN"/>
    <appender-ref ref="SystemActivityAppender"/>
    <appender-ref ref="UserActivityAppender"/>
  </root>
...

Step 6. Create a new IIS application called nsiws-design in port 81, using appcmd command

Make sure git bash is running in admin mode.

/c/Windows/System32/inetsrv/appcmd add site /name:nsiws-design /physicalPath:C:\\dotstatsuite-website\\nsiws-design /bindings:http/*:81:

Step 6. Create a new IIS application pool (NSIWSDesignAppPool)

/c/Windows/System32/inetsrv/appcmd add apppool /name:NSIWSDesignAppPool /managedRuntimeVersion:"" /managedPipelineMode:Integrated
  • Add the nsiws-design application to the newly created application pool
/c/Windows/System32/inetsrv/appcmd set app "nsiws-design/" /applicationPool:NSIWSDesignAppPool

Step 8. Configure the authentication and the authorization plugin for retrieval of .Stat authorization rules

There are two options to configure the authorization plugin:
1 . Json config file.- NOT RECOMMENDED

Download the sample files (based on SQL server) nsiws-authentication.json and nsiws-design-authorization.json from dotstatsuite-core-sdmxri-nsi-ws repository to the deployment folder (C:\dotstatsuite-website\nsiws-design\config).

2 . Saving the configuration setting as environment variables for the IIS site (nsiws-design). Recommended

See more about the configuration settings.

For this example we will use the second option:

  • Set authentication turned OFF
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__enabled',value='false']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__allowAnonymous',value='true']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__requireHttps',value='false']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__validateIssuer',value='false']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__showPii',value='false']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__clientId',value='stat-suite']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__authority',value='']" /commit:apphost
  • Set authorization turned ON. When using MariaDb make sure to set DbType value to MariaDb.
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='authorization__enabled',value='true']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='authorization__method',value='dotstat']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='authorization__PrincipalFrom',value='context']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='DotStatSuiteCoreCommonDbConnectionString',value='Server=localhost;Database=CommonDb;User=testLoginCommon;Password=testLogin(\!)Password']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='DbType',value='SqlServer']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='mappingStore__Id__Default',value='design']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-design" -section:system.webServer/aspNetCore /+"environmentVariables.[name='mappingStore__Id__FromQueryParameter',value='Never']" /commit:apphost

Please note that with the configuration above the NSI WS instance is configured to allow only anonymous access (as authentication is turned off). The authorization rules can be managed via Authorization Management service (e.g. using the Swagger UI of the service). Permissions granted to anonymous user is controlled by rules assigned to all users (userId="*").

Step 9. Start the new application

/c/Windows/System32/inetsrv/appcmd start site /site.name:nsiws-design

Step 10. Test that the application is up and running

  • Using curl
curl localhost:81/health
  • Using a web browser
    Open a web browser and open the url localhost:81/health

You should see similar result:

{
  "service": {
    "status": "Healthy",
    "details": {
      "version": "8.17.0",
      "retriever": "MappingStoreRetrieversFactory",
      "retriever-version": "8.17.0",
      "middleware": "CorsMiddlewareBuilder,OpenIdMiddlewareBuilder,LoggingOptionsBuilder,UserAuthorizationRulesMiddlerwareBuilder",
      "maxRequestBodySize": 52428800
    },
    "responseTime": 0.3649
  },
  "db": {
    "status": "Healthy",
    "details": {
      "storeId": "design",
      "version": "6.20",
      "isLatest": true
    },
    "responseTime": 4.1766
  },
  "memory": {
    "status": "Healthy",
    "details": {
      "allocatedMb": 75.5947,
      "gen0Collections": 22351,
      "gen1Collections": 4900,
      "gen2Collections": 1480
    },
    "responseTime": 0.0096
  },
  "disseminationDb": {
    "status": "Healthy",
    "details": {},
    "responseTime": 0.0442
  },
  "totalResponseTime": 5.405
}

Note: By default all the logs will be stored at C:/ProgramData/Eurostat/logs/


6 Deploy the Disseminate NSI web service in port 80

.Stat Core topology nsiwsDisseminate

Step 1. Create a new folder to create the web service

mkdir -p /c/dotstatsuite-website/nsiws-disseminate/

Step 2. Add full access rights to the users “IUSR” and “IIS_IUSRS” to the folder

icacls "C:\dotstatsuite-website\nsiws-disseminate" /grant:r "IUSR":"(OI)(CI)F"
icacls "C:\dotstatsuite-website\nsiws-disseminate" /grant:r "IIS_IUSRS":"(OI)(CI)F"

Step 3. Copy the compiled binaries to the new folder

cp -r /c/git/nsiws.net/src/NSIWebServiceCore/bin/Debug/net6.0/publish/* /c/dotstatsuite-website/nsiws-disseminate/

Step 4. Copy the following binaries from authorization.net to the Plugins folder. This will allow the NSI web service to retrieve authorization rules from .Stat common database when the authorization is enabled.

  • DotStat.Common.dll
  • DotStat.DB.dll
  • DotStat.Domain.dll
  • DotStat.MappingStore.dll
  • estat.sri.ws.auth.dotstat.dll
  • estat.sri.ws.auth.dotstat.deps.json
cp -r /c/git/authorization.net/src/estat.sri.ws.auth.dotstat/bin/Debug/net6.0/publish/{DotStat.Common.dll,DotStat.DB.dll,DotStat.Domain.dll,DotStat.MappingStore.dll,estat.sri.ws.auth.dotstat.dll,estat.sri.ws.auth.dotstat.deps.json} /c/dotstatsuite-website/nsiws-disseminate/Plugins

Step 5. Configure the nsi web service
From the dotstatsuite-core-sdmxri-nsi-ws repository, download the following sample configuration files to the deployment folder:

This sample configuration file has been set to use SQL server the databases and users that were previously created in this guide. See more about the configuration settings.

curl https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-core-sdmxri-nsi-ws/-/raw/master/docs/installation/config-examples/nsiws-disseminate-app.config?inline=false >/c/dotstatsuite-website/nsiws-disseminate/config/app.config
  • When using MariaDb:
  • Change connection string to match your MariaDb conenction string to the database created previously:
  • Change providerName from System.Data.SqlClient to MySql.Data.MySqlClient:
  • NEW - Set data database connection string Modify the configuration file /config/Properties.json - Add the value for the disseminationDbConnection connectionString:

    • Change From:
  "disseminationDbConnection": {
    "dbType": "SqlServer",
    "connectionString": ""
  }
  • To
  "disseminationDbConnection": {
    "dbType": "SqlServer",
    "connectionString": "Server=localhost;Database=DisseminateDataDb;User=testLoginDisseminateData;Password=testLogin(\!)Password"
  }
  • When using MariaDb change dbType to:
  "disseminationDbConnection": {
    "dbType": "MySql",
    "connectionString": "Server=localhost;Database=DisseminateDataDb;User=testLoginDisseminateData;Password=testLogin(\!)Password"
  }

Logging configuration.- By default, the service has been configured to log all activity. This causes performance issues during data extractions. To avoid performance issues on production envirements, please make the following changes to the file /config/log4net.config

In line 80 modify the xml block FROM:

...
  <root>
    <level value="ALL"/>
    <appender-ref ref="SystemActivityAppender"/>
    <appender-ref ref="UserActivityAppender"/>
  </root>
...

TO:

...
  <root>
    <level value="WARN"/>
    <appender-ref ref="SystemActivityAppender"/>
    <appender-ref ref="UserActivityAppender"/>
  </root>
...

Step 6. Create a new IIS application called nsiws-disseminate in port 80, using appcmd command

Make sure git bash is running in admin mode.

/c/Windows/System32/inetsrv/appcmd add site /name:nsiws-disseminate /physicalPath:C:\\dotstatsuite-website\\nsiws-disseminate /bindings:http/*:80:

Step 7. Create a new IIS application pool (NSIWSDisseminateAppPool)

/c/Windows/System32/inetsrv/appcmd add apppool /name:NSIWSDisseminateAppPool /managedRuntimeVersion:"" /managedPipelineMode:Integrated
  • Add the nsiws-disseminate application to the newly created application pool
/c/Windows/System32/inetsrv/appcmd set app "nsiws-disseminate/" /applicationPool:NSIWSDisseminateAppPool

Step 8. Configure the authentication and the authorization plugin for retrieval of .Stat authorization rules

There are two options to configure the authorization plugin:
1 . Json config file.- NOT RECOMMENDED

Download the sample files based on SQL Server nsiws-authentication.json and nsiws-design-authorization.json from dotstatsuite-core-sdmxri-nsi-ws repository to the deployment folder (C:\dotstatsuite-website\nsiws-disseminate\config). When using MariaDb, make sure you have the correct connection strings.

2 . Saving the configuration setting as environment variables for the IIS site (nsiws-disseminate). Recommended

See more about the configuration settings.

For this example we will use the second option:

  • Set authentication turned OFF
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__enabled',value='false']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__allowAnonymous',value='true']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__requireHttps',value='false']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__validateIssuer',value='false']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__showPii',value='false']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__clientId',value='stat-suite']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='auth__authority',value='']" /commit:apphost
  • Set authorization turned ON. When using MariaDb make sure to set DbType value to MariaDb.
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='authorization__enabled',value='true']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='authorization__method',value='dotstat']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='authorization__PrincipalFrom',value='context']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='DotStatSuiteCoreCommonDbConnectionString',value='Server=localhost;Database=CommonDb;User=testLoginCommon;Password=testLogin(\!)Password']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='DbType',value='SqlServer']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='mappingStore__Id__Default',value='disseminate']" /commit:apphost
/c/Windows/System32/inetsrv/appcmd set config "nsiws-disseminate" -section:system.webServer/aspNetCore /+"environmentVariables.[name='mappingStore__Id__FromQueryParameter',value='Never']" /commit:apphost

Please note that with the configuration above the NSI WS instance is configured to allow only anonymous access (as authentication is turned off). The authorization rules can be managed via Authorization Management service (e.g. using the Swagger UI of the service). Permissions granted to anonymous user is controlled by rules assigned to all users (userId="*").

Step 9. Start the new application

/c/Windows/System32/inetsrv/appcmd start site /site.name:nsiws-disseminate

Step 10. Test that the application is up and running

  • Using curl
curl localhost:80/health
  • Using a web browser
    Open a web browser and open the url localhost:80/health

You should see similar result:

{
  "service": {
    "status": "Healthy",
    "details": {
      "version": "8.17.0",
      "retriever": "MappingStoreRetrieversFactory",
      "retriever-version": "8.17.0",
      "middleware": "CorsMiddlewareBuilder,OpenIdMiddlewareBuilder,LoggingOptionsBuilder,UserAuthorizationRulesMiddlerwareBuilder",
      "maxRequestBodySize": 52428800
    },
    "responseTime": 0.3649
  },
  "db": {
    "status": "Healthy",
    "details": {
      "storeId": "disseminate",
      "version": "6.20",
      "isLatest": true
    },
    "responseTime": 4.1766
  },
  "memory": {
    "status": "Healthy",
    "details": {
      "allocatedMb": 75.5947,
      "gen0Collections": 22351,
      "gen1Collections": 4900,
      "gen2Collections": 1480
    },
    "responseTime": 0.0096
  },
  "disseminationDb": {
    "status": "Healthy",
    "details": {},
    "responseTime": 0.0442
  },
  "totalResponseTime": 5.405
}

Note: By default all the logs will be stored at C:/ProgramData/Eurostat/logs/


7. Additional IIS configuration

7.1 Changing maximum lenght of URL segments

Symptoms:
A valid SDMX data query with many query filter values produces the following error, even though the length of the url is below 4096 characters:

400 Bad request
HTTP Error 400. The request URL is invalid.

When the filter section of the query is shorter (than 260 characters) or removed, then the request produces regular response.

Solution:
When using IIS, the maximum number of characters in a URL path segment (the area between the slashes in the URL) is controled by a HTTP.sys configuration parameter called UrlSegmentMaxLength.
This can be changed in HKLM\SYSTEM\CurrentControlSet\Services\HTTP\Parameters with DWORD value UrlSegmentMaxLength.
The default value of this parameter (when this value is missing from registry) is 260, so by default this restriction is applied on IIS.

Increasing this parameter to a larger value (e.g. to 2048) resolves the issue reported.
If value is set to zero, then the length is bounded by the maximum value of a ULONG, but having no limitation on this value is not recommended. Please note that other settings may still limit the entire url, e.g. maximum number of url segments or maximum url length.

Please also note that after changing this value in registry, the restart of the Windows machine is required.


8. Authorization configuration

Please note that with the current installation the NSI WS and transfer service instances will not use authentication, all requests are treated as anonymous. On the other hand, the authorization is turned on. As the initial authorization configuration does not contain any authorization rule for anonymous user, the NSI WS instances will not allow any action, e.g. listing or manipulation of artefacts, retrieval of data. In order to allow anonymous users do actions, specific authorization rule(s) should be created. These are the generic rules that are applicable for any users, i.e. the user mask is *. For further info on the permissions please see the following page in the documentation: Permission business rules The simplest way to create such a rule is using the authorization management service via Manage permissions page of DLM or directly on the swagger UI of the authorization management sevice. Please note that both cases require a working authorization management service. This installation manual does not contain (yet) the description of the installation of authorization management service. Other way could be to apply the needed permissions directly in common database using an SQL tool.