Mono-tenant installation of DE and DLM services from gitlab pipeline artifacts

Table of Content

pre-requisites
summary
1. setup
2. config service
3. search service
4. share service
5. data-explorer app
6. data-viewer app
7. data-lifecycle-manager app

slightly different than the source code approach; git is not mandatory anymore and side-effects related to dependencies are avoided.

pre-requisites

(reminder) an accessible (CORS-enabled, anonymous access enabled) SDMX v6.x endpoint
nodejs 14.x and npm 6.14.x installed
git installed ; only to use git bash instead of cmd.exe for process variables
solr 7.x to solr 8.7 (since May 19, 2021 Release .Stat Suite JS 8.0.0) installed and running on port 8983 (default) with a core created with solr create -c sdmx-facet-search from solr console
redis downloaded and running on port 6379 (default)
mongo downloaded and running on port 27017 (default)
keycloak v7.x (minimum) - v12.0.4 (latest tested and recommended) downloaded and running on port 8080 (default)

notes:

solr core name can be freely picked, default configuration targets sdmx-facet-search
ports can be freely picked, if different than defaults they should be specified when launching services

summary

from gitlab artifacts summary

1. setup

(temporary) npm i -g cross-env
create folders as follow:

.
├── dotstatsuite
│   ├── config
│   ├── search
│   ├── data-explorer
│   ├── data-viewer
│   ├── share
|   ├── data-lifecycle-manager

2. config service

download artifact archives and package.json file from gitlab:

extract archives and organize folders/files as follow:

.
├── dotstatsuite
│   ├── config
│   │   ├── node_modules                       # from setup artifact
│   │   ├── dist                               # from build artifact
│   │   ├── configs                            # see note a
│   │   │   ├── tenants.json                   # tenants & datasources definition
│   │   │   │   ├── default
│   │   │   │   │   ├── data-explorer
│   │   │   │   │   │   ├── i18n
│   │   │   │   │   │   ├── settings.json
│   │   │   │   │   ├── data-viewer
│   │   │   │   │   │   ├── i18n
│   │   │   │   │   │   ├── settings.json
│   │   ├── assets                             # see note b
│   │   │   ├── default
│   │   │   │   ├── data-explorer
│   │   │   │   │   │   ├── images
│   │   │   │   │   │   ├── styles*
│   │   │   │   ├── data-viewer
│   │   │   │   │   │   ├── images
│   │   │   │   │   │   ├── styles*
│   │   ├── package.json

notes:

a: see configs samples here
b: see assets samples here
in settings, assets should have the right path, if you change the name folder by default make sure to change path assets url

"viewer": {
  "logo": "/assets/default/data-explorer/images/sis-cc-logo.png",
}

3 . start the service:

(in git bash) run PORT=5007 npm run dist:run
check if everything is fine: http://localhost:5007/healthcheck

3. search service

download artifact archives and package.json files from gitlab:

extract archives and organize folders/files as follow:

.
├── dotstatsuite
│   ├── search
│   │   ├── node_modules                       # from setup artifact
│   │   ├── dist                               # from build artifact
│   │   ├── package.json

start the service:

(temporary) edit the 1st line of dist/params/production.js as follow: const server = { host: '0.0.0.0', port: 3007 };
(in git bash) run PORT=3007 CONFIG_URL=http://localhost:5007 MONGODB_URL=mongodb://localhost:27017 MONGODB_DATABASE=sfs API_KEY="secret" SOLR_HOST=localhost npm run dist:run
check if everything is fine: http://localhost:3007/healthcheck
check if the config (mostly to check datasources) is fine: http://localhost:3007/api/config
create a collection How to create a collection

handle index (that are indexable):

get config: curl -X GET http://localhost:3007/admin/config?api-key=secret
index dataflows: curl -X POST http://localhost:3007/admin/dataflows?api-key=secret -d {}
get report: curl -X GET http://localhost:3007/admin/report?api-key=secret
delete dataflows and config: curl -X DELETE http://localhost:3007/admin/config?api-key=secret

notes:

process variables are SOLR_HOST, SOLR_PORT, MONGODB_URL, MONGODB_DATABASE, CONFIG_URL, API_KEY and PORT
PORT is not available yet, which explain why dist/params/production.js should be updated
if the config is not fine, update the datasources.json and/or settings.json file(s) in the config service, restart the config service then restart the search service
the search service is not coupled with its clients, the search endpoint is configurable from settings.json of the data-explorer app in the config service

download artifact archives and package.json files from gitlab:

extract archives and organize folders/files as follow:

.
├── dotstatsuite
│   ├── share
│   │   ├── node_modules                       # from setup artifact
│   │   ├── dist                               # from build artifact
│   │   ├── package.json

start the service:

(in git bash) run PORT=3006 CONFIG_URL=http://localhost:5007 REDIS_HOST=localhost SITE_URL=http://localhost:3006 npm run dist:run
check if everything is fine: http://localhost:3006/healthcheck

notes:

SITE_URL should be an accessible url for a user, ie https://share-demo.siscc.org/healthcheck if used out of localhost

5. data-explorer app

download artifact archives and package.json files from gitlab:

go to tags and select the one you want to use
download the artifacts from the tag (ie for tachyon):
package.json

extract archives and organize folders/files as follow:

.
├── dotstatsuite
│   ├── data-explorer
│   │   ├── node_modules                       # from setup artifact
│   │   ├── dist                               # from build-dist artifact (server files)
│   │   ├── build                              # from build-srv artifact (client bundle)
│   │   │   ├── assets                         # assets from config
│   │   ├── package.json

Copy your assets from config and put them in build folder
start the service:

(in git bash) run SERVER_PORT=3009 CONFIG_URL=http://localhost:5007 AUTH_SERVER_URL=http://localhost:8080 npm run start:run
check if everything is fine: http://localhost:3009 (proxy with a route mapped to data-explorer)

6. data-viewer app

download artifact archives and package.json files from gitlab:

go to tags and select the one you want to use
download the artifacts from the tag (ie for tachyon):
package.json

extract archives and organize folders/files as follow:

.
├── dotstatsuite
│   ├── data-viewer
│   │   ├── node_modules                       # from setup artifact
│   │   ├── dist                               # from build-dist artifact (server files)
│   │   ├── build                              # from build-srv artifact (client bundle)
│   │   │   ├── assets                         # assets from config
│   │   ├── package.json

Copy your assets from config and put them in build folder
start the service:

(in git bash) run SERVER_PORT=3005 CONFIG_URL=http://localhost:5007 npm run start:run

7. data-lifecycle-manager app

download artifact archives and package.json files from gitlab:

go to tags and select the one you want to use
download the artifacts from the tag (ie for tachyon):
package.json

extract archives and organize folders/files as follow:

.
├── dotstatsuite
│   ├── data-lifecycle-manager
│   │   ├── node_modules                       # from setup artifact
│   │   ├── dist                               # from build-dist artifact (server files)
│   │   ├── build                              # from build-srv artifact (client bundle)
│   │   │   ├── assets                         # assets from config
│   │   ├── package.json

Copy your assets from config and put them in build folder
start the service:

(in git bash) run SERVER_PORT=7000 CONFIG_URL=http://localhost:5007 AUTH_SERVER_URL=http://localhost:8080 npm run start:run

Mono-tenant installation of DE and DLM services from gitlab pipeline artifacts

Table of Content

pre-requisites

summary

1. setup

2. config service

3. search service

4. share service

5. data-explorer app

6. data-viewer app

7. data-lifecycle-manager app