.Stat DevOps implementation
Table of Content
- open-source development business requirements
- technical stack
- technical environment (cloud)
- flow
- process
- mapping
- git
- gitlab
- npm
- nuget
- list of webapps/services/packages
open-source development business requirements
quality assurance (qa) environment
- Automated update triggered by Dev branch merge (part of “peer-review” Kanban step)
- Used by Product Management team to:
- directly test through API/Swagger/Postman candidate .Stat CORE services:
° NSI ws with stable content (upgraded database) called ‘stable’
° NSI ws with re-initialised content (new database) called ‘reset’
° Transfer ws
° Authorisation ws - test ‘PM’ tenant of candidate DE plugged to:
° SIS-CC Demo-related staging instance of NSI ws with stable content (upgraded database) called ‘SIS-CC-stable’
° SIS-CC Demo-related staging instance of NSI ws with re-initialised content (new database) called ‘SIS-CC-reset’
° Data viewer app - test ‘PM’ tenant of candidate DLM plugged to:
° SIS-CC Demo-related staging instance of NSI ws with stable content (upgraded database) called ‘SIS-CC-stable’
° SIS-CC Demo-related staging instance of NSI ws with re-initialised content (new database) called ‘SIS-CC-reset’
° SIS-CC Demo-related staging instance of Transfer ws
° SIS-CC Demo-related staging instance of Authorisation ws
- directly test through API/Swagger/Postman candidate .Stat CORE services:
- Other links:
functional pre-production (staging) environment
- Automated update triggered by Master branch merge (part of “release” Kanban step)
- Hosting stable instances for OECD Practice Building team of:
- NSI ws with stable content (upgraded database) called ‘OECD-design’
- NSI ws with stable content (upgraded database) called ‘OECD-staging’
- Transfer ws
- Authorisation ws
- ‘OECD’ tenant of DE plugged to:
° stable NSI ws with stable content (upgraded database) called ‘OECD-design’
° stable NSI ws with stable content (upgraded database) called ‘OECD-staging’
° Data viewer app - ‘OECD’ tenant of DLM plugged to:
° stable NSI ws with stable content (upgraded database) called ‘OECD-design’
° stable NSI ws with stable content (upgraded database) called ‘OECD-staging’
° stable Transfer ws
° stable Authorisation ws
- Hosting stable instances for SIS-CC (for Demo purposes) of:
- NSI ws with stable content (upgraded database) called ‘SIS-CC-stable’
- NSI ws with re-initialised content (new database) called ‘SIS-CC-reset’
- Transfer ws
- Authorisation ws
- ‘SIS-CC’ tenant of DE plugged to:
° stable NSI ws with stable content (upgraded database) called ‘SIS-CC-stable’
° stable NSI ws with re-initialised content (new database) called ‘SIS-CC-reset’
° Data viewer app - ‘SIS-CC’ tenant of DLM plugged to:
° stable NSI ws with stable content (upgraded database) called ‘SIS-CC-stable’
° stable NSI ws with re-initialised content (new database) called ‘SIS-CC-reset’
° stable Transfer ws
° stable Authorisation ws
- Other links:
- Other tenants of DE and DLM for specific SIS-CC members
- either plugged to their own SDMX APIs
- or plugged into above mentioned SIS-CC Demo-related instances of .Stat CORE services
technical stack
The following technologies and cloud services are used for the DevOps implementation:
technical environment (cloud)
- .Stat Data Explorer components and .Stat Data Lifecycle Manager
- 1 cluster of 3 nodes with 2 namespaces (qa and staging) on google cloud platform
- https
- probes
- kubernetes, how to update topology, not automated
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-kube-rp
° holds kubernetes configuration files
- .Stat Core components
- 1 cluster of 3 nodes with 2 namespaces (qa and staging) on google cloud platform
- https
- probes
- kubernetes, how to update topology, not automated
- repository: https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-kube-core-rp
° holds kubernetes configuration files
flow
graph TB
id0[gitlab repositories]
id1[gitlab pipelines]
id2[docker registry]
id4[kubernetes cluster]
id5[dotstatsuite-kube-rp]
id6[kubectl]
id7((developer))
id9((admin))
id7 --> id0
id0 -->|1. push code | id1
id1 -->|2. push image | id2
id1 -->|3. connect gcloud | id6
id6 -->|4. deploy image | id4
id9 -->|A. connect gcloud | id6
id6 -->|B. update topology | id4
id9 -->|C. push code | id5
id2 -. pull image .-> id4
id5 -. pull code .-> id4
process
graph TD;
id0((peer-reviewer/code guard));
id1[gitlab application repository: dev branch];
id2[gitlab application repository: master branch];
id4[gitlab library repository: master branch];
id5[gitlab application repository: pipeline];
id6[gitlab library repository: pipeline];
id7[test environment];
id8[library registry npm/nuget];
id9[docker registry];
id10[kubernetes cluster: qa environment];
id14[kubernetes cluster: staging environment];
id11[gitlab dotstatsuite-kube-rp/dotstatsuite-kube-core-rp repositories: master branch]
id12[kubectl]
id13((admin))
id0 -->|A1. merge code| id4;
id0 -->|B1. merge code| id1;
id0 -->|C1. merge code| id2;
id4 -->|A2. triggers launch| id6;
id1 -->|B2. triggers launch| id5;
id2 -->|C2. triggers launch| id5;
id6 -->|A4. create & push package| id8;
id6 -->|A3. build, test| id7;
id5 -->|BC3. build, test| id7;
id5 -->|BC4. create & push docker image| id9;
id5 -->|BC5. connect gcloud| id12
id12 -->|B6. deploy image| id10
id13 -->|1. connect gcloud| id12
id12 -->|2. update topology| id10
id13 -->|3. push code| id11
id8 -. pull package .-> id10
id9 -. pull image .-> id10
id11 -. pull code .-> id10
id12 -->|C6. deploy image| id14
id12 -->|2. update topology| id14
id8 -. pull package .-> id14
id9 -. pull image .-> id14
id11 -. pull code .-> id14
mapping
| env | git branch | cluster namespace |
|---|---|---|
| qa | develop | qa |
| staging | master | staging |
git
- see git-flow
gitlab
- all repositories are under https://gitlab.com/sis-cc/.stat-suite
- each repository defines its pipelines in
gitlab-ci.ymlfile - 2 types of pipeline:
webapp/service
graph LR
subgraph CI all branches
setup --> unit-tests
unit-tests --> build
end
subgraph CD only develop & master
build --> release
release --> deploy
end
npm/nuget package
graph LR
subgraph CI all branches
setup --> unit-tests
unit-tests --> build
end
subgraph CD only tags*
build --> publish
end
npm
- npm packages are published under https://www.npmjs.com/settings/sis-cc/packages
- tags are only on commits in master and trigger a publish
nuget
- nuget packages are published under https://www.nuget.org/profiles/SIS-CC
- tags are only on commits in master and trigger a publish
- this page provides information about Usage of semantic versioning in .Stat Suite CORE components and services
list of webapps/services/packages
| kind | name | status master | status dev | coverage |
|---|---|---|---|---|
| infra | kubernetes de | - | - | - |
| infra | kubernetes core | - | - | - |
| - | - | - | - | - |
| service | config |  |
 |
 |
| service | sdmx-faceted-search |  |
 |
 |
| service | proxy |  |
 |
 |
| service | share |  |
 |
 |
| - | - | - | - | - |
| service | core-auth-management |  |
 |
 |
| service | core-transfer |  |
 |
 |
| - | - | - | - | - |
| webapp | data-explorer |  |
 |
 |
| webapp | data-viewer |  |
 |
 |
| webapp | data-lifecycle-manager |  |
 |
 |
| - | - | - | - | - |
| package | ui-footer |  |
- |  |
| package | ui-header |  |
- |  |
| package | components |  |
- |  |
| package | ui-components |  |
- |  |
| package | d3-charts |  |
- |  |
| package | sdmxjs |  |
- |  |
| package | visions |  |
- |  |
| - | - | - | - | - |
| package | core-data-access |  |
 |
 |
| package | core-common |  |
 |
 |
| package | core-sdmxri-nsi-plugin |  |
 |
 |
packages
| name | version | status | coverage | demo | note(s) |
|---|---|---|---|---|---|
| sdmxjs |  |
 |
|
sdmxjs | :muscle: @RedPDRoncoli |
| visions |  |
 |
|
visions | :sparkles: @mike.velluet |
| ui-footer |  |
 |
|
- | will be included visions |
| ui-header |  |
 |
|
- | will be included visions |
| components |  |
 |
|
- | tests are scattered inside |
| ui-components |  |
 |
|
- | will be merged with visions |
| d3-charts |  |
 |
|
- | complex to test |