How the search service can support API keys

Released in November 30, 2020 Release .Stat Suite JS 6.1.0

Original use case

Some externally-available NSI services have an API Gateway on front that limits the rate at which users can call the services. It is possible to call the service with or without an API Key attached to the request. Requests with an API Key are throttled according to that key’s usage plan. Requests without an API Key are considered to be all on the same plan, with a relatively low acceptable rate of calls.

The SDMX Faceted Search calling the services in order to index them without an API Key could quickly reach the rate limit and fail. The ability to specify an API Key for the service to use was therefore required, or the ability for the SDMX Faceted Search service to avoid calling the endpoints that require an API key.

Implemented solution

The feature is implemented to answer the use case is adding a new config. file named datasources.json and depending on the env, it will overrides the datasources.json from the configuration service, without being spread over network and with a lot of flexibility. In other words, it overrides datasources at the sfs level thus allowing using API keys while indexing. An example of overriden datasource.json file:

{
  "staging:SIS-CC-stable": {
    "url": "http://nsi-stable-siscc.redpelicans.com/rest",
    "headers": {
      "X-API-KEY-ID": "staging:SIS-CC-stable"
    }
  },
  "dev:unstable": {
    "url": "http://localhost:9001/rest"
  }
}

In dev (source-code installation), add a local file in /data folder.
In Docker, add the file in the /data folder which is mount.
In Kube, add a config map from a file named sfs-datasources (will be automatically used).

Notes:

• If the sources do not exist in the original datasource, then nothing is done (allow using a single configmap for many overrides);
• All HTTP headers can be used, SDMX headers will override them in order to make sfs always work properly with an expected API;
• The entire datasources content can be redefined.