Project

General

Profile

User story #9698

Import node properties from external data sources

Added by Vincent MEMBRÉ 10 months ago. Updated 8 months ago.

Status:
Released
Priority:
N/A
Assignee:
-
Category:
Web - Config management
Target version:
Target version (plugin):
Suggestion strength:
User visibility:
Effort required:
Pull Request:

Description

Some data may be available outside of Rudder to be used to configure your nodes.

To be able to use them in Rudder you need to add them into Rudder as Node properties.

That task may be complicated to achieve and even more to automate (so data stays up to date), we want to add a way to do this easily in Rudder.

We will have to define those external data sources in Rudder. Rudder will then fetch data from them and add them as properties for each nodes.

Data structure:

  • name
  • url
  • request type (one by node, one for all nodes)
  • headers
  • A way to define data access into received data
  • Update frequency

We need:

  • Backend storage for datasources
  • An api Rest to define datasource
  • An ui to define datasource
  • System to fetch data sources and add them as properties
  • Scheduler to launch it asynchronously
  • Use these data in policy generation

Subtasks

User story #9699: UI to define datasources ReleasedVincent MEMBRÉ

User story #9700: API Rest to define Data sourcesReleasedFrançois ARMAND

Bug #9725: Data source Scala modelReleasedFrançois ARMAND

User story #9897: Move datasource parameters into a specific entry in rest jsonReleasedFrançois ARMAND

User story #9899: Change format of time duration in datasource Rest APIReleasedNicolas CHARLES

User story #9701: Store data sources in backendReleasedFrançois ARMAND

User story #9724: Query HTTP datasourceReleasedFrançois ARMAND

User story #9931: Test UpdateHttpDatasetTest does not run anymoreReleasedFrançois ARMAND

User story #9898: 404 on request should empty the properties on NodeReleasedFrançois ARMAND

Bug #9910: Bugs on datasource initilization and updateReleasedNicolas CHARLES

User story #9942: Migration script to add missing table in PostgresReleasedFrançois ARMAND

User story #9944: Migration script to add missing table in PostgresReleasedFrançois ARMAND

Bug #9947: Build broken (missed refactoring)ReleasedVincent MEMBRÉ

User story #9978: Document datasource APIReleasedVincent MEMBRÉ

Bug #10191: Update datasource API now that it is a module with new endpointsReleasedVincent MEMBRÉ

Bug #9987: Cannot use " in the result of a DatasourceRejectedFrançois ARMAND

Bug #9995: Forbid modification of properties set by datasourceReleasedVincent MEMBRÉ

Bug #10026: Datasource config: missing interpolation of rudder params in request headerReleasedVincent MEMBRÉ

Bug #10043: Missing refresh API for one data source onlyReleasedVincent MEMBRÉ

User story #10027: Add a POST option for datasource queriesReleasedVincent MEMBRÉ

User story #10039: Document data sources featureReleasedAlexis MOUSSET


Related issues

Related to Rudder - User story #6264: Get directive parameter value from external source (kv store, rest, etc) New 2015-02-13

History

#1 Updated by Janos Mattyasovszky 10 months ago

Also don't forget to have somekind of failsafe to handle the case when no data is returned (no error but empty dataset) => Would you delete all existing properties?

#2 Updated by François ARMAND 10 months ago

@Janos: in all case, the behavior is limited to the key "data-source-name".

We have thought to have an option in the data source configuration with the following option, when nothing is found for a node:
- ignore (i.e: don't modify the existing state)
- raise an error;
- delete value for "data-source-name" key
- use a default value = ....

#3 Updated by Vincent MEMBRÉ 8 months ago

  • Target version changed from 4.1.0~beta1 to 4.1.0~beta2

#4 Updated by Vincent MEMBRÉ 8 months ago

  • Status changed from New to Pending technical review
  • Target version changed from 4.1.0~beta2 to 4.1.0~beta1

#5 Updated by Vincent MEMBRÉ 8 months ago

  • Status changed from Pending technical review to 12

#6 Updated by Vincent MEMBRÉ 8 months ago

  • Status changed from 12 to Pending release

#7 Updated by Vincent MEMBRÉ 8 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 4.1.0 which was released today.

#8 Updated by Benoît PECCATTE 7 months ago

  • Related to User story #6264: Get directive parameter value from external source (kv store, rest, etc) added

Also available in: Atom PDF