Architecture #4782: Split rudder variable parsing and variable binding - Rudder - Issue Tracker

Actions

Copy link

Architecture #4782

closed

Architecture #4778: Optimize promise generation algorithm: Rudder variables lookup

Split rudder variable parsing and variable binding

Added by François ARMAND almost 10 years ago. Updated about 9 years ago.

Status:

Released

Priority:

Assignee:

Nicolas CHARLES

Category:

Performance and scalability

Target version:

2.11.0~beta1

Pull Request:

https://github.com/Normation/rudder/p...

Effort required:

Name check:

Fix check:

Regression:

Description

Today, we parse Rudder variable in Directives FOR EACH NODE. That does not make the least sense. We only need to bind them for each node, and even that is two wide: we can bind them as soon as the full context is qualified, or even remove free variable until there is no more.

So, we have to:

- create a new data structure for directive variable (Parametrized Variables ?) which allows to build the variable from a context (node infos, etc): a kind of Context => Variable type
- create a new parsing phase in the promise generation process to parse Directive parameters and extracts Rudder variables
- and it could (should ?) even be done at Directive update-time, so that we are able to give the user some feed-back on it.
- bind Parametrized Variable when their context is known

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by François ARMAND almost 10 years ago

Status changed from 8 to In progress
Pull Request set to https://github.com/Normation/rudder/pull/507

A first pull request is available here: https://github.com/Normation/rudder/pull/507

Everything mostly works, but:

- a lot of tests have to be changed, because they are now mostly irrelevant (test copy on case classes), and other part are missing (only the parsing is really well tested for now)
- interpolation in Rudder Parameters is not implemented (but it should be quite straight forward).

Actions

Copy link

Updated by François ARMAND almost 10 years ago

Ho, I missed the intersting part: node configuration building step of promise generation for ~250 nodes, ~20 rules, ~50 directives is down from ~25s to <5 secondes.

Actions

Copy link

Updated by François ARMAND almost 10 years ago

So, now everything is implemented, and timings (for ~250 nodes, 20 rules, some params and interpolation to actually solve) are:

Old:

Build current rule values :        162 ms, 23 ms, 24 ms
Expand rule parameters    :        223 ms, 64 ms, 41 ms
Build target configuration:      29709 ms, 27838 ms, 27555 ms
Update rule vals          :       1643 ms, 111 ms, 88 ms

Sum: 31737, 28036, 27708

New:

Build current rule values :         12 ms, 22 ms, 12 ms
Build target configuration:       2788 ms, 2751 ms, 2969 ms
Update rule vals          :        719 ms, 100 ms, 78 ms

Sum: 3519, 2873, 3059

So that's roughtly a x10 improvment.

Actions

Copy link

Updated by François ARMAND almost 10 years ago

Status changed from In progress to Pending technical review
Assignee changed from François ARMAND to Nicolas CHARLES

There is some tests that don't seem to test anything anymore, but I didn't succeed in identifying what we wanted to test on them.
A lot of other tests were added (more on the interpolation variable interpreation code, thought).

Actions

Copy link