Rudder

== Configuration concepts

=== Techniques

==== Concepts

A Technique defines a set of operations and configurations to reach the

desired behaviour. This includes the initial set-up, but also a regular check on

the parameters, and automatic repairs (when possible).

All the Techniques are built with the possibility to change only part of a

service configuration: each parameter may be either active, either set on the

"Don't change" value, that will let the default values or in place. This allows

for a progressive deployment of the configuration management.

Finally, the Techniques will generate a set of reports which are sent to

the Rudder Root Server, which will let you analyse the percentage of compliance

of your policies, and soon, detailed reports on their application.

==== Manage the Techniques

The Techniques shipped with Rudder are presented in a library that you can

reorganize in *Configuration > Techniques*. The library

is organized in two parts: the available Techniques, and the selection

made by the user.

include::../glossary/technique-library.txt[]

include::../glossary/active-techniques.txt[]

[TIP]

====

The current version of Rudder has only an handful of *Techniques*. We are

aware that it considerably limits the use of the application, but we choose to

hold back other Techniques that did not, from our point of view, have the

sufficient quality. In the future, there will be some upgrades including more

Techniques.

====

[WARNING]

====

The creation of new Techniques is not covered by the Web interface. This

is an advanced task which is currently not covered by this guide.

====

=== Directives

Once you have selected and organized your Techniques, you can create your

configurations in the *Configuration Management > Directives* section.

include::../glossary/directive.txt[]

The screen is divided in three parts:

- on the left, your list of Techniques and Directives,

- on the right the description of the selected Technique or Directive.

- at the bottom, the configuration items of the selected Directive.

Click on the name of a Technique to show its description.

Click on the name of a Directive to see the Directive Summary containing the

description of the Technique its derived from, and the configuration items

of the Directive.

.Create a Directive for Name resolution

====

Use the Technique 'Name resolution' to create a new Directive called

+Google DNS Servers+, and shortly described as 'Use Google DNS Server'. Check in

the options 'Set nameservers' and 'Set DNS search suffix'. Set the value of the

variable 'DNS resolver' to +8.8.8.8+ and of 'Domain search suffix' according to

your organization, like +rudder-project.org+.

====

=== Rules

include::../glossary/rule.txt[]

When a Rule is created or modified, the promises for the target nodes are generated. Rudder computes all the promises each nodes must have, and makes them available for the nodes. This process can take up to several minutes, depending on the number of managed nodes and the Policy Server configuration. During this time, the "Regenerate now" button is replaced by a moving bar and a message stating "Generating rules".

You can also press the "Regenerate now" button on the top of the interface if you feel the generated promises should be modified (for instance, if you changed the configuration of Rudder)

=== Variables

==== User defined parameters

Rudder provides a simple way to add common and reusable variables in either plain Directives, or techniques created using the Technique editor: the parameters.

image::./images/rudder-parameters.png[Parameters]

The parameters enable the user to specify a content that can be put anywhere, using the following syntax:

* In Directives: '${rudder.param.name}' will expand the content of the "name" parameter.

* In the Technique Editor: '${rudder_parameters.name}' will do the same.

Using this, you can specify common file headers (this is the default parameter, "rudder_file_edit_header"), common DNS or domain names, backup servers,

site-specific elements...

==== System variables

Rudder also provides system variables that contain information about nodes

and their policy server. You can use them like user defined parameters.

The information about a Node:

* '${rudder.node.id}' returns the Rudder generated id of the Node

* '${rudder.node.hostname}' returns the hostname of the Node

* '${rudder.node.admin}' returns the administrator login of the Node

The information about a Node's policy server.

* '${rudder.node.policyserver.id}' returns the Rudder generated id of the Policy Server

* '${rudder.node.policyserver.hostname}' returns the hostname of the Policy Server

* '${rudder.node.policyserver.admin}' returns the administrator login of the Policy Server

=== Compliance

A Directive contains one or multiple components. Each component generates

one or multiple reports, based on the number of keys in this component. For

example, for a Sudoers Directive, each user is a key. These states are

available in reports:

Success::

The system is already in the desired state. No change is needed. Conformity is

gained.

Repaired::

The system was not in the desired state. Rudder applied some change and repaired

what was not correct. Now the system is in the desired state. Conformity is

gained.

Error::

The system is not in the desired state. Rudder couldn't repair the system.

Applying::

When a Directive is applied, Rudder waits during 10 minutes for a report.

During this period, the Directive is said 'Applying'.

No report::

The system didn't send any reports. Rudder waited for 10 minutes and no report

was received.

A Directive has gained conformity on a Node if every report for each

component, for each key, is in 'Success' state. This is the only condition.

Based on these facts, the compliance of a Rule is calculated like

this:

Number of Nodes for which conformity is reached for every Directive of the

Rule / Total number of Nodes on which the Rule has

been applied

.Compliance on a Rule

image::./images/rudder-rule-compliance.png[Rule compliance]

The Rule detailed compliance screen will also graph compliance deviations on

a recent period as well as display a deviation log history for this period.

.Compliance history on a Rule

image::./images/rudder-rule-compliance-history.png[Rule compliance history]

=== Validation workflow in Rudder

The validation workflow is a feature whose purpose is to hold any change (Rule, Directive, Group) made by users in the web interface,

to be reviewed first by other users with the adequate privileges before actual deployment.

The goal is to improve safety and knowledge sharing in the team that is using Rudder.

To enable it, you only have to tick "Enable Change Requests" in the Administration - Settings tab of the web interface. (This feature

is optional and can be disabled at any time without any problem, besides risking the invalidation of yet-unapproved changes)

image::./images/workflows/Enabling.png[]

==== What is a Change request ?

A Change request represents a modification of a Rule/Directive/Group from an old state to a new one.

The Change is not saved and applied by the configuration, before that, it needs to be reviewed and approved by other members of the team.

A Change request has:

- An Id (an integer > 0)

- A title.

- A description.

- A creator.

- A status.

- Its own history.

This information can be updated on the change request detail page.

For now, a Change request is linked to one change at a time.

===== Change request status

There is 4 Change request status:

Pending validation::

- The change has to be reviewed and validated.

- Can be send to: Pending deployment, Deployed, Cancelled.

Pending deployment::

- The change was validated, but now require to be deployed.

- Can be send to: Deployed, Cancelled.

Deployed::

- The change is deployed.

- This is a final state, it can't be moved anymore.

Cancelled::

- The change was not approved.

- This is a final state, it can't be moved anymore.

Here is a diagram about all those states and transitions:

image::./images/workflows/States.png[]

==== Change request management page

All Change requests can be seen on the /secure/utilities/changeRequests page.

There is a table containing all requests, you can access to each of them by clicking on their id.

You can filter change requests by status and only display what you need.

image::./images/workflows/Management.png[]

===== Change request detail page

Each Change request is reachable on the /secure/utilities/changeRequest/id.

image::./images/workflows/Details.png[]

The page is divided into two sections:

Change request information::

display common information (title, description, status, id) and a form to edit them.

image::./images/workflows/Informations.png[]

Change request content::

In this section, there is two tabs:

- History about that change request

image:./images/workflows/History.png[]

- Display the change proposed

image:./images/workflows/Rule_Update_Diff.png[]

==== How to create a Change request ?

If they are enabled in Rudder, every change in Rudder will make you create a Change request.

You will have a popup to enter the name of your change request and a change message.

The change message will be used as description for you Change Request, so we advise to fill it anyway to keep an explanation ab out your change.

image::./images/workflows/Popup.png[]

Change request are not available for Rule/Directive/Groups creation, they are only active if the Rule/Directive/Groups existed before:

Here is a small table about all possibilities:

image::./images/workflows/Table.png[]

==== How to validate a Change request ?

===== Roles

Not every user can validate or deploy change in Rudder.

Only those with one of the following roles can act on Change request:

Validator::

Can validate Change request

Deployer::

To deploy Change Request

Both of those roles:

- Give you access to pending Change requests

- Allow you to perform actions on them (validate or cancel)

You have to change users in */opt/rudder/etc/rudder-users.xml* and include those rights.

Without one of those roles, you can only access Change Request in 'Deployed' or 'Cancelled' and those you opened before.

You can deploy directly if you have both the validator and deployer roles.

The *administrator* Role gives you both the deployer and valdiator role.

There is also the possibility to access Change requests in Read only mode by using the role 'validator_read' or 'deployer_read'.

image::./images/workflows/Validation.png[]

===== Self Validations

Using Change requests means that you want your team to share knowledge, and validate each other change.

So by default:

- *Self validation* is disabled.

- *Self deployment* is enabled.

Those two behaviours can be changed in the property file */opt/rudder/etc/rudder-web.properties*.

'rudder.workflow.self.validation' and 'rudder.workflow.self.deployment' are the properties that define this behaviour.

==== Change request and conflicts

When the initial state of a Change request has changed (i.e.: you want to modify a Directive, but someone else changes about that Directive has been accepted before yours), your change can't be validated anymore.

image::./images/workflows/Conflict.png[]

For now, we decided to reduce to the possibility of an error or inconsistency when there are concurrent changes.

In a future version of Rudder, there will be a system to handle those conflicts, and make sure actual changes are not overwritten.

==== Notifications:

In several parts of Rudder webapp there are some Notifications about Change requests.

===== Pending change requests

This notification is displayed only if the validator/deployer role is active on your user account.

It shows you how many Change requests are waiting to be reviewed/deployed.

Clicking on it will lead you to the Change request management page, with a filter already applied.

image::./images/workflows/Notification.png[]

===== Change already proposed on Rule/Directive/Group

When there is a change about the Rule/Directive/Group already proposed but not deployed/cancelled, you will be notified that there are some pending Change requests about that element.

You will be provided a Link to those change request, So you can check if the change is already proposed.

image::./images/workflows/Warning.png[]

=== Policy Mode (Audit/Enforce)

Rudder 4.0 includes a policy mode setting, that allows two distinct behaviors:

* *Audit*: Test if the system is in the desired state, and report about it

* *Enforce*: Test if the system is in the desired state, if not, try to act to get to this state, and report about actions taken and final state

This mode can be set:

* Globally on the Rudder root server. In this can case there are two options: allow to override this mode on specific items, or use the global configuration everywhere.

* On a directive.

* On a node.

A lot of attention and several safeguards have been put in place to ensure that if you choose to use "Audit"

for a target, nothing will be changed on the node for that target (except Rudder's own configuration under `/var/rudder`), and only some harmless

commands will be run (like listing installed packages or refreshing package lists).

Nodes are fully aware of exactly what directives need to be executed in Audit or in Enforce mode, and the "rudder agent" command line has been enhanced to let you see the result with a glimpse: the first column in `rudder agent run` output is now the mode (*A* for *Audit* and *E* for *Enforce*), and the compliance summary is split by audit mode.

In addition to pre-existing technical reports, new ones have been added to report on "audit-compliant" (the check was OK), "audit-non-compliant" (the check was done, but the result is not the one expected), "audit-not-applicable" (the check is not applicable for that node, for example because of a limitation on the OS type), "audit-error" (the check wasn't able to finish correctly) status.

==== How is the effective mode computed?

We will here explain what is the computation made during generation to

decide which mode to apply to a directive on a node, based on the current settings.

The short rule is: *Override wins, then Audit wins*

For a given directive on a given node at a given time, we have three different policy mode

settings:

* The global mode, called *G*, which can be *Audit* or *Enforce*

* The node mode called *N*, which can be *Global* (if not overridden), *Audit, or *Enforce*

* The directive mode, called *D*, which can be *Global* (if not overridden), *Audit, or *Enforce*

The result is:

* If override is not allowed, the policy mode is *always* the global mode *G*.

* If override is allowed:

** If *N* and *D* are set to use the *Global* default value (i.e. no override), the policy mode is the global mode *G*.

** If *N* uses the *global* value and *D* is overriden to *Audit* or *Enforce*, the *D* value is used.

** If *D* uses the *global* value and *N* is overriden to *Audit* or *Enforce*, the *N* value is used.

** If *N* and *D* are overriden to *Audit* or *Enforce*, the value is *Audit* if at least one of *N* or *D* is *Audit*, *Enforce* if both are in *Enforce* mode

[[_manage_your_it]]

== Configuration policies

=== How to

==== Enforce a line is present in a file only once

Enforcing that a line to be present in a single occurence in a file is not an easy process to automate. Providing templates is an easy way to achieve this but not always possible.

If you don't want to use a template, you can use Technique *Enforce a File content* to control the content of a file.

The whole logic to edit a file so it contain only one occurence of a line is:

* Add the line, so it will be added if missing)

* Replace line that looks almost like our line by the line

* Delete all duplicated lines

With these 3 steps, You will end with one line!

So, here is a small example: let's say you want /etc/sysconfig/sysctl to contain line 'ENABLE_SYSRQ="yes"'

You will need to create a Directive based on Enforce a File content with the following content:

image::./images/oneLine.png[oneLine]

==== Share files between nodes

Rudder 4.1 introduced a way to share files from one node to another.

It allows a node to send a file to its relay, which will make it available

for another target node, that has to to specifically download it.

This file sharing method is secured by:

* The control of uploaded file signature by the server, to check it matches the source node's private key.

* The same mechanism as standard file copy in Rudder to download the shared file from the server.

It also includes a ttl mechanism that allows sharing a file for a limited amount of time.

To use this feature, two generic methods are available in the technique editor:

* <<sharedfile_from_node, sharedfile_from_node>>: To download a file shared from another node.

* <<sharedfile_to_node, sharedfile_to_node>>: To make a file available to another node.

See the documentation of these methods for details about the required parameters,

and especially <<sharedfile_to_node, sharedfile_to_node>> for a complete usage example.

=== Security considerations

==== Data confidentiality

Rudder is designed to strictly separate policies between nodes,

and to only let a node access its own policies.

This section will give details about how the policies are secured, and which

content is node-specific or global.

===== Private data

All confidential information should be stored in private data, namely:

* the directives, groups, rules, and their parameters

* the techniques parameters in the Technique Editor

* the shared-files directory

There are:

* always transfered encrypted between nodes (using agent copy protocol or https for the interface and the API)

* only available to the nodes that need it

* only accessible locally by the users that need it

More precisely:

* root server:

** all the data is present on it

** files are readable and writable only by the root user and (for some of them) the rudder group

** some data is also accessible from our backends (PostgreSQL, OpenLDAP), but only locally (the services are listenning on loopback) and from Rudder-specific users, whith passwords only accessible to the root user

** accessible remotely by the Web interface (needs an authorized user account) or the API (needs a token)

* relay: only the data needed for the served nodes and the relay itself are available and stored locally, only accessible to the root user

* node: only the data needed to configure the node is available and stored locally, only accessible to the root user

===== Common data

This refers to content available from all nodes in the authorized networks, readable from all users

on the nodes (and that can be transfered withtout encryption when using initial promises of a pre-4.0 node).

These unprotected contents are:

* the tools (`/var/rudder/tools`)

* the common ncf part (`/var/rudder/ncf/common`), which includes all the content distibuted in the `ncf` package

* the Rudder techniques sources (without parameters), which includes all the content distibuted in the `rudder-techniques` package

==== Node-Server communication security

This section gives more details about the different flows between nodes and servers.

===== File copy

File copy is used to get policies and files copied during policy execution (named *shared-files*).

In 4.0 servers, two protocols are available:

* The old protocol (CFEngine "1" protocol), which is plain text by default with the ability to encrypt certain file transfers,

kept for compatibility with older Rudder releases. It is deprecated and will disappear in future releases.

* The new protocol (CFEngine "2" protocol), which is TLS based. Everything is encrypted, still using the CFEngine keys.

Note: The new protocol was already available on the server since Rudder 2.11, but never used by the nodes.

The access policy is:

* Peer to peer key exchange, without central authority

* TOFU (Trust On First Use): keys are sent and accepted at first connection (from the policy server on nodes

and from nodes with an IP in the Allowed Networks on policy servers).

* Node-specific files have an ACL containg the public key of the node, as found in the inventory

* Common files have an IP-based ACL based on the Allowed Networks

The old hostame and IP ACL are still generated for node-specific files to ensure compatibility with older nodes,

but will be removed in the future.

===== Inventory

Nodes send an inventory to the server after installation or upgrade, and once a day.

This inventory contains various information, including:

* The node's public key

* The node's policy server

The inventory security policy is:

* Inventories are sent by the node to its configured policy_server over HTTPS, currently whithout certificate validation.

* Inventories are signed by the node using its private key, which allows the server to check this signature using

the public key coming from previous inventory and to ensure it really comes from the right node.It avoids treating a

malicious (or bogus) inventory coming from another node, and you should check the public key when accepting a new node.

* Once a node has sent a signed inventory, no unsigned inventory will be accepted for this node.

=== Usecases

This chapter gives a few examples for using Rudder. We have no doubt that you'll

have your own ideas, that we're impatient to hear about...

==== Dynamic groups by operating system

Create dynamic groups for each operating system you administer, so that you can

apply specific policies to each type of OS. When new nodes are added to Rudder,

these policies will automatically be enforced upon them.

==== Library of preventive policies

Why not create policies for emergency situations in advance? You can then put

your IT infrastructure in "panic" mode in just a few clicks.

For example, using the provided Techniques, you could create a Name

resolution Directive to use your own internal DNS servers for normal situations,

and a second, alternative Directive, to use Google's public DNS servers, in case

your internal DNS servers are no longer available.

==== Standardizing configurations

You certainly have your own best practices (let's call them good habits) for

setting up your SSH servers.

But is that configuration the same on all your servers? Enforce the settings

your really want using an OpenSSH server policy and apply it to all your Linux

servers. SSH servers can then be stopped or reconfigured manually many times,

Rudder will always restore your preferred settings and restart the SSH server in

less than 5 minutes.

==== Using Rudder as an Audit tool

Using Rudder as an Audit tool is useful if you do not want to make any changes on the system,

temporarily (freeze period, etc.) or permanently.

To use Rudder as an Audit tool without modifying any configuration on your systems,

set the Policy Mode to *Audit* in the Settings, and do not allow overriding.

==== Using Audit mode to validate a policy before applying it

Before applying a configuration policy to some systems (a new policy or a new system),

you can switch the policy mode of the directive defining this policy or of the nodes

it is applied to to *Audit*.

This is particularly useful when adding rules to enforce policies that are supposed to be already applied:

you can measure the gap between expected and actual state, and check what changes would be made before applying them.

== Advanced Node management

=== Node management

==== Reinitialize policies for a Node

To reinitialize the policies for a Node, delete the local copy of the Applied

Policies fetched from the Rudder Server, and create a new local copy of the

initial promises.

----

rudder agent reset

----

At next run of the Rudder Agent (it runs every five minutes), the initial promises will be used.

[CAUTION]

====

Use this procedure with caution: the Applied Policies of a Node should never get

broken, unless some major change has occurred on the Rudder infrastructure, like

a full reinstallation of the Rudder Server.

====

==== Completely reinitialize a Node

You may want to completely reinitialize a Node to make it seen as a new node

on the server, for example after cloning a VM.

[WARNING]

====

This command will permanently delete your node uuid and keys, and no configuration will

be applied before re-accepting and configuring the node on the server.

====

The command to reinitialize a Node is:

----

rudder agent reinit

----

This command will delete all local agent data, including its uuid and keys, and

also reset the agent internal state. The only configuration kept is the server

hostname or ip configured in +policy_server.dat+. It will also send an inventory

to the server, which will treat it as a new node inventory.

==== Change the agent run schedule

By default, the agent runs on all nodes every 5 minutes. You can modify this value in

*Administration* -> *Settings* -> *Agent Run Schedule*, as well as the "splay time"

across nodes (a random delay that alters scheduled run time, intended to spread

load across nodes).

[WARNING]

====

When reducing notably the run interval length, reporting can be in 'No report' state

until the next run of the agent, which can take up to the previous (longer) interval.

====

==== Installation of the Rudder Agent

===== Static files

At installation of the Rudder Agent, files and directories are created in

following places:

+/etc+:: Scripts to integrate Rudder Agent in the system (init, cron).

+/opt/rudder/share/initial-promises+:: Initialization promises for the Rudder

Agent. These promises are used until the Node has been validated in Rudder. They

are kept available at this place afterwards.

+/opt/rudder/lib/perl5+:: The FusionInventory Inventory tool and its Perl

dependencies.

+/opt/rudder/bin/run-inventory+:: Wrapper script to launch the inventory.

+/opt/rudder/sbin+:: Binaries for CFEngine Community.

+/var/rudder/cfengine-community+:: This is the working directory for CFEngine

Community.

===== Generated files

At the end of installation, the CFEngine Community working directory is

populated for first use, and unique identifiers for the Node are generated.

+/var/rudder/cfengine-community/bin/+:: CFEngine Community binaries are copied

there.

+/var/rudder/cfengine-community/inputs+:: Contains the actual working CFEngine

Community promises. Initial promises are copied here at installation. After

validation of the Node, Applied Policies, which are the CFEngine promises

generated by Rudder for this particular Node, will be stored here.

+/var/rudder/cfengine-community/ppkeys+:: An unique SSL key generated for the

Node at installation time.

+/opt/rudder/etc/uuid.hive+:: An unique identifier for the Node is generated

into this file.

===== Services

After all of these files are in place, the CFEngine Community daemons are

launched:

include::../glossary/cf-execd.txt[]

include::../glossary/cf-serverd.txt[]

===== Configuration

At this point, you should configure the Rudder Agent to actually enable the

contact with the server. Type in the IP address of the Rudder Root Server in the

following file:

----

echo *root_server_IP_address* > /var/rudder/cfengine-community/policy_server.dat

----

==== Rudder Agent interactive

You can force the Rudder Agent to run from the console and observe what happens.

----

rudder agent run

----

[CAUTION]

.Error: the name of the Rudder Root Server can't be resolved

====

If the Rudder Root Server name is not resolvable, the Rudder Agent will issue

this error:

----

rudder agent run

Unable to lookup hostname (rudder-root) or cfengine service: Name or service not known

----

To fix it, either you set up the agent to use the IP address of the Rudder root

server instead of its Domain name, either you set up accurately the name

resolution of your Rudder Root Server, in your DNS server or in the hosts file.

The Rudder Root Server name is defined in this file

----

echo *IP_of_root_server* > /var/rudder/cfengine-community/policy_server.dat

----

====

[CAUTION]

.Error: the CFEngine service is not responding on the Rudder Root Server

====

If the CFEngine is stopped on the Rudder Root Server you will get this error:

----

# rudder agent run

!! Error connecting to server (timeout)

!!! System error for connect: "Operation now in progress"

!! No server is responding on this port

Unable to establish connection with rudder-root

----

Restart the CFEngine service:

----

service rudder-agent restart

----

====

==== Processing new inventories on the server

===== Verify the inventory has been received by the Rudder Root Server

There is some delay between the time when the first inventory of the Node is

sent, and the time when the Node appears in the New Nodes of the web interface.

For the brave and impatient, you can check if the inventory was sent by listing

incoming Nodes on the server:

----

ls /var/rudder/inventories/incoming/

----

===== Process incoming inventories

On the next run of the CFEngine agent on Rudder Root Server, the new inventory

will be detected and sent to the Inventory Endpoint. The inventory will be then

moved in the directory of received inventories. The Inventory Endpoint do

its job and the new Node appears in the interface.

You can force the execution of CFEngine agent on the console:

----

rudder agent run

----

===== Validate new Nodes

User interaction is required to validate new Nodes.

===== Prepare policies for the Node

Policies are not shared between the Nodes for obvious security and

confidentiality reasons. Each Node has its own set of policies. Policies are

generated for Nodes according in the following states:

. Node is new;

. Inventory has changed;

. Technique has changed;

. Directive has changed;

. Group of Node has changed;

. Rule has changed;

. Regeneration was forced by the user.

["graphviz", "generate_policy_workflow.png"]

.Generate policy workflow

-------

include::../graphviz/generate_policy_workflow.dot[]

------

==== Agent execution frequency on nodes

===== Checking configuration (CFEngine)

Rudder is configured to check and repair configurations using the CFEngine

agent every 5 minutes, at 5 minutes past the hour, 10 minutes past the hour,

etc.

The exact run time on each machine will be delayed by a random interval, in

order to "smooth" the load across your infrastructure (also known as "splay

time"). This reduces simultaneous connections on relay and root servers (both

for the CFEngine server and for sending reports).

Up to and including Rudder 2.10.x, this random interval is between 0 and 1

minutes. As of Rudder 2.10.x and later, this random interval is between 0 and

5 minutes.

===== Inventory (FusionInventory)

The FusionInventory agent collects data about the node it's running on such as

machine type, OS details, hardware, software, networks, running virtual

machines, running processes, environment variables...

This inventory is scheduled once every 24 hours, and will happen in between

0:00 and 5:00 AM. The exact time is randomized across nodes to "smooth" the

load across your infrastructure.