[rudder-dev] Strenghten the integrity of the node policy if using Relays

Wed Mar 15 16:42:52 CET 2017

Answers inline.

Janos Mattyasovszky

-------- Original Message --------
Subject: Re: [rudder-dev] Strenghten the integrity of the node policy if using Relays
Local Time: 15. März 2017 3:52 PM
UTC Time: 15. März 2017 14:52
From: benoit.peccatte at normation.com
To: Janos Mattyasovszky <mail at matya.hu>
rudder-dev at lists.rudder-project.org <rudder-dev at lists.rudder-project.org>, rudder-users at lists.rudder-project.org <rudder-users at lists.rudder-project.org>

Le 15/03/2017 à 15:37, Janos Mattyasovszky a écrit :

Hi,

In my opinion it is not necessary for the Relay to sign anything not originating directly from that relay. It could of course double-sign it so the node could also verify that there is no any kind of MITM, but if we trust cf-serverd to ensure a secure end-to-end transport of policies, putting on the signature of the relay is just a small benefit, probably not directly worth it. If a Relay would become compromised, it would also mean the private key used to sign anything by the Relay would also be compromised.

The original idea I had in mind is for each Node to validate that the Policy-updates are originating from the Root server, and to ensure no bogus policy could be sneaked in if a Relay is compromised.

If the Relay has the task to distribute Files to Nodes on behalf of the Root server, the Files should be signed by the Root, and the Nodes should trust the Root and verify the authenticity if the Signature belongs to the Root.

I see multiple possible directions you could go:

- X509-Certificates:
The Root-Server would be a Certificate Authority (CA) to sign SSL Certificates for it's Policy Servers

- The Policy Servers' Apache installation (Root and Relays) would get a signed Certificate by this CA:

- The WebDAV endpoints would be secured with this certficate on the Root/Relays

- A newly set up Nodes (not having set up any out-of-band trust before) could fetch the Certificate-Chain from the WebDAV endpoint directly at the time they are preparing the first inventory upload and could establish initial trust to that (since the Webserver is actually sending it's SSL Chain to the Client)

- The Root Server could use the Private Key for the CA to sign any Policy intended for an accepted Node.

- The Node could verify the authenticity of Policy updates belonging to the same "instance" it established trust initially (via the SSL Certificate Chain) and would only accept inventory updates if a checksum file of all updated files is signed by the RootCA.

- Pros:

- You make sure all communication over TLS (WebDAV) is secured properly and trust is established the first time a Node connects to a WebDAV endpoint of the Server in it's policy_server.dat.

- It would solve trust of Web-Transport and also File-Signature by only trusting one "public key", retrievable only with the knowledge of the policy server by standard HTTPS-Protocol.

You still need 2 trusts, since the transport over https is done by communicating with the relay, you also must trust it.

Well, actually not, this is the major benefit of the PKI Structure: You need to trust the Issuer-CA which issues the End-Certificate.
You are not trusting www.rudder-project.org directly (which is "s:/OU=Domain Control Validated/OU=Gandi Standard Wildcard SSL/CN=*.rudder-project.org)", but the issuer up in the chain, which is: "/C=FR/ST=Paris/L=Paris/O=Gandi/CN=Gandi Standard SSL CA 2". And you can fetch the public certificate of the issuer via openssl up to the top, which is always a self-signed cert, because it is (or should) sent by the webserver so the client can validate the certificates up the chain.

The only difference is, on the regular web, you need a pre-defined trust to the Root Certificates, which come by Chrome / Firefox / IE / Opera / whatever, and with this solution, you'd basically download the whole chain up on the first time you are "unregistered", and trust that but that only chain:

> openssl s_client -showcerts -connect www.rudder-project.org:443 < /dev/null
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = FR, ST = Paris, L = Paris, O = Gandi, CN = Gandi Standard SSL CA 2
verify return:1
depth=0 OU = Domain Control Validated, OU = Gandi Standard Wildcard SSL, CN = *.rudder-project.org
verify return:1
---
Certificate chain
0 s:/OU=Domain Control Validated/OU=Gandi Standard Wildcard SSL/CN=*.rudder-project.org
i:/C=FR/ST=Paris/L=Paris/O=Gandi/CN=Gandi Standard SSL CA 2
-----BEGIN CERTIFICATE-----
MIIFDjCCA/agAwIBAgIQETQ0aAhHLuYl/W+5I3hZaDANBgkqhkiG9w0BAQsFADBf
[...]
NrHviM6Ah3FE+usyVuziNwAJsBUL0wLOfFOaZ7Eeupve2oM6Y6Lcr1k2MIokS740
YLc=
-----END CERTIFICATE-----
1 s:/C=FR/ST=Paris/L=Paris/O=Gandi/CN=Gandi Standard SSL CA 2
i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
-----BEGIN CERTIFICATE-----
MIIF6TCCA9GgAwIBAgIQBeTcO5Q4qzuFl8umoZhQ4zANBgkqhkiG9w0BAQwFADCB
[...]
BT02Vf6Dsuimrdfp5gJ0iHRc2jTbkNJtUQoj1iM=
-----END CERTIFICATE-----
2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
-----BEGIN CERTIFICATE-----
MIIFdzCCBF+gAwIBAgIQE+oocFv07O0MNmMJgGFDNjANBgkqhkiG9w0BAQwFADBv
[...]
0fKtirOMxyHNwu8=
-----END CERTIFICATE-----

=> this is returned by querying www.rudder-project.org:443, but it also returns the Certificate Chain up to the last intermediate CA, I presume because the SSLCertificateChainFile does not include the actual RootCA's PEM, only the intermediates, but according to the Apache docu, "This starts with the issuing CA certificate of the server certificate and can range up to the root CA certificate.", so including also the RootCA's PEM would be possible.

If this would be a Rudder PolicyServer, and the Chain would include the RootCA's PEM as well, and a Node would be initially have a policy_server set and the first agent run started, it could download this chain of trust, and establish a trust-relationshit with the RootCA within the Chain, regardless of it is a Relay or the RootServer itself, or even if it would be a multi-level chained Relay...

To mention it again: The whole part of this is that the Certificate of the RootCA is also to be sent by the Webserver, so you would only have to find out which one of the returned certificates is the RootCA and trust that on first encounter, regardless what the returned certificate of a Relay is.

Hope you could follow it :-)

- Cons:

- File-Signing and -Verification with X509 is not easy and also not designed for, but within the scope of possible

- Rudder-Managing the SSL-Certificates requires all hostnames / frontends to precisely match of what Rudder thinks it's called

Since we would use client certificates only for signature and not for TLS, we can put any name we want in them, and in our case i would prefer using their uuid instead of their hostname.
With 4.1 we added relay API, and now every relay already have the information of which hosts has which key behind itself.

Well, yes, the only benefit of also managig the certificates for TLS would be that you could use the Chain to determine the RootCA's public cert. If you would not manage them, then you'd have to provide a way to download the RootCA's PEM from the Relay-Endpoint (just like /uuid it would be something like /rootcert).

- If a Node connects to a compromised Relay, it will trust that (there is actually no good way to trust something you know initially nothing of)

Except if you have deployed a CA on the relay and a relay certificate signed by this CA.

There is no way for a Node on initial connect to determine if a CA actually belongs to the real Root-Server, or a CA is faked on the Relay, the incoming policy updates then re-signed by this Relay-faked-CA, and propagated as the real RootCA-signed once. You could only determine if the signature would change after the first trust is established, or if the trusted CA would be put in place before the first connection to the Relay is made (just like on regular webbrowsers). But this is always a problem by estalishing trust on first use, like accepting SSH pubkeys or ppkeys for cfengine etc...

- GPG Trust:
The Root-Server would have a private GPG Key to a publicly propagated Signature

- The Policy Server propagates the signature of the Root Server's GPG Key via HTTP(s) to anybody asking for it (just like /uuid).

- The Nodes would query the GPG Key's Signature in addition to the uuid of it's policy server and trust it the first time they retrieve it.

- The Nodes would verify any Policy-Update with this GPG Key for authenticity

- Pros

- GPG is very mature and has good tooling if it comes to File encryption / verification

openssl also has the feature

Well, it has something, but I'd not call it suitable for secure encryption / verification, as it lacks a good KDF and it is not used primarily for that, and one should not experiment with security.
This comment might bring some deeper details on that topic:
https://security.stackexchange.com/questions/29106/openssl-recover-key-and-iv-by-passphrase/29139#29139

-
If would also enable you later to use a GPG-Based solution to encrypt Files for dedicated nodes, so only they can open it
(if the Nodes were to generate a private GPG key and send the public key in the initial inventory)
- Cons:

- The Trust of the Web-Endpoints are still subject for the User to provide valid certificates (btw, currently the curl has "-k", so that's not really a "trusted" channel anyway)

- The public part has to be propagated in a way.

The question becomes of course a lot trickier if you want to share files without involving the Root between Nodes and want the Receiver Node to have the ability to verify the authenticity of the received files without the direct knowledge of any cryptographic identifies of the Sender Node :-/ ... Not sure about that use case, but I think that if you want to make things secure, you have to first establish some kind of trust relationship with the Root-Instance, and then let the Root-Instance distribute any further knowledge about possible inter-Node trust relationship ...

Thanks for reading,

--
Janos Mattyasovszky

-------- Original Message --------
Subject: Re: [rudder-dev] Strenghten the integrity of the node policy if using Relays
Local Time: 15. März 2017 12:20 PM
UTC Time: 15. März 2017 11:20
From: benoit.peccatte at normation.com
To: Janos Mattyasovszky [<mail at matya.eu>](mailto:mail at matya.eu), rudder-dev at lists.rudder-project.org [<rudder-dev at lists.rudder-project.org>](mailto:rudder-dev at lists.rudder-project.org), rudder-users at lists.rudder-project.org [<rudder-users at lists.rudder-project.org>](mailto:rudder-users at lists.rudder-project.org)

Le 14/03/2017 à 10:16, Janos Mattyasovszky a écrit :

Hi dear Rudder Community,

The issue:
The policy generated by the Root server is transmitted encrypted via the Relay servers, but this provides only transport encryption between the endpoints, and the Relays basically are by-design MITM hosts, which have the ability to modify policy files and reports going back through them (the inventories are signed - so they would break). This requires that every relay has a high need for integrity, since there is no real way to determine from a Rudder-Root-Server point of view if any of the relay behaves rogue and injects bogus policy and modifies the reports stream back to represent that all nodes are good, even if they are not and are executing an attacker-provided modified policy.

Proposed solution:
Use cryptographic signature on the generated policy with the Root Server's RSA key.

With the usage of PKI a client can validate the policy received from the Master before executing it by trusting the public key of it. This would require the pubkey of the Root Server to be known to the Nodes. Currently if you have any Relays in between, they become the effective policy server for the nodes, and the nodes will not know anything about the Relay not being the root server (they just behave identical as if they would be connected to a root server in the POW of an end-of-the-leaf node).

Does that mean you think the signature should be done by the relay ?

By using a logic like "trust on first use", where the root server includes it's pubkey in any policy being generated, and then the node would trust the first key that it would receive if it has no policy yet, it could establish a trust until a "rudder agent reset/reinit" would be issued. After that the node could verify any further policy by checking the signature of a file containing the hashes of all the policy files.

This would work as long the nodes are not connecting initially to a compromised relay, or if the Pubkey of the Root Server is also deployed out-of-band at the time the rudder-agent package is installed and policy_server.dat is configured, so basically the node has already an initial knowledge of the root server's pubkey, and would as of that only trust policy signed by that root server, regardless of the path the policy would travel.

Distributing the public key out of band is a possibility, distributing a CA and checking signature may be better for long term key management.
This CA could be managed locally by rudder or be managed by Normation for its clients.

This would raise the overall security level and reducing the criticality of a relay to "only" require confidentiality, since any compromise would result in worst case the nodes behind a relay not executing the compromised policy and if the relay was faking the expected reports the nodes would have to send through the relays, so we'd go from "compromising all nodes below the relay to execute our code" down to "cutting off the nodes from any new policy update without being detected by the Root server", which is still a great improvement, and if you have out-of-Rudder monitoring for policy updates (#7282), you could detect this by having nodes not receiving policy updates as scheduled.

A second step could be not to send the reports via unencrpyted UDP Syslog, but use the same method as sending the inventories: one file with the current run's reports, signed by the node's key, this would also solve the issue of not being able to detect any compromised relay.

We took a step forward having signature everywhere, since the 4.1 we have a relay api to share files between hosts. This API uses the same signature mechanism as inventories that is checked everywhere.
This API could also be used in place of syslog to transmit signed reporting to the server.

Thanks for reading,

Best Regards,

Janos Mattyasovszky

_______________________________________________
rudder-users mailing listrudder-users at lists.rudder-project.orghttp://www.rudder-project.org/mailman/listinfo/rudder-users

--

---------------------------------------------------------------

[Logo Normation]Benoît Peccatte
Architecte
[Normation](http://www.normation.com)
---------------------------------------------------------------

87, Rue de Turbigo, 75003 Paris, France

Phone:
+33 (0)1 85 08 48 96

---------------------------------------------------------------

--

---------------------------------------------------------------

[Logo Normation]Benoît Peccatte
Architecte
[Normation](http://www.normation.com)
---------------------------------------------------------------

87, Rue de Turbigo, 75003 Paris, France

Phone:
+33 (0)1 85 08 48 96

---------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.rudder-project.org/pipermail/rudder-dev/attachments/20170315/f5eaac05/attachment-0001.html>