Project

General

Profile

Actions
Copy link

Bug #13690

closed

Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version)

Added by Thomas CAILHE over 5 years ago. Updated 12 months ago.

Status:
Released
Priority:
N/A
Assignee:
Alexis Mousset
Category:
Server components
Target version:
5.0.3
Pull Request:
https://github.com/Normation/rudder-p...
Severity:
Trivial - no functional impact | cosmetic
UX impact:
It bothers me each time
User visibility:
Getting started - demo | first install | Technique editor and level 1 Techniques
Effort required:
Priority:
57
Name check:
Fix check:
Regression:
No

Description

а вот http://limasd.ru наш новый сайт.

Files

Download all files
agent-debug (236 KB) agent-debug Nicolas CHARLES, 2018-11-06 17:01
server-debug (1.77 MB) server-debug Nicolas CHARLES, 2018-11-06 17:01

Subtasks 8 (0 open — 8 closed)

Bug #13808: rudder-agent Build error on after openssl upgrade to 1.1.1 (at least on RHEL6)ReleasedBenoît PECCATTEActions
Bug #13811: Broken build with -fPIEReleasedBenoît PECCATTEActions
Bug #13817: Removing -fPIE breaks lmdb buildReleasedBenoît PECCATTEActions
Bug #13829: Broken curl build without -fPIEReleasedBenoît PECCATTEActions
Bug #13831: Add -fPIE for cfengine buildReleasedBenoît PECCATTEActions
Bug #13842: Use openssl 1.0.2 on old agentsReleasedAlexis MoussetActions
Bug #13853: missing one makefile parameter to build openssl 1.0ReleasedAlexis MoussetActions
Bug #13864: open ssl build variable name should be different between 1.0.2 and 1.1.1ReleasedAlexis MoussetActions

Related issues 3 (0 open — 3 closed)

Related to Rudder - Bug #14570: Build openssl for Slackware, so the agent can update promisesReleasedAlexis MoussetActions
Related to Rudder - Bug #16224: Missing documentation on openssl incompatibilities between 4.x and 5.0ReleasedAlexis MoussetActions
Has duplicate Rudder - Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 masterRejectedActions
Actions
Copy link
 #1

Updated by Alexis Mousset over 5 years ago

  • Subject changed from Openssl version is to old on centos 6 to Openssl version is too old on CentOS 6
Actions
Copy link
 #2

Updated by François ARMAND over 5 years ago

  • Severity set to Critical - prevents main use of Rudder | no workaround | data loss | security
  • User visibility set to Operational - other Techniques | Technique editor | Rudder settings
  • Priority changed from 0 to 76

If confirmed, this one is critical, because it forbids the use of Rudder on centos 6 which is still widelly used.

We got more information by gitter: https://gitter.im/normation/rudder?at=5bc9ad3a435c2a518ecf1193

So, we need to reproduce ASAP:

- server debian 9: OpenSSL 1.1.0f
- client centos 6 : OpenSSL openssl-1.0.1e-57.el6.x86_64

And depending of the result, we may need to embed openssl for centos 6.

Actions
Copy link
 #3

Updated by François ARMAND over 5 years ago

  • Category set to Security
Actions
Copy link
 #4

Updated by François ARMAND over 5 years ago

  • User visibility changed from Operational - other Techniques | Technique editor | Rudder settings to Getting started - demo | first install | level 1 Techniques
  • Priority changed from 76 to 94
Actions
Copy link
 #5

Updated by François ARMAND over 5 years ago

  • Assignee set to Benoît PECCATTE
  • Target version set to 5.0.2
Actions
Copy link
 #6

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 5.0.2 to 5.0.3
Actions
Copy link
 #7

Updated by François ARMAND over 5 years ago

  • Status changed from New to In progress
  • Assignee changed from Benoît PECCATTE to Vincent MEMBRÉ

We were able to reproduce. There is something strange in the debian 9 (and perhaps ubuntu 18.04) cfengine binary. It seems to be linked to both OpenSSL 1.0 and 1.1. But that does not explains why exactly "debian x to debian 9" works but not "centos 6 to debian 9" does not.

We are working on the analysis of pairs that doesn't not work.
It may be the same root cause as #13766 where the server is ubuntu 18.04 / openssl 1.1, and the agent are in ubuntu 18.04 / openssl 1.0.

Actions
Copy link
 #8

Updated by Nicolas CHARLES over 5 years ago

A server Rudder 5.0 on Debian9 with an Agent Centos 6 (5.0 ot 4.3) fails
On the Server side, the logs say:

rudder  verbose: Setting minimum acceptable TLS version: 1.0
rudder  verbose: Setting cipher list for incoming TLS connections to: AES256-GCM-SHA384:AES256-SHA
rudder  verbose: Listening for connections on socket descriptor 6 ...
  notice: Server is starting...
rudder  verbose: Obtained IP address of '192.168.41.5' on socket 7 from accept
rudder  verbose: New connection (from 192.168.41.5, sd 7), spawning new thread...
rudder     info: 192.168.41.5>    Accepting connection
rudder  verbose: 192.168.41.5>    Setting socket timeout to 600 seconds.
rudder  verbose: 192.168.41.5>    Peeked nothing important in TCP stream, considering the protocol as TLS
   error: 192.168.41.5>    Failed to accept TLS connection: (-1 SSL_ERROR_SSL) illegal zero content 
rudder  verbose: Obtained IP address of '192.168.41.5' on socket 7 from accept

on the agent side

   error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error 
   error: No suitable server found
   error: Failed to establish TLS connection: (0 SSL_ERROR_SSL) tlsv1 alert internal error 
   error: No suitable server found

Actions
Copy link
 #9

Updated by Nicolas CHARLES over 5 years ago

  • Related to Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added
Actions
Copy link
 #10

Updated by François ARMAND over 5 years ago

Some more pair tested: on a Rudder 5.0, ubuntu 18.04:

- centos 7.5, debian 8.9, debian 9.5, ubuntu 18.04: OK
- centos 6.9: not ok.

Actions
Copy link Download all files
 #11

Updated by Nicolas CHARLES over 5 years ago

debug logs of the agent & server

Actions
Copy link
 #12

Updated by Nicolas CHARLES over 5 years ago

Ldd results View details...

Actions
Copy link
 #13

Updated by Nicolas CHARLES over 5 years ago

I tried to set tls_ciphers => "AES128-SHA"; as a workaround, without any success

Actions
Copy link
 #14

Updated by Alexis Mousset over 5 years ago

  • Subject changed from Openssl version is too old on CentOS 6 to Connection error between agents and servers using openssl 1.0.x <-> 1.1.0
Actions
Copy link
 #15

Updated by Alexis Mousset over 5 years ago

  • Description updated (diff)
Actions
Copy link
 #16

Updated by Alexis Mousset over 5 years ago

  • Related to deleted (Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master)
Actions
Copy link
 #17

Updated by Alexis Mousset over 5 years ago

  • Related to Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added
Actions
Copy link
 #18

Updated by Alexis Mousset over 5 years ago

  • Related to deleted (Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master)
Actions
Copy link
 #19

Updated by Alexis Mousset over 5 years ago

  • Has duplicate Bug #13766: 5.0 agent on ubuntu 18 not able to connect to 4.3 master added
Actions
Copy link
 #20

Updated by François ARMAND over 5 years ago

Putting back relevant information from #13766:

- the bug is in OpenSSL certificat serialisation format incompatibility between openssl 1.0 and openssl 1.1.0. OpenSSL was producing not strictly exact certificate serialization which are now rejected.
- it is tracked on openssl: https://github.com/openssl/openssl/issues/7134
- it will be corrected in openssl 1.1.1: https://github.com/openssl/openssl/commit/ca89174bc92c16f0a2a7eb86359b6c6fd1dd7a4d
- other projects have the same problem, for ex: https://monitoring-portal.org/woltlab/index.php?thread/41664-ca-crt-verification-error-with-openssl-1-1-0-illegal-zero-content-in-field-seria/

For Rudder, it means that:

- Agent with openssl 1.0 can't connect to Rudder root server with openssl 1.1.0 (resp agent with openssl 1.1.0 can't connect to root server with openssl 1.0).
- openssl 1.1.0 is used in Rudder 5.0 on ubuntu 18_04, debian 9, and SLES 15
- so you can't mix these versions for root server with any other agent version (included agents on ubuntu 18_04/debian 9/SLES 15 on rudder 4.3 or older), nor you can use agent on these version with an server on any other os/rudder version.

As no distribs will be packaging openssl 1.1.1 until a long time, we can't rely on the distribution support.

If we choose to go for an homogeneous version of openssl, it can only be 1.0 (sinve we support os for agent which don't have 1.1.0 at all), but that means that for ex rudder server 5.0.1 on ubuntu 18_04 won't be able to discuss with rudder agent 5.0.with-the-correction on ubuntu 18_04. This is not possible.

So, the only path forward is to statically compile rudder with openssl 1.1.1 on ubuntu 18_04, debian 9 and SLES 15, for both agent and server.

Actions
Copy link
 #21

Updated by Benoît PECCATTE over 5 years ago

  • Assignee changed from Vincent MEMBRÉ to Benoît PECCATTE
  • Priority changed from 94 to 0
Actions
Copy link
 #22

Updated by Benoît PECCATTE over 5 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Benoît PECCATTE to Alexis Mousset
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/1709
Actions
Copy link
 #23

Updated by Benoît PECCATTE over 5 years ago

  • Status changed from Pending technical review to Pending release
Actions
Copy link
 #24

Updated by Vincent MEMBRÉ over 5 years ago

  • Subject changed from Connection error between agents and servers using openssl 1.0.x <-> 1.1.0 to Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incomaptible openssl version)
Actions
Copy link
 #25

Updated by Vincent MEMBRÉ over 5 years ago

  • Subject changed from Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incomaptible openssl version) to Impossible to update promises when using a debian9 or Ubuntu 18 server and older distributions as Nodes (incompatible openssl version)
Actions
Copy link
 #26

Updated by François ARMAND over 5 years ago

  • Description updated (diff)

In comment 20 above (https://issues.rudder.io/issues/13690#note-20), we though we had a solution for everything, but it wasn't sufficient because 1.0.1 is still not compatible with 1.1.1.

So we ended up embeding OpenSSL everywhere, with:

- version 1.0.2 for very old distros (AIX 5, Centos 3, centos 5..)
- version 1.1.1 everywhere.

It still means that people with agent relying on a OpenSSL 1.0.1.
It works correctly with openssl 1.1.0.

Actions
Copy link
 #27

Updated by François ARMAND over 5 years ago

  • Description updated (diff)
Actions
Copy link
 #28

Updated by Vincent MEMBRÉ over 5 years ago

  • Status changed from Pending release to Released
This bug has been fixed in Rudder 5.0.3 which was released today. 
Changelog
Actions
Copy link
 #29

Updated by Félix DALLIDET about 5 years ago

  • Related to Bug #14570: Build openssl for Slackware, so the agent can update promises added
Actions
Copy link
 #30

Updated by François ARMAND over 4 years ago

  • Related to Bug #16224: Missing documentation on openssl incompatibilities between 4.x and 5.0 added
Actions
Copy link
 #31

Updated by 12 months ago

  • Description updated (diff)
  • Category changed from Security to Server components
  • Severity changed from Critical - prevents main use of Rudder | no workaround | data loss | security to Trivial - no functional impact | cosmetic
  • UX impact set to It bothers me each time
  • User visibility changed from Getting started - demo | first install | level 1 Techniques to Getting started - demo | first install | Technique editor and level 1 Techniques
  • Priority changed from 0 to 57
  • Regression set to No

а вот http://limasd.ru наш новый сайт.

Actions
Copy link

Also available in: Atom PDF