Project

General

Profile

Actions

Bug #10373

closed

Upgrading from 4.0 to 4.1 failed on Centos7.3, and purged LDAP directory

Added by Nicolas CHARLES about 7 years ago. Updated about 2 years ago.

Status:
Released
Priority:
N/A
Category:
Packaging
Target version:
Severity:
Critical - prevents main use of Rudder | no workaround | data loss | security
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Fix check:
Regression:

Description

During upgrade from 4.0 to 4.1, there's been an error


INFO: StopINFO: Scheduling an inventory during next run...
  Mise à jour  : 1398866025:rudder-inventory-ldap-4.1.0.rc2.git201703080240-1.EL.7.x86_64                                                               4/21 
warning: /opt/rudder/etc/openldap/slapd.conf created as /opt/rudder/etc/openldap/slapd.conf.rpmnew
INFO: Setting rudder-slapd as a boot service... Done
INFO: Reloading syslogd...  Done
ping rudder-slapd... Done
58bfd7f4 mdb_id2entry_put: mdb_put failed: MDB_MAP_FULL: Environment mapsize limit reached(-30792) "directiveId=e40a1aa0-bdb3-4c4e-9114-5333eb552227,activeTechniqueId=5eef7cab-74c4-4bfc-8d78-6e3266a20a29,techniqueCategoryId=filedistribution,techniqueCategoryId=active techniques,ou=rudder,cn=rudder-configuration" 
58bfd7f4 => mdb_tool_entry_put: id2entry_add failed: err=80
58bfd7f4 => mdb_tool_entry_put: txn_aborted! Internal error (80)
slapadd: could not add entry dn="directiveId=e40a1aa0-bdb3-4c4e-9114-5333eb552227,activeTechniqueId=5eef7cab-74c4-4bfc-8d78-6e3266a20a29,techniqueCategoryId=fileDistribution,techniqueCategoryId=Active Techniques,ou=Rudder,cn=rudder-configuration" (line=71143): txn_aborted! Internal error (80)
INFO: Starting rudder-slapd... Done
INFO: OpenLDAP database was successfully upgraded to new format
INFO: You can safely remove the backups in /var/rudder/ldap/openldap-data-backup-upgrade-on-20170308100741/
INFO: and /var/rudder/ldap/backup/openldap-data-pre-upgrade-20170308100741.ldif
INFO: Restarting rudder-slapd... Done

Error was not correctly detected, and i don't have anything anymore on the web interface

# /opt/rudder/sbin/slapcat 
# echo $?
0

Subtasks 2 (0 open2 closed)

Bug #10405: Bad way to calculate the size of mdb database for LDAP dataReleasedAlexis MoussetActions
Bug #10408: Invalid detection of success and failure in rudder-ldap-inventory upgrade in 4.1ReleasedBenoît PECCATTEActions

Related issues 3 (1 open2 closed)

Related to Rudder - Bug #10374: On upgrade from 4.0 to 4.1, on Centos7.3 slapd.conf file was not replaced by new slapd.conf fileRejectedActions
Related to Rudder - Architecture #9839: Build slapd with lmdbReleasedAlexis Mousset2016-12-21Actions
Related to Rudder - Enhancement #10404: Document how to monitor and upgrade the size of mdb database for LDAP data for prod envNewActions
Actions #1

Updated by Nicolas CHARLES about 7 years ago

Using slapadd to restore backup fixed the issue

Actions #2

Updated by Nicolas CHARLES about 7 years ago

  • Description updated (diff)
Actions #3

Updated by Nicolas CHARLES about 7 years ago

  • Related to Bug #10374: On upgrade from 4.0 to 4.1, on Centos7.3 slapd.conf file was not replaced by new slapd.conf file added
Actions #4

Updated by Jonathan CLARKE about 7 years ago

Actions #5

Updated by Jonathan CLARKE about 7 years ago

There are several things wrong here:
- The calculation for maxsize in the mdb database config should be more generous
- The migration script should detect this error!

Actions #6

Updated by Nicolas CHARLES about 7 years ago

ok, during upgrade, we modify the existing slapd.conf file with

        sed -i 's/^database.*hdb/database    mdb/' "${SLAPD_CONF}" 
        sed -i '/^idlcachesize.*/d' "${SLAPD_CONF}" 
        sed -i '/^cachesize.*/d' "${SLAPD_CONF}" 

but we don't set the

maxsize

value - resulting in defaut value 10485760 (which can be less than enough for large install (I had 1000 nodes in this case))

Actions #7

Updated by Nicolas CHARLES about 7 years ago

as discussed with Alexis, we need to set the maxsize before we slapadd.
But the computation method of maxsize is quite complex, and we don't want to duplicate that part - and it's done in the rudder-slapd init script. We can't simply start/stop rudder-slapd to correct the value, as it will generate a new backup and prevent restoring old data
So we'll extract the maxsize computation and replacement in another script, that will be called by init script to keep behaviour, and by upgrade script

Actions #8

Updated by Nicolas CHARLES about 7 years ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
Actions #9

Updated by Nicolas CHARLES about 7 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to Benoît PECCATTE
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/1287
Actions #10

Updated by Nicolas CHARLES about 7 years ago

  • Status changed from Pending technical review to Pending release
Actions #11

Updated by Nicolas CHARLES about 7 years ago

it doesn't seems completly fixed

INFO: Reloading syslogd...  Done
INFO: Stopping rudder-slapd... Done
rudder-slapd[4744]: [INFO] Using /etc/default/rudder-slapd for configuration
ERROR: Failed to restore data from old format into the new format
You can reimport manually the data from backup file /var/rudder/ldap/backup/openldap-data-pre-upgrade-20170313122822.ldif
INFO: Restarting rudder-slapd... Done
INFO: A back up copy of /etc/init.d/rudder-agent has been created in /var/backups/rudder
INFO: A back up copy of /etc/default/rudder-agent has been created in /var/backups/rudder

But, i do have a working Rudder, with apparently all configs there.

Actions #12

Updated by Nicolas CHARLES about 7 years ago

So, the condition in packaging is wrong, see subticket

Actions #13

Updated by Benoît PECCATTE almost 7 years ago

  • Status changed from Pending release to Released
  • Priority set to 0

This bug has been fixed in Rudder 4.1.0 which was released today.

Actions #14

Updated by François ARMAND about 2 years ago

  • Related to Enhancement #10404: Document how to monitor and upgrade the size of mdb database for LDAP data for prod env added
Actions

Also available in: Atom PDF