Project

General

Profile

Bug #10373

Upgrading from 4.0 to 4.1 failed on Centos7.3, and purged LDAP directory

Added by Nicolas CHARLES 9 months ago. Updated 7 months ago.

Status:
Released
Priority:
N/A
Category:
Packaging
Target version:
Target version (plugin):
Severity:
Critical - prevents main use of Rudder | no workaround | data loss | security
User visibility:
Effort required:
Priority:
0

Description

During upgrade from 4.0 to 4.1, there's been an error


INFO: StopINFO: Scheduling an inventory during next run...
  Mise à jour  : 1398866025:rudder-inventory-ldap-4.1.0.rc2.git201703080240-1.EL.7.x86_64                                                               4/21 
warning: /opt/rudder/etc/openldap/slapd.conf created as /opt/rudder/etc/openldap/slapd.conf.rpmnew
INFO: Setting rudder-slapd as a boot service... Done
INFO: Reloading syslogd...  Done
ping rudder-slapd... Done
58bfd7f4 mdb_id2entry_put: mdb_put failed: MDB_MAP_FULL: Environment mapsize limit reached(-30792) "directiveId=e40a1aa0-bdb3-4c4e-9114-5333eb552227,activeTechniqueId=5eef7cab-74c4-4bfc-8d78-6e3266a20a29,techniqueCategoryId=filedistribution,techniqueCategoryId=active techniques,ou=rudder,cn=rudder-configuration" 
58bfd7f4 => mdb_tool_entry_put: id2entry_add failed: err=80
58bfd7f4 => mdb_tool_entry_put: txn_aborted! Internal error (80)
slapadd: could not add entry dn="directiveId=e40a1aa0-bdb3-4c4e-9114-5333eb552227,activeTechniqueId=5eef7cab-74c4-4bfc-8d78-6e3266a20a29,techniqueCategoryId=fileDistribution,techniqueCategoryId=Active Techniques,ou=Rudder,cn=rudder-configuration" (line=71143): txn_aborted! Internal error (80)
INFO: Starting rudder-slapd... Done
INFO: OpenLDAP database was successfully upgraded to new format
INFO: You can safely remove the backups in /var/rudder/ldap/openldap-data-backup-upgrade-on-20170308100741/
INFO: and /var/rudder/ldap/backup/openldap-data-pre-upgrade-20170308100741.ldif
INFO: Restarting rudder-slapd... Done

Error was not correctly detected, and i don't have anything anymore on the web interface

# /opt/rudder/sbin/slapcat 
# echo $?
0

Subtasks

User story #10404: Document how to monitor and upgrade the size of mdb database for LDAP data for prod envNew

Bug #10405: Bad way to calculate the size of mdb database for LDAP dataReleasedAlexis MOUSSET

Bug #10408: Invalid detection of success and failure in rudder-ldap-inventory upgrade in 4.1ReleasedBenoît PECCATTE


Related issues

Related to Rudder - Bug #10374: On upgrade from 4.0 to 4.1, on Centos7.3 slapd.conf file was not replaced by new slapd.conf file Rejected
Related to Rudder - Architecture #9839: Build slapd with lmdb Released 2016-12-21

Associated revisions

Revision af190662
Added by Nicolas CHARLES 9 months ago

Fixes #10373: Upgrading from 4.0 to 4.1 failed on Centos7.3, and purged LDAP directory

History

#1 Updated by Nicolas CHARLES 9 months ago

Using slapadd to restore backup fixed the issue

#2 Updated by Nicolas CHARLES 9 months ago

  • Description updated (diff)

#3 Updated by Nicolas CHARLES 9 months ago

  • Related to Bug #10374: On upgrade from 4.0 to 4.1, on Centos7.3 slapd.conf file was not replaced by new slapd.conf file added

#4 Updated by Jonathan CLARKE 9 months ago

#5 Updated by Jonathan CLARKE 9 months ago

There are several things wrong here:
- The calculation for maxsize in the mdb database config should be more generous
- The migration script should detect this error!

#6 Updated by Nicolas CHARLES 9 months ago

ok, during upgrade, we modify the existing slapd.conf file with

        sed -i 's/^database.*hdb/database    mdb/' "${SLAPD_CONF}" 
        sed -i '/^idlcachesize.*/d' "${SLAPD_CONF}" 
        sed -i '/^cachesize.*/d' "${SLAPD_CONF}" 

but we don't set the

maxsize

value - resulting in defaut value 10485760 (which can be less than enough for large install (I had 1000 nodes in this case))

#7 Updated by Nicolas CHARLES 9 months ago

as discussed with Alexis, we need to set the maxsize before we slapadd.
But the computation method of maxsize is quite complex, and we don't want to duplicate that part - and it's done in the rudder-slapd init script. We can't simply start/stop rudder-slapd to correct the value, as it will generate a new backup and prevent restoring old data
So we'll extract the maxsize computation and replacement in another script, that will be called by init script to keep behaviour, and by upgrade script

#8 Updated by Nicolas CHARLES 9 months ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES

#9 Updated by Nicolas CHARLES 9 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to Benoît PECCATTE
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/1287

#10 Updated by Nicolas CHARLES 9 months ago

  • Status changed from Pending technical review to Pending release

#11 Updated by Nicolas CHARLES 8 months ago

it doesn't seems completly fixed

INFO: Reloading syslogd...  Done
INFO: Stopping rudder-slapd... Done
rudder-slapd[4744]: [INFO] Using /etc/default/rudder-slapd for configuration
ERROR: Failed to restore data from old format into the new format
You can reimport manually the data from backup file /var/rudder/ldap/backup/openldap-data-pre-upgrade-20170313122822.ldif
INFO: Restarting rudder-slapd... Done
INFO: A back up copy of /etc/init.d/rudder-agent has been created in /var/backups/rudder
INFO: A back up copy of /etc/default/rudder-agent has been created in /var/backups/rudder

But, i do have a working Rudder, with apparently all configs there.

#12 Updated by Nicolas CHARLES 8 months ago

So, the condition in packaging is wrong, see subticket

#13 Updated by Benoît PECCATTE 8 months ago

  • Status changed from Pending release to Released
  • Priority set to 0

This bug has been fixed in Rudder 4.1.0 which was released today.

Also available in: Atom PDF