[rudder-users] LDAP issue
Francois Armand
francois.armand at normation.com
Wed Mar 5 14:05:53 CET 2014
Hello,
What you are reporting is very strange, and you are the first one to
encounter all these problem with the LDAP back-end, so I suspect that
something want really bad when the /var was full, and lead to some kind
of corruption of some part of LDAP storage (likelly BDB-related).
All the problem you are seing seems to be linked with a BDB corruption.
As soon as the coruption happen, everything starts to fall appart.
I'm fearing that you will have to delete data in base and restart from a
back-up with the following process:
|/etc/init.d/slapd stop
/etc/init.d/slapd backup
#remove corrupted data
rm -f alock __db.* *.bdb log.*
#restore your backup
slapadd -l /var/rudder/ldap/backup/openldap-data-20140305xxxxx.ldif
#restart
/etc/init.d/slapd start
|
Hope it helps !
On 05/03/2014 11:48, Prestasit01 wrote:
>
> Hello all,
>
>
>
> I retried this morning (recover & reindex), then reload techniques +
> dynamic groups + clear caches :
>
>
>
>
>
>
>
> But after that my rules / nodes / groups / directives are not seen,
> Here is the errors seen in the logs :
>
>
>
> ….
>
> ….
>
> Mar 5 11:09:31 SERVERPROD slapd[1372]: connection_input: conn=1050
> deferring operation: too many executing
>
> Mar 5 11:09:32 SERVERPROD slapd[1372]: connection_input: conn=1050
> deferring operation: too many executing
>
> Mar 5 11:09:33 SERVERPROD slapd[1372]: connection_input: conn=1050
> deferring operation: too many executing
>
> Mar 5 11:09:36 SERVERPROD slapd[1372]: bdb(cn=rudder-configuration):
> id2entry.bdb page 91 is on free list with type 7
>
> Mar 5 11:09:36 SERVERPROD slapd[1372]: bdb(cn=rudder-configuration):
> PANIC: Invalid argument
>
> Mar 5 11:09:36 SERVERPROD slapd[1372]: bdb(cn=rudder-configuration):
> PANIC: fatal region error detected; run recovery
>
> Mar 5 11:09:36 SERVERPROD slapd[1372]: bdb(cn=rudder-configuration):
> PANIC: fatal region error detected; run recovery
>
> ….(loop of that last line)
>
> ….
>
>
>
> If i reboot my server (not only rudder-server-root services but the OS) :
>
> - rudder is reloading
>
> - i can log in (web interface is ok)
>
> - I can see my rules / nodes / groups / directives
>
>
>
> There is no errors in /var/log/rudder/webapp/2014_03_05.stderrout.log
> and /var/log/rudder/ldap/slapd.log
>
> Rules seems to be applied (as we can see in
> /var/log/rudder/reports/all.log)
>
>
>
> *_BUT_*Then any actions done in the web interface will drive to our
> previous bug (example of action : modification just add text on a
> description field =of a rules or a techniques and save)
>
>
>
>
>
>
>
> /var/log/rudder/ldap/slapd.log =>
>
> Mar 5 11:32:14 SERVERPROD slapd[1383]: connection_input: conn=1003
> deferring operation: too many executing
>
> Mar 5 11:32:16 SERVERPROD slapd[1383]: bdb(cn=rudder-configuration):
> id2entry.bdb page 91 is on free list with type 7
>
> Mar 5 11:32:16 SERVERPROD slapd[1383]: bdb(cn=rudder-configuration):
> PANIC: Invalid argument
>
> Mar 5 11:32:16 SERVERPROD slapd[1383]: bdb(cn=rudder-configuration):
> PANIC: fatal region error detected; run recovery
>
>
>
> /var/log/rudder/webapp/2014_03_05.stderrout.log =>
>
> [2014-03-05 11:32:16] ERROR
> com.normation.rudder.services.servers.NodeConfigurationServiceImpl -
> Can not delete the inconsistent node configuration with id: root. The
> error does not seems to be corrected automatically, please contact
> your administrator.
>
> [2014-03-05 11:32:16] ERROR
> com.normation.rudder.batch.AsyncDeploymentAgent$DeployerAgent - Error
> when doing deployment, reason Cannot set target configuration node <-
> Can not delete 'nodeId=root,ou=Nodes
> Configuration,ou=Rudder,cn=rudder-configuration': internal error
>
> [2014-03-05 11:32:16] ERROR
> com.normation.rudder.batch.AsyncDeploymentAgent - Deployment error for
> process '952' at 2014/03/05 11:32:16: Cannot set target configuration node
>
> [2014-03-05 11:32:43] ERROR
> com.normation.ldap.sdk.ROPooledSimpleAuthConnectionProvider - Can't
> execute LDAP request
>
> com.unboundid.ldap.sdk.LDAPSearchException: internal error
>
> at
> com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3112)
> ~[unboundid-ldapsdk-2.3.1.jar:2.3.1]
>
>
>
> /var/log/rudder/reports/all.log =>
>
> Mar 5 11:32:14 SERVERPROD slapd[1383]: connection_input: conn=1003
> deferring operation: too many executing
>
> Mar 5 11:32:16 SERVERPROD slapd[1383]: bdb(cn=rudder-configuration):
> id2entry.bdb page 91 is on free list with type 7
>
> Mar 5 11:32:16 SERVERPROD slapd[1383]: bdb(cn=rudder-configuration):
> PANIC: Invalid argument
>
> Mar 5 11:32:16 SERVERPROD slapd[1383]: bdb(cn=rudder-configuration):
> PANIC: fatal region error detected; run recovery
>
>
>
> If we reboot the os, all will be up & running and rules seems to be
> modificated… And it’s not a solution to reboot at each crash /
> modification
>
>
>
> If you got any other idea to correct that issue… J
>
>
>
> BR
>
>
>
> Cedric
>
>
>
> *De :*Prestasit01
> *Envoyé :* mardi 4 mars 2014 18:41
> *À :* 'Francois Armand'; 'Matthieu CERDA'
> *Cc :* rudder-users at lists.rudder-project.org
> *Objet :* RE: [rudder-users] LDAP issue
>
>
>
> Hello,
>
>
>
> Bad news… it came back
>
>
>
>
>
> This time i can see other info :
>
>
>
> this night =>
>
>
>
> [2014-03-04 00:25:49] INFO
> com.normation.inventory.provisioning.endpoint.Fusion
> ReportEndpoint - New input report: 'monserveurnode
> -2013-11-27-16-26-37.o
> cs'
>
> [2014-03-04 00:25:49] WARN
> com.normation.inventory.provisioning.fusion.FusionRe
> portUnmarshaller - Error when parsing date for last
> user loggin. Awaited
> format
> is EEE MMM dd HH:mm, found: 2.6.32-431.5.1.e Mon Mar 3
>
>
>
>
>
> Now =>
>
>
>
> Mar 4 18:31:54 SERVERPROD slapd[10465]: slapd starting
>
> Mar 4 18:31:55 SERVERPROD slapd[10470]: [OK] OpenLDAP started on port
> 389 and 636
>
> Mar 4 18:34:14 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:15 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:16 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:16 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:16 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:17 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:19 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:19 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:19 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:20 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:21 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:21 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:21 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:24 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:25 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:25 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:27 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:29 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:30 SERVERPROD slapd[10465]: connection_input: conn=1002
> deferring operation: too many executing
>
> Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration):
> id2entry.bdb page 91 is on free list with type 7
>
> Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration):
> PANIC: Invalid argument
>
> Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration):
> PANIC: fatal region error detected; run recovery
>
> Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration):
> PANIC: fatal region error detected; run recovery
>
> Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration):
> PANIC: fatal region error detected; run recovery
>
> Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration):
> PANIC: fatal region error detected; run recovery
>
> Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration):
> txn_checkpoint: log failed at LSN [265 527847]: DB_RUNRECOVERY: Fatal
> error, run database recovery
>
> Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration):
> PANIC: fatal region error detected; run recovery
>
>
>
> And also (in loop)
>
>
>
> [2014-03-04 18:37:18] ERROR
> com.normation.ldap.sdk.ROPooledSimpleAuthConnectionProvider - Can't
> execute LDAP request
>
> com.unboundid.ldap.sdk.LDAPSearchException: internal error
>
>
>
> The previous workaround don’t work now :(
>
>
>
> Any idea ???
>
>
>
> BR
>
>
>
> Cedric
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *De :*Francois Armand [mailto:francois.armand at normation.com]
> *Envoyé :* lundi 3 mars 2014 17:11
> *À :* Prestasit01; 'Matthieu CERDA'
> *Cc :* rudder-users at lists.rudder-project.org
> <mailto:rudder-users at lists.rudder-project.org>
> *Objet :* Re: [rudder-users] LDAP issue
>
>
>
> Answer in line:
>
>
> On 03/03/2014 16:55, Prestasit01 wrote:
>
>
>
> Mathieu,
>
>
>
> It seems work !
>
>
> Great !
>
>
>
>
> I retried but this time add some actions :
>
>
>
> * First, you need the Berkeley DB utilities: yum install db4-utils
> * Then, you need to make sure slapd is off: /etc/init.d/slapd stop
> * Delete bad ldif files (0 size) and files created after the issue
> in /var/rudder/ldap/backup
> * Try to recover the databases: db_recover -h
> /var/rudder/ldap/openldap-data
> * Reindex the database: /etc/init.d/slapd reindex
> * Restart slapd: /etc/init.d/slapd start
> * In the web app : clear caches, it will give you the following
> messages* _many times_*
>
>
> Yeah... These warning message will be removed in the 2.10 version,
> they are sequel of the past.
>
> Thanks,
>
> *
>
> /var/log/rudder/webapp/2014_03_03.stderrout.log =>
>
> [2014-03-03 16:38:42] WARN
> com.normation.rudder.services.reports.ReportingServiceImpl - Somewhere
> in the expansion of variables, the bounded variable TRACKINGKEY for
> TRACKINGKEY in DirectiveVal inventory-all was lost
>
>
>
> /var/log/rudder/ldap/slapd.log =>
>
> Mar 3 16:42:56 p2rudder01 slapd[4436]: connection_input: conn=1003
> deferring operation: too many executing
>
> * (reload groups/techniques also if wanted)
> * Take a look at the logs to see if the error happens again while
> using the webapp
>
>
>
> Now I don’t have the error anymore.
>
>
>
> Thanks for the help ! J
>
>
>
> BR
>
>
>
> Cedric
>
>
>
>
>
> *De :*rudder-users-bounces at lists.rudder-project.org
> <mailto:rudder-users-bounces at lists.rudder-project.org>
> [mailto:rudder-users-bounces at lists.rudder-project.org] *De la part de*
> Prestasit01
> *Envoyé :* lundi 3 mars 2014 16:00
> *À :* 'Matthieu CERDA'
> *Cc :* rudder-users at lists.rudder-project.org
> <mailto:rudder-users at lists.rudder-project.org>
> *Objet :* Re: [rudder-users] LDAP issue
>
>
>
> Hello Matthieu,
>
>
>
> Unfortunately it doesn't change anything.
>
>
>
> [root at SERVER ldap]# tail -f slapd.log
>
> Mar 3 15:40:59 SERVER slapd[1328]: [INFO] Launching OpenLDAP
> replication...
>
> Mar 3 15:40:59 SERVER slapd[1329]: [INFO] no replica found in
> configuration, aborting lauching slurpd
>
> Mar 3 15:40:59 SERVER slapd[1330]: [INFO] no db_recover done
>
> Mar 3 15:40:59 SERVER slapd[1331]: [INFO] Launching OpenLDAP...
>
> Mar 3 15:40:59 SERVER slapd[1332]: [OK] file descriptor limit set to 1024
>
> Mar 3 15:40:59 SERVER slapd[1333]: @(#) $OpenLDAP: slapd 2.4.30 (Jan
> 15 2014 21:29:11) $
>
>
> root at centos-builder-6-64.labo.normation.com:/usr/src/redhat/BUILD/openldap-source/servers/slapd
> <mailto:root at centos-builder-6-64.labo.normation.com:/usr/src/redhat/BUILD/openldap-source/servers/slapd>
>
> Mar 3 15:40:59 SERVER slapd[1334]: hdb_db_open: database
> "cn=rudder-configuration": unclean shutdown detected; attempting recovery.
>
> Mar 3 15:41:00 SERVER slapd[1334]: slapd starting
>
> Mar 3 15:41:01 SERVER slapd[1339]: [OK] OpenLDAP started on port 389
> and 636
>
> Mar 3 15:50:10 SERVER slapd[1334]: bdb(cn=rudder-configuration):
> id2entry.bdb page 91 is on free list with type 7
>
> Mar 3 15:50:10 SERVER slapd[1334]: bdb(cn=rudder-configuration):
> PANIC: Invalid argument
>
> Mar 3 15:50:54 SERVER slapd[1829]: [INFO] Using /etc/default/slapd
> for configuration
>
> Mar 3 15:50:54 SERVER slapd[1834]: [INFO] Halting OpenLDAP...
>
> Mar 3 15:50:54 SERVER slapd[1836]: [OK] OpenLDAP stopped after 0 seconds
>
> Mar 3 15:50:54 SERVER slapd[1837]: [INFO] Launching OpenLDAP database
> backup...
>
> Mar 3 15:50:54 SERVER slapd[1854]: [ALERT] OpenLDAP database backup
> failed
>
>
>
> :’(
>
>
>
> Cedric
>
>
>
>
>
>
>
>
>
> *De :*Matthieu CERDA [mailto:matthieu.cerda at normation.com]
> *Envoyé :* lundi 3 mars 2014 15:14
> *À :* Prestasit01
> *Cc :* rudder-users at lists.rudder-project.org
> <mailto:rudder-users at lists.rudder-project.org>
> *Objet :* Re: [rudder-users] LDAP issue
>
>
>
> Hi Cédric,
>
> It seems the disk full condition happened just during a write
> operation in slapd, causing a corruption of it's BDB databases.
>
> Let's fix it:
>
> * First, you need the Berkeley DB utilities: yum install db4-utils
> * Then, you need to make sure slapd is off: /etc/init.d/slapd stop
> * Try to recover the databases: db_recover -h
> /var/rudder/ldap/openldap-data
> * Reindex the database: /etc/init.d/slapd reindex
> * Restart slapd: /etc/init.d/slapd start
> * Take a look at the logs to see if the error happens again while
> using the webapp
>
> Can you tell me if it solves the issue ?
>
> Thanks in advance,
> --
> Matthieu CERDA
>
> Le 03/03/2014 14:20, Prestasit01 a écrit :
>
> Hi François ,
>
>
>
> Current Slapd.log and ldapsearch seems normal.
>
>
>
> But I found that information in *slapd.log-20140303* :
>
>
>
> Mar 3 11:10:08 p2rudder01 slapd[1345]: [OK] OpenLDAP started on
> port 389 and 636
>
> Mar 3 11:11:47 p2rudder01 slapd[1340]:
> bdb(cn=rudder-configuration): id2entry.bdb page 91 is on free list
> with type 7
>
> Mar 3 11:11:47 p2rudder01 slapd[1340]:
> bdb(cn=rudder-configuration): PANIC: Invalid argument
>
> Mar 3 11:13:22 p2rudder01 slapd[2430]: [INFO] Using
> /etc/default/slapd for configuration
>
> Mar 3 11:13:22 p2rudder01 slapd[2435]: [INFO] Halting OpenLDAP...
>
> Mar 3 11:13:22 p2rudder01 slapd[2437]: [OK] OpenLDAP stopped
> after 0 seconds
>
> Mar 3 11:13:22 p2rudder01 slapd[2438]: [INFO] Launching OpenLDAP
> database backup...
>
> Mar 3 11:13:22 p2rudder01 slapd[2455]: [ALERT] OpenLDAP database
> backup failed
>
> Mar 3 11:13:22 p2rudder01 slapd[2469]: [INFO] Using
> /etc/default/slapd for configuration
>
> Mar 3 11:13:22 p2rudder01 slapd[2474]: [INFO] Launching OpenLDAP
> configuration test...
>
> Mar 3 11:13:22 p2rudder01 slapd[2476]: [OK] OpenLDAP
> configuration test successful
>
> Mar 3 11:13:22 p2rudder01 slapd[2477]: [INFO] Launching OpenLDAP
> replication...
>
> Mar 3 11:13:22 p2rudder01 slapd[2478]: [INFO] no replica found in
> configuration, aborting lauching slurpd
>
> Mar 3 11:13:22 p2rudder01 slapd[2479]: [INFO] no db_recover done
>
> Mar 3 11:13:22 p2rudder01 slapd[2480]: [INFO] Launching OpenLDAP...
>
> Mar 3 11:13:22 p2rudder01 slapd[2481]: [OK] file descriptor limit
> set to 1024
>
> Mar 3 11:13:22 p2rudder01 slapd[2482]: @(#) $OpenLDAP: slapd
> 2.4.30 (Jan 15 2014 21:29:11) $
>
>
> root at centos-builder-6-64.labo.normation.com:/usr/src/redhat/BUILD/openldap-source/servers/slapd
> <mailto:root at centos-builder-6-64.labo.normation.com:/usr/src/redhat/BUILD/openldap-source/servers/slapd>
>
> Mar 3 11:13:22 p2rudder01 slapd[2483]: hdb_db_open: database
> "cn=rudder-configuration": unclean shutdown detected; attempting
> recovery.
>
> Mar 3 11:13:23 p2rudder01 slapd[2483]: slapd starting
>
> Mar 3 11:13:24 p2rudder01 slapd[2488]: [OK] OpenLDAP started on
> port 389 and 636
>
>
>
> Is that talking to you ?
>
>
>
> BR
>
> Cédric
>
> -------------------------------------------------------------------------
> Les informations ou pieces jointes contenues dans ce message sont
> confidentielles. Seul le destinataire expressement vise peut en
> prendre connaissance. Toute autre personne qui en divulguera,
> diffusera ou prendra des copies sera passible de poursuites. La
> societe Ouest-France decline en outre, toute responsabilite de quelque
> nature que ce soit au titre de ce message s'il a ete altere, deforme
> ou falsifie.
>
> -------------------------------------------------------------------------
> Les informations ou pieces jointes contenues dans ce message sont
> confidentielles. Seul le destinataire expressement vise peut en
> prendre connaissance. Toute autre personne qui en divulguera,
> diffusera ou prendra des copies sera passible de poursuites. La
> societe Ouest-France decline en outre, toute responsabilite de quelque
> nature que ce soit au titre de ce message s'il a ete altere, deforme
> ou falsifie.
>
> _______________________________________________
> rudder-users mailing list
> rudder-users at lists.rudder-project.org <mailto:rudder-users at lists.rudder-project.org>
> http://www.rudder-project.org/mailman/listinfo/rudder-users
>
>
>
> --
>
> ------------------------------------------------------------------------
>
> *François ARMAND*
> /Directeur de la R&D/
> Normation <http://www.normation.com>
>
> ------------------------------------------------------------------------
>
> *87 rue de Turbigo, 75003 Paris, France*
>
> Telephone:
>
>
>
> +33 (0)1 83 62 99 23
>
> Mobile:
>
>
>
> +33 (0)6 63 37 60 55
>
> ------------------------------------------------------------------------
>
>
>
> -------------------------------------------------------------------------
> Les informations ou pieces jointes contenues dans ce message sont
> confidentielles. Seul le destinataire expressement vise peut en
> prendre connaissance. Toute autre personne qui en divulguera,
> diffusera ou prendra des copies sera passible de poursuites. La
> societe Ouest-France decline en outre, toute responsabilite de quelque
> nature que ce soit au titre de ce message s'il a ete altere, deforme
> ou falsifie.
--
------------------------------------------------------------------------
*François ARMAND*
/Directeur de la R&D/
Normation <http://www.normation.com>
------------------------------------------------------------------------
*87 rue de Turbigo, 75003 Paris, France*
Telephone: +33 (0)1 83 62 99 23
Mobile: +33 (0)6 63 37 60 55
------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20140305/e0572d41/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 4991 bytes
Desc: not available
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20140305/e0572d41/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 9273 bytes
Desc: not available
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20140305/e0572d41/attachment-0002.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 17382 bytes
Desc: not available
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20140305/e0572d41/attachment-0003.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 4098 bytes
Desc: not available
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20140305/e0572d41/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sig-normation-logo-square.png
Type: image/png
Size: 3503 bytes
Desc: not available
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20140305/e0572d41/attachment-0005.png>
More information about the rudder-users
mailing list