[rudder-users] LDAP issue
Prestasit01
prestasit01 at ouest-france.fr
Tue Mar 4 18:41:16 CET 2014
Hello,
Bad news… it came back
This time i can see other info :
this night =>
[2014-03-04 00:25:49] INFO com.normation.inventory.provisioning.endpoint.Fusion ReportEndpoint - New input report: 'monserveurnode -2013-11-27-16-26-37.o cs'
[2014-03-04 00:25:49] WARN com.normation.inventory.provisioning.fusion.FusionRe portUnmarshaller - Error when parsing date for last user loggin. Awaited format is EEE MMM dd HH:mm, found: 2.6.32-431.5.1.e Mon Mar 3
Now =>
Mar 4 18:31:54 SERVERPROD slapd[10465]: slapd starting
Mar 4 18:31:55 SERVERPROD slapd[10470]: [OK] OpenLDAP started on port 389 and 636
Mar 4 18:34:14 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:15 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:16 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:16 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:16 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:17 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:19 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:19 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:19 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:20 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:21 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:21 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:21 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:24 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:25 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:25 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:27 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:29 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:30 SERVERPROD slapd[10465]: connection_input: conn=1002 deferring operation: too many executing
Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration): id2entry.bdb page 91 is on free list with type 7
Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration): PANIC: Invalid argument
Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration): PANIC: fatal region error detected; run recovery
Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration): PANIC: fatal region error detected; run recovery
Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration): PANIC: fatal region error detected; run recovery
Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration): PANIC: fatal region error detected; run recovery
Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration): txn_checkpoint: log failed at LSN [265 527847]: DB_RUNRECOVERY: Fatal error, run database recovery
Mar 4 18:34:31 SERVERPROD slapd[10465]: bdb(cn=rudder-configuration): PANIC: fatal region error detected; run recovery
And also (in loop)
[2014-03-04 18:37:18] ERROR com.normation.ldap.sdk.ROPooledSimpleAuthConnectionProvider - Can't execute LDAP request
com.unboundid.ldap.sdk.LDAPSearchException: internal error
The previous workaround don’t work now :(
Any idea ???
BR
Cedric
De : Francois Armand [mailto:francois.armand at normation.com]
Envoyé : lundi 3 mars 2014 17:11
À : Prestasit01; 'Matthieu CERDA'
Cc : rudder-users at lists.rudder-project.org
Objet : Re: [rudder-users] LDAP issue
Answer in line:
On 03/03/2014 16:55, Prestasit01 wrote:
Mathieu,
It seems work !
Great !
I retried but this time add some actions :
* First, you need the Berkeley DB utilities: yum install db4-utils
* Then, you need to make sure slapd is off: /etc/init.d/slapd stop
* Delete bad ldif files (0 size) and files created after the issue in /var/rudder/ldap/backup
* Try to recover the databases: db_recover -h /var/rudder/ldap/openldap-data
* Reindex the database: /etc/init.d/slapd reindex
* Restart slapd: /etc/init.d/slapd start
* In the web app : clear caches, it will give you the following messages many times
Yeah... These warning message will be removed in the 2.10 version, they are sequel of the past.
Thanks,
*
/var/log/rudder/webapp/2014_03_03.stderrout.log =>
[2014-03-03 16:38:42] WARN com.normation.rudder.services.reports.ReportingServiceImpl - Somewhere in the expansion of variables, the bounded variable TRACKINGKEY for TRACKINGKEY in DirectiveVal inventory-all was lost
/var/log/rudder/ldap/slapd.log =>
Mar 3 16:42:56 p2rudder01 slapd[4436]: connection_input: conn=1003 deferring operation: too many executing
* (reload groups/techniques also if wanted)
* Take a look at the logs to see if the error happens again while using the webapp
Now I don’t have the error anymore.
Thanks for the help ! ☺
BR
Cedric
De : rudder-users-bounces at lists.rudder-project.org<mailto:rudder-users-bounces at lists.rudder-project.org> [mailto:rudder-users-bounces at lists.rudder-project.org] De la part de Prestasit01
Envoyé : lundi 3 mars 2014 16:00
À : 'Matthieu CERDA'
Cc : rudder-users at lists.rudder-project.org<mailto:rudder-users at lists.rudder-project.org>
Objet : Re: [rudder-users] LDAP issue
Hello Matthieu,
Unfortunately it doesn't change anything.
[root at SERVER ldap]# tail -f slapd.log
Mar 3 15:40:59 SERVER slapd[1328]: [INFO] Launching OpenLDAP replication...
Mar 3 15:40:59 SERVER slapd[1329]: [INFO] no replica found in configuration, aborting lauching slurpd
Mar 3 15:40:59 SERVER slapd[1330]: [INFO] no db_recover done
Mar 3 15:40:59 SERVER slapd[1331]: [INFO] Launching OpenLDAP...
Mar 3 15:40:59 SERVER slapd[1332]: [OK] file descriptor limit set to 1024
Mar 3 15:40:59 SERVER slapd[1333]: @(#) $OpenLDAP: slapd 2.4.30 (Jan 15 2014 21:29:11) $
root at centos-builder-6-64.labo.normation.com:/usr/src/redhat/BUILD/openldap-source/servers/slapd<mailto:root at centos-builder-6-64.labo.normation.com:/usr/src/redhat/BUILD/openldap-source/servers/slapd>
Mar 3 15:40:59 SERVER slapd[1334]: hdb_db_open: database "cn=rudder-configuration": unclean shutdown detected; attempting recovery.
Mar 3 15:41:00 SERVER slapd[1334]: slapd starting
Mar 3 15:41:01 SERVER slapd[1339]: [OK] OpenLDAP started on port 389 and 636
Mar 3 15:50:10 SERVER slapd[1334]: bdb(cn=rudder-configuration): id2entry.bdb page 91 is on free list with type 7
Mar 3 15:50:10 SERVER slapd[1334]: bdb(cn=rudder-configuration): PANIC: Invalid argument
Mar 3 15:50:54 SERVER slapd[1829]: [INFO] Using /etc/default/slapd for configuration
Mar 3 15:50:54 SERVER slapd[1834]: [INFO] Halting OpenLDAP...
Mar 3 15:50:54 SERVER slapd[1836]: [OK] OpenLDAP stopped after 0 seconds
Mar 3 15:50:54 SERVER slapd[1837]: [INFO] Launching OpenLDAP database backup...
Mar 3 15:50:54 SERVER slapd[1854]: [ALERT] OpenLDAP database backup failed
:’(
Cedric
De : Matthieu CERDA [mailto:matthieu.cerda at normation.com]
Envoyé : lundi 3 mars 2014 15:14
À : Prestasit01
Cc : rudder-users at lists.rudder-project.org<mailto:rudder-users at lists.rudder-project.org>
Objet : Re: [rudder-users] LDAP issue
Hi Cédric,
It seems the disk full condition happened just during a write operation in slapd, causing a corruption of it's BDB databases.
Let's fix it:
* First, you need the Berkeley DB utilities: yum install db4-utils
* Then, you need to make sure slapd is off: /etc/init.d/slapd stop
* Try to recover the databases: db_recover -h /var/rudder/ldap/openldap-data
* Reindex the database: /etc/init.d/slapd reindex
* Restart slapd: /etc/init.d/slapd start
* Take a look at the logs to see if the error happens again while using the webapp
Can you tell me if it solves the issue ?
Thanks in advance,
--
Matthieu CERDA
Le 03/03/2014 14:20, Prestasit01 a écrit :
Hi François ,
Current Slapd.log and ldapsearch seems normal.
But I found that information in slapd.log-20140303 :
Mar 3 11:10:08 p2rudder01 slapd[1345]: [OK] OpenLDAP started on port 389 and 636
Mar 3 11:11:47 p2rudder01 slapd[1340]: bdb(cn=rudder-configuration): id2entry.bdb page 91 is on free list with type 7
Mar 3 11:11:47 p2rudder01 slapd[1340]: bdb(cn=rudder-configuration): PANIC: Invalid argument
Mar 3 11:13:22 p2rudder01 slapd[2430]: [INFO] Using /etc/default/slapd for configuration
Mar 3 11:13:22 p2rudder01 slapd[2435]: [INFO] Halting OpenLDAP...
Mar 3 11:13:22 p2rudder01 slapd[2437]: [OK] OpenLDAP stopped after 0 seconds
Mar 3 11:13:22 p2rudder01 slapd[2438]: [INFO] Launching OpenLDAP database backup...
Mar 3 11:13:22 p2rudder01 slapd[2455]: [ALERT] OpenLDAP database backup failed
Mar 3 11:13:22 p2rudder01 slapd[2469]: [INFO] Using /etc/default/slapd for configuration
Mar 3 11:13:22 p2rudder01 slapd[2474]: [INFO] Launching OpenLDAP configuration test...
Mar 3 11:13:22 p2rudder01 slapd[2476]: [OK] OpenLDAP configuration test successful
Mar 3 11:13:22 p2rudder01 slapd[2477]: [INFO] Launching OpenLDAP replication...
Mar 3 11:13:22 p2rudder01 slapd[2478]: [INFO] no replica found in configuration, aborting lauching slurpd
Mar 3 11:13:22 p2rudder01 slapd[2479]: [INFO] no db_recover done
Mar 3 11:13:22 p2rudder01 slapd[2480]: [INFO] Launching OpenLDAP...
Mar 3 11:13:22 p2rudder01 slapd[2481]: [OK] file descriptor limit set to 1024
Mar 3 11:13:22 p2rudder01 slapd[2482]: @(#) $OpenLDAP: slapd 2.4.30 (Jan 15 2014 21:29:11) $
root at centos-builder-6-64.labo.normation.com:/usr/src/redhat/BUILD/openldap-source/servers/slapd<mailto:root at centos-builder-6-64.labo.normation.com:/usr/src/redhat/BUILD/openldap-source/servers/slapd>
Mar 3 11:13:22 p2rudder01 slapd[2483]: hdb_db_open: database "cn=rudder-configuration": unclean shutdown detected; attempting recovery.
Mar 3 11:13:23 p2rudder01 slapd[2483]: slapd starting
Mar 3 11:13:24 p2rudder01 slapd[2488]: [OK] OpenLDAP started on port 389 and 636
Is that talking to you ?
BR
Cédric
-------------------------------------------------------------------------
Les informations ou pieces jointes contenues dans ce message sont confidentielles. Seul le destinataire expressement vise peut en prendre connaissance. Toute autre personne qui en divulguera, diffusera ou prendra des copies sera passible de poursuites. La societe Ouest-France decline en outre, toute responsabilite de quelque nature que ce soit au titre de ce message s'il a ete altere, deforme ou falsifie.
-------------------------------------------------------------------------
Les informations ou pieces jointes contenues dans ce message sont confidentielles. Seul le destinataire expressement vise peut en prendre connaissance. Toute autre personne qui en divulguera, diffusera ou prendra des copies sera passible de poursuites. La societe Ouest-France decline en outre, toute responsabilite de quelque nature que ce soit au titre de ce message s'il a ete altere, deforme ou falsifie.
_______________________________________________
rudder-users mailing list
rudder-users at lists.rudder-project.org<mailto:rudder-users at lists.rudder-project.org>
http://www.rudder-project.org/mailman/listinfo/rudder-users
--
________________________________
[cid:image002.png at 01CF37D7.AA4EC8E0]François ARMAND
Directeur de la R&D
Normation<http://www.normation.com>
________________________________
87 rue de Turbigo, 75003 Paris, France
Telephone:
+33 (0)1 83 62 99 23
Mobile:
+33 (0)6 63 37 60 55
________________________________
-------------------------------------------------------------------------<br>
Les informations ou pieces jointes contenues dans ce message sont
confidentielles. Seul le destinataire expressement vise peut en prendre
connaissance. Toute autre personne qui en divulguera, diffusera ou prendra des copies sera passible de poursuites. La societe Ouest-France decline en outre, toute responsabilite de quelque nature que ce soit au titre de ce message s'il a ete altere, deforme ou falsifie.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20140304/a7de2a5c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 3503 bytes
Desc: image001.png
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20140304/a7de2a5c/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 4099 bytes
Desc: image002.png
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20140304/a7de2a5c/attachment-0003.png>
More information about the rudder-users
mailing list