Bug #6448: Error 404 after rebooting system - Rudder - Issue Tracker

Actions

Copy link

Bug #6448

closed

Error 404 after rebooting system

Added by Jérémy HOCDÉ about 9 years ago. Updated over 7 years ago.

Status:

Rejected

Priority:

Assignee:

François ARMAND

Category:

System integration

Target version:

3.1.17

Pull Request:

Severity:

UX impact:

User visibility:

Effort required:

Priority:

Name check:

Fix check:

Regression:

Description

After rebooting system, rudder does not work properly

HTTP ERROR: 404

Problem accessing /rudder. Reason:

    Not Found

Powered by Jetty:/

FiX: service rudder-jetty restart

tested on ubuntu 14.04 server adm64 - rudder-server-root 3.0.3

Files

2015_03_31.stderrout.log (80.2 KB) 2015_03_31.stderrout.log

Jérémy HOCDÉ, 2015-03-31 16:51

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Dennis Cabooter about 9 years ago

Actions

Copy link

Updated by Vincent MEMBRÉ about 9 years ago

Category set to System integration
Assignee set to Matthieu CERDA
Priority changed from N/A to 2
Target version set to 3.0.4
How to reproduce updated (diff)

This is strange, I'm assigning Matthieu, since I think he will be the best at this

Actions

Copy link

Updated by Matthieu CERDA about 9 years ago

Description updated (diff)
Status changed from New to Discussion
Assignee changed from Matthieu CERDA to Jérémy HOCDÉ

Jeremy, dennis:

Can you please give me what's in /var/log/rudder/webapp/ in the log corresponding to the day when it happened ?

I especially look for a stacktrace near the end of the file, if Jetty outputs a 404 it means Rudder did not initialize properly.

Actions

Copy link

Updated by Florian Heigl about 9 years ago

At least on SLES this was caused by a race:

If slapd isn't done starting when jetty wants to connect it will never successfully initialize.

Let's see if this is the same thing.

Actions

Copy link

Updated by Jérémy HOCDÉ about 9 years ago

File 2015_03_31.stderrout.log 2015_03_31.stderrout.log added

Thks Matthieu for helping :o)

Actions

Copy link

Updated by Matthieu CERDA about 9 years ago

Related to Bug #6263: Startup links for rudder-server-root on Ubuntu are not correct - before 3.1 added

Actions

Copy link

Updated by Matthieu CERDA about 9 years ago

Assignee changed from Jérémy HOCDÉ to François ARMAND
Priority changed from 2 to 3

François, the problem is:

SysV init script dependency is ... flaky, at best. It does'nt always work even with correct dependency definition => j(etty) comes before s(lapd), it starts first => problem
New init systems worsen the problem: they handle dependencies natively, but try by default to run init scripts concurrently => jetty will certainly run before slapd again, unless you create a native job/unit/whatever specifying "slapd must be started first"

Also, on splitted installations, slapd will not be on the same machine. We can not simply add a hard dependency on rudder-slapd because it might be on another machine.

The solution would be either to have:

Jetty able to start without a ready database (since they get started right after, the error page would not be here for long :) )
Have an intelligent init script that tests database connections before starting (but it would be difficult to do it properly, it would bloat some more an already bloated script and make it harder to create native init jobs/units...

The best approach would be number one for me, but I remember you told me it would be hard to do it. Can you tell here how and why it would be difficult to have a webapp able to start without a database connection ? (even if it were to display "unable to connect, please try again later" or something like that in the meantime). Actually, anything else than 404 + webapp unloaded by Jetty would be cool.

Also, an important note: this 404 will not last for more than 5/10mn, as the Rudder agent will notice the webapp does not run properly and restart it automatically after 2 consecutive failed verifications.

This bug report is thus not over-critical, setting priority as 3.

Actions

Copy link

Updated by François ARMAND about 9 years ago

Matthieu,

The problem is that we are doing a lot of consistency checks at boot time (in stored data, migration, etc). The way it is done, it's quite hard to delay them "when connection is available" - or a least, it needs an architecture change that is non trivial. (and of course, we had as hypothesis in a lot of code that data are coherent / up to date / etc, because of that architecture choice).

It could be easier to have a check about connection availability, and display some "you must restart app when connection ready" (but looking at the code, I'm not even sure of that).

Actions

Copy link

Updated by Nicolas CHARLES about 9 years ago

On this point, I think we could try, in the code base of the web interface, have a check at start that checks if database is reachable, and if ldap is reachable. If not, wait for 10 secondes, and try again. If it fails, then abort/display an error page

Actions

Copy link

#10