https://issues.rudder.io/https://issues.rudder.io/themes/rudder7/favicon/favicon.ico?17096450182017-03-17T15:42:04ZIssue TrackerRudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=733872017-03-17T15:42:04ZAlexis Moussetalexis.mousset@rudder.io
<ul><li><strong>Category</strong> set to <i>Web - Compliance & node report</i></li><li><strong>Target version</strong> set to <i>4.1.0</i></li><li><strong>Severity</strong> set to <i>Critical - prevents main use of Rudder | no workaround | data loss | security</i></li></ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=733912017-03-17T16:29:36ZJanos Mattyasovszky
<ul></ul><p>This happend with currently 2287 in the system, I will try to go back to ~1000 nodes and test again.</p> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=733962017-03-20T10:25:56ZFrançois ARMANDfrancois.armand@rudder.io
<ul></ul><p>This is most likelly the same bug as <a class="issue tracker-1 status-6 priority-16 priority-default closed" title="Bug: Dashboard do not load due to stackoverflow error in 4.1 (Rejected)" href="https://issues.rudder.io/issues/10380">#10380</a>, but it is more impacting.</p>
<p>A first, simple workarounnd may be to just increase the available stack space. <a class="user active user-mention" href="https://issues.rudder.io/users/309">@Janos Matya</a>, would you mind trying to add in `/etc/default/rudder-jetty` the following arg:</p>
<pre>
JAVA_OPTIONS="-Xss4M"
</pre>
<p>But the correct solution is to check the number of element in the "in" clause, and if more than, say 200, create a temporary table + join.</p> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=733972017-03-20T10:26:06ZFrançois ARMANDfrancois.armand@rudder.io
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-6 priority-16 priority-default closed" href="/issues/10380">Bug #10380</a>: Dashboard do not load due to stackoverflow error in 4.1</i> added</li></ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=734002017-03-20T10:30:31ZNicolas CHARLESnicolas.charles@rudder.io
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-6 priority-16 priority-default closed" href="/issues/10453">Bug #10453</a>: Error during policy generation</i> added</li></ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=734012017-03-20T10:35:27ZFrançois ARMANDfrancois.armand@rudder.io
<ul><li><strong>Subject</strong> changed from <i>Cannot historize names</i> to <i>Stackoverflow during generation for 2200 nodes</i></li></ul><p>And as Nicolas reported, the "historization: null" is most likelly something that is not correctly initialized, and we will be correcting it in <a class="issue tracker-1 status-6 priority-16 priority-default closed" title="Bug: Error during policy generation (Rejected)" href="https://issues.rudder.io/issues/10453">#10453</a></p> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=734072017-03-20T12:39:00ZJanos Mattyasovszky
<ul></ul><p>that did work.</p>
<pre>
[2017-03-20 13:06:43] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Node configuration written on filesystem in 439848 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Node configuration cached in LDAP in 296 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Node configuration written in 440261 ms, start to update expected reports.
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Reports updated in 866 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Post-policy-generation hooks ran in 74 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Timing summary:
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Run pre-gen scripts hooks : 29 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Run pre-gen modules hooks : 2 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Fetch all information : 358 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Historize names : 901 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Build current rule values : 15 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Build target configuration: 766 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Update rule vals : 65 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Increment rule serials : 59 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Write node configurations : 440261 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Save expected reports : 866 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Run post generation hooks : 74 ms
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Number of nodes updated : 1087
[2017-03-20 13:06:44] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Policy generation completed in 445502 ms
[2017-03-20 13:06:44] INFO com.normation.rudder.batch.AsyncDeploymentAgent - Successful policy update '21' [started 2017-03-20 12:59:19 - ended 2017-03-20 13:06:44]
</pre>
<p>PS: These times were with writing to a ramdisk:</p>
<pre>
# df -hTP /var/rudder/share
Filesystem Type Size Used Avail Use% Mounted on
none tmpfs 2.0G 876M 1.2G 43% /var/rudder/share
</pre> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=734092017-03-20T13:26:38ZFrançois ARMANDfrancois.armand@rudder.io
<ul></ul><p>OK, nice, we have a workaround.</p>
<p>Numbers are interesting, to. We are spending just far too much time writting files :/</p> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=734552017-03-21T10:21:27ZFrançois ARMANDfrancois.armand@rudder.io
<ul></ul><p>Let merge all the stackoverflow problem under that one.</p>
<p>To sum-up, the problem is in our use of doobie <strong>Fragments.in</strong> as reported here: <a class="external" href="https://github.com/tpolecat/doobie/issues/426">https://github.com/tpolecat/doobie/issues/426</a></p> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=734582017-03-21T10:21:59ZFrançois ARMANDfrancois.armand@rudder.io
<ul><li><strong>Related to</strong> deleted (<i><a class="issue tracker-1 status-6 priority-16 priority-default closed" href="/issues/10453">Bug #10453</a>: Error during policy generation</i>)</li></ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=734602017-03-21T10:22:07ZFrançois ARMANDfrancois.armand@rudder.io
<ul><li><strong>Has duplicate</strong> <i><a class="issue tracker-1 status-6 priority-16 priority-default closed" href="/issues/10453">Bug #10453</a>: Error during policy generation</i> added</li></ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=734612017-03-21T10:22:24ZFrançois ARMANDfrancois.armand@rudder.io
<ul><li><strong>Related to</strong> deleted (<i><a class="issue tracker-1 status-6 priority-16 priority-default closed" href="/issues/10380">Bug #10380</a>: Dashboard do not load due to stackoverflow error in 4.1</i>)</li></ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=734652017-03-21T10:22:41ZFrançois ARMANDfrancois.armand@rudder.io
<ul><li><strong>Has duplicate</strong> <i><a class="issue tracker-1 status-6 priority-16 priority-default closed" href="/issues/10380">Bug #10380</a>: Dashboard do not load due to stackoverflow error in 4.1</i> added</li></ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=734862017-03-21T14:25:36ZFrançois ARMANDfrancois.armand@rudder.io
<ul><li><strong>Subject</strong> changed from <i>Stackoverflow during generation for 2200 nodes</i> to <i>Several core features don't work anymore with more than 1000 nodes</i></li><li><strong>User visibility</strong> set to <i>Operational - other Techniques | Technique editor | Rudder settings</i></li></ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=735742017-03-21T17:13:33ZFrançois ARMANDfrancois.armand@rudder.io
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In progress</i></li><li><strong>Assignee</strong> set to <i>François ARMAND</i></li></ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=735752017-03-21T17:16:18ZFrançois ARMANDfrancois.armand@rudder.io
<ul><li><strong>Status</strong> changed from <i>In progress</i> to <i>Pending technical review</i></li><li><strong>Assignee</strong> changed from <i>François ARMAND</i> to <i>Nicolas CHARLES</i></li><li><strong>Pull Request</strong> set to <i>https://github.com/Normation/rudder/pull/1605</i></li></ul><p>PR <a class="external" href="https://github.com/Normation/rudder/pull/1605">https://github.com/Normation/rudder/pull/1605</a></p> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=736062017-03-22T11:13:16ZNicolas CHARLESnicolas.charles@rudder.io
<ul><li><strong>Assignee</strong> changed from <i>Nicolas CHARLES</i> to <i>Alexis Mousset</i></li><li><strong>Pull Request</strong> changed from <i>https://github.com/Normation/rudder/pull/1605</i> to <i>https://github.com/Normation/rudder/pull/1606</i></li></ul><p>PR <a class="external" href="https://github.com/Normation/rudder/pull/1606">https://github.com/Normation/rudder/pull/1606</a></p> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=736082017-03-22T11:19:59ZFrançois ARMANDfrancois.armand@rudder.io
<ul><li><strong>Status</strong> changed from <i>Pending technical review</i> to <i>Pending release</i></li></ul><p>Applied in changeset <a class="changeset" title="Fixes #10456: Several core features don't work anymore with more than 1000 nodes" href="https://issues.rudder.io/projects/rudder/repository/rudder/revisions/916000fe6888fce9b1aae5a01d182dfc61190a1f">rudder|916000fe6888fce9b1aae5a01d182dfc61190a1f</a>.</p> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=740082017-03-28T14:31:29ZBenoît PECCATTEbenoit.peccatte@rudder.io
<ul><li><strong>Priority</strong> set to <i>60</i></li></ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=742642017-03-30T09:47:14ZBenoît PECCATTEbenoit.peccatte@rudder.io
<ul><li><strong>Status</strong> changed from <i>Pending release</i> to <i>Released</i></li></ul><p>This bug has been fixed in Rudder 4.1.0 which was released today.</p>
<ul>
<li>4.1.0: <a href="http://www.rudder-project.org/pipermail/rudder-announce/2017-March/thread.html000225.html" class="external">Announce</a> <a href="http://www.rudder-project.org/changelog-4.1" class="external">Changelog</a></li>
<li>Download: <a class="external" href="https://www.rudder-project.org/site/get-rudder/downloads/">https://www.rudder-project.org/site/get-rudder/downloads/</a></li>
</ul> Rudder - Bug #10456: Several core features don't work anymore with more than 1000 nodeshttps://issues.rudder.io/issues/10456?journal_id=810852017-08-29T15:21:27ZNicolas CHARLESnicolas.charles@rudder.io
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-5 priority-16 priority-default closed" href="/issues/11037">Bug #11037</a>: Missing agent reports after Rudder server restart</i> added</li></ul>