<p dir="ltr">Hi</p>
<p dir="ltr">Just some minor questions / summaries of your summaries :) <br>
- the node will not begin with uploading an inventory on the initial "bootstrapping" run, only after it received its initial promises from the policy server. <br>
- the bootstrapping code will still have a hook to define the Hostname, but it can be overwritten by later customer hooks?<br>
-> Q: will the Hooks be executed by their names sorted lexically? <br>
-> Q: will all nodes receive the same additional hook plugins? <br>
-> Q: will the inventory contain the json output 1:1 or will it be converted to xml? // can one only use key: value pairs or more complex data structures to extend the inventory? </p>
<p dir="ltr">(NB: Using the collected "facts" as variables is cool, but please don't forget to also add the centrally managed node properties as vars to the node...) </p>
<p dir="ltr">Thanks <br>
Janos <br>
</p>
<br><div class="gmail_quote"><div dir="ltr">Francois Armand <<a href="mailto:francois.armand@normation.com">francois.armand@normation.com</a>> ezt írta (időpont: 2016. máj. 13., P 16:59):<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<div>On 02/05/2016 22:13, Mattyasovszky
János wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi<br>
<br>
<i>Answers Inline.<br>
</i><br>
Regards,<br>
Matya
<div><br>
<div class="gmail_quote">
<div dir="ltr">Jonathan Clarke <<a href="mailto:jonathan.clarke@normation.com" target="_blank">jonathan.clarke@normation.com</a>>
ezt írta (időpont: 2016. máj. 2., H, 18:29):<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi,</div>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<div><br>
<br>
On 04/13/2016 11:48 AM, Francois wrote:<br>
</div>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div>On 13/04/2016 09:49, Nicolas Charles wrote:<br>
</div>
<blockquote type="cite">
<div>Hello Francois, <br>
<br>
Thank you for the nice sum-up. My answers in the
text<br>
</div>
</blockquote>
</blockquote>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> +1</div>
<div text="#000000" bgcolor="#FFFFFF"><br>
<br>
<blockquote type="cite">
<blockquote type="cite">
<div> Le 07/04/2016 15:33, Francois Armand a écrit :<br>
</div>
<blockquote type="cite">
<div>On 07/04/2016 15:08, Vincent Membré wrote:<br>
</div>
<blockquote type="cite">
<div>Le 31/03/2016 12:37, Francois Armand a
écrit :<br>
</div>
<blockquote type="cite"> Hello, <br>
<br>
So, here goes for a summary of <a href="http://www.rudder-project.org/redmine/issues/8022" target="_blank"><a href="http://www.rudder-project.org/redmine/issues/8022" target="_blank">http://www.rudder-project.org/redmine/issues/8022</a></a>,
"Node's FQDN-Resolution is sometimes invalid"
and related tickets. <br>
The problem cover up several sub-cases, which
need to be addressed systematically to achieve
some result.<br>
They are: <br>
<ul>
<li>1/ the node FQDN is used for identifying
a node, and then manage authentication and
authorization to access its promises. If
Rudder server, CFEngine promise server,
and the node don't agree on it, the node
can't get its promises. This is a hard
problem because:</li>
<ul>
<li>FQDN is fragile. It needs a perfectly
up to date and shared DNS environment.
But "it's always a DNS problem", what
gives an idea. <br>
</li>
<li>FQDN tools are notoriously broken, and
don't always agree about what is the
FQDN of a host</li>
<li>even in a perfectly working and
up-to-date DNS env, there may have
voluntary decision not to have the same
FQDN from the host and the server, as
explained in #8022 ticket. <br>
</li>
</ul>
<li>2/ if the node FQDN is not correct in
the first sent inventory, the node
configuration is delayed by one day,
because from the node point of view, the
inventory was correctly sent and so the
standard frequency for sending inventories
is applied. <br>
</li>
</ul>
<br>
The long-term solution for 1/ is to use
something else that FQDN to identify the node
- we have for example an UUID for that. The
problem here is that it is a hard limitation
of the protocol used by cf-serverd. So to use
another identification scheme (and why not our
own authentication/authorization), we need to
either patch cf-serverd or use a different
client-server protocol for promises transfer.
Both solutions are open, but are out of scope.<br>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> Actually, it is
already possible to do authorizations in cf-serverd
based on the key hash of connecting agents: <a href="https://docs.cfengine.com/docs/3.7/reference-promise-types-access.html#admit_keys" target="_blank"><a href="https://docs.cfengine.com/docs/3.7/reference-promise-types-access.html#admit_keys" target="_blank">https://docs.cfengine.com/docs/3.7/reference-promise-types-access.html#admit_keys</a></a>.
This has been possible since CFEngine 3.6 (so works on
Rudder 2.11+ agents).<br>
<br>
We could pretty easily implement this, since we already
have an inventory with nodes CFEngine keys in. It could
be opt-in as a start, while we ensure it's reliable (as
in "more reliable that hostnames"). WDYT?</div>
<div text="#000000" bgcolor="#FFFFFF"><br>
</div>
</blockquote>
<div><br>
</div>
<div><i>This is the most welcome!</i></div>
<div><i><br>
</i></div>
<div><i>The only thing one should not forget about is that
apache is still having its allowed IP Subnets, which are
also not making one's life too easy if you have a ton of
subnets your clients connect from.</i></div>
<div><i><br>
</i></div>
<div><i>Please also do not forget about <a href="https://www.rudder-project.org/redmine/issues/7893" target="_blank"><a href="https://www.rudder-project.org/redmine/issues/7893" target="_blank">https://www.rudder-project.org/redmine/issues/7893</a></a>,
which is about extending the Rudder API with the
possibility to edit the allowed subnets for the policy
servers, which one currently has to do either in the
WebGUI or over some direct-brain-surgery out-of-band
LDAP editing...</i></div>
<div><span style="line-height:1.5"><br>
</span></div>
<div><span style="line-height:1.5"> </span><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"> Meanwhile, we can
address at best 1/ and 2/ <br>
<br>
<b>For 1/, </b>we need to prevent as much as
we can to give bad FQDN and in all case, give
the user a possibility to hook what he knows
should be the correct value. <br>
<b>- Prevent more bad FQDN:</b><b><br>
</b>=> update perl version: <a href="http://www.rudder-project.org/redmine/issues/8123" target="_blank"><a href="http://www.rudder-project.org/redmine/issues/8123" target="_blank">http://www.rudder-project.org/redmine/issues/8123</a></a><br>
I didn't find anything else on that subject,
compared to what we are doing now. <br>
<br>
<b>- Let the user hook the correct value: </b><b><br>
</b>=> <a href="http://www.rudder-project.org/redmine/issues/8022#note-20" target="_blank">http://www.rudder-project.org/redmine/issues/8022#note-20</a><br>
Here, we still need to specify the path
convention for command and file to look for
the correct FDQN. <br>
</blockquote>
I have some remarks/questions here:<br>
<br>
* If I understand correctly the process will be:
Inventory is ran on the Node, then inventory is
modified to add data from commands / file ? or
will this be used when runnig fusionInventory ?
<br>
</blockquote>
<br>
The idea is to patch fusion inventory plugin for
Rudder to use that logic. <br>
</blockquote>
There are already a way to extend inventories: <br>
<a href="http://fusioninventory.org/documentation/agent/additional_content.html" target="_blank">http://fusioninventory.org/documentation/agent/additional_content.html</a><br>
I think it will be much easier and future proof to
use the FusionInventory way rather than
reimplementing it ourselves<br>
<br>
We could use this principle to create our own task
that would read data or run external script - but
maybe that's what you were implying ?<br>
</blockquote>
<br>
No, I meant that we should evolve the logic to fill
<RUDDER><HOSTNAME>, contribute the
evolution to rudder plugin, and patch our own until
then. <br>
I'm not about using the what you linked to for that,
it's really not an extension, but the actual
"business" logic we want to implement for Rudder
<HOSTNAME>. <br>
<br>
On the other hand, what you propose seems really nice
for 4670, I updated the ticket accordingly. <br>
<br>
</blockquote>
</div>
</blockquote>
<div><br>
</div>
<div><i>#4670 is really something very useful, as this would
allow to merge other inventory-collecting systems into
the one rudder collects with fusioninventory, which is
currently not providing any interface to get any
user-defined data, which could be collected on the
server (like Firmware version of different cards, SAP
SID etc.)</i><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"> * In Which entry will
they be stored ? RUDDER/HOSTAME? another one</blockquote>
<br>
Yes, RUDDER/HOSTNAME<br>
<br>
<blockquote type="cite"> * Do we want to define
Hostname only ? Would it not be better if the
solution was much more adaptable and can modify
any entry from the inventory ? <br>
</blockquote>
<br>
It's an other ticket, #4670, linked in #8022.
HOSTNAME (FQDN, really) is different from the
general use case (at least adding information into
inventory) because of the special importance of
FQDN in CFengine server/agent identification
protocol. So it is kind of ok to see #8022 as a
bug in the existing versions, and #4670 (or an
extension of it) as a new feature going to next
versions. <br>
</blockquote>
Yes, let start with HOSTNAME in released version<br>
<blockquote type="cite"> <br>
<blockquote type="cite"> <br>
<br>
About the path, I suggest we should put them
under /var/rudder/inventories :<br>
<br>
<ul>
<li>If we modify only hostname
(RUDDER/HOSTNAME):</li>
<ul>
<li>/var/rudder/inventories/hostname-command:
Command to execute the get the correct
hostname</li>
<li>/var/rudder/inventories/hostname-file:
File containing the path the file
containing the correct hostname</li>
</ul>
</ul>
</blockquote>
<br>
No specific feeling on that... Any idea, other ? <br>
</blockquote>
I'm not sure it should be in /var - if these are
scripts to run to get data, or data user should put,
I feel they would live in other folder.</blockquote>
</blockquote>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> It is pretty
unusual to configure anything in /var. The right place
for configs is /etc. (yes, policy_server.dat is a nasty
exception to this rule). How about
/opt/rudder/etc/hooks/inventory.d/ ? This would be a
good start to having extensible inventory.<br>
<br>
(and yes, it would be desirable to have configs in /etc,
without the /opt/rudder prefix, but we're talking about
a bug fix in existing versions here, so that it clearly
out of scope)</div>
<div text="#000000" bgcolor="#FFFFFF"><br>
<br>
</div>
</blockquote>
<div><br>
</div>
<div><i>Well, you already have stuff in /opt/rudder/etc, so
adding more stuff there would actually make more sense
then opening a third location (well, there <b>is</b> actually
/etc/sysconfig/rudder-apache, which is already a config
below /etc, so you have already stuff laying in there...</i></div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<blockquote type="cite">Plus, if we want user to
preseed the system with these scripts/data, before
installing agent, /var/rudder will not exist<br>
</blockquote>
</blockquote>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> This is irrelevant,
I think. If you need to install a file in a directory,
it's pretty easy to create - in fact, just good
practice.</div>
<div text="#000000" bgcolor="#FFFFFF"><br>
<br>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"> <b>For 2/, </b>we
need to prevent the sending of inventory that
will be rejected by the server for sure. <b><br>
</b>=> <a href="http://www.rudder-project.org/redmine/issues/8127" target="_blank">http://www.rudder-project.org/redmine/issues/8127</a><br>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> FYI, Benoît is
working on this this week. We will implement simple
checks on the most important fields using a Perl script
(since the inventory is already run in Perl, we know
it's available).<br>
<br>
Jon<br>
</div>
_______________________________________________<br>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
<br></div><div text="#000000" bgcolor="#FFFFFF">
So, we sum up everything we have on the subject, and we have
(finally) something coherent with a consistent long term vision. So
let's start by that, it will make other decision clearer. <br>
<br>
In the long term (i.e, not in a bug correction releaes, but can and
should go in next major):<br>
<ul>
<li>nodes don't get a set of complexe initial promise. They have
only a really small one, which mostly like say "go searching
your initial promises from the policy server at that place"</li>
<ul>
<li>this system can be use to also download a set of initial
configuration files, among which can be a set of inventory
hooks, <br>
</li>
</ul>
<li>the inventory processing is done in three steps :</li>
<ul>
<li>1/ produce an inventory file from fusion, <br>
</li>
<li>2/ run hooks from <b>/var/rudder/hooks/inventory.d/*</b> <br>
</li>
<ul>
<li>each hook produce a json file, merged with the previous
json</li>
<li>values are overloaded is a newer hooks define the same key</li>
<li>the resulting JSON file is check for the existence os a
"hostname" key. If it exists, it is extracted (removed) and
its value is used to overwrite
<RUDDER><HOSTNAME></li>
<li>the resulting final JSON file is added in the inventory
file, somewhere in <RUDDER> - it could make sense to
send it apport inventory.xml (like the key part) but the
gain, appart for not yielling because there is JSON in XML,
does not seems obvious</li>
<li>REMARQUE: it would very much make sense that the resulting
JSON file is left on the node, and can be used as a
datasource for node properties in directives etc. Or it will
make sense that along with promise download, the node
download such a JSON file, sourced for data for the
remaining of the run. <br>
</li>
</ul>
<li>3/ the final inventory final is sanity check so that is
necessary data are missing (empty hostname, etc), it is not
sent to the policy server and an error is logged. <br>
</li>
</ul>
<li>all the content from the json file is added as properties of
the node (and will be searchable, usable to define group, usable
as variable in directives, etc). <br>
</li>
</ul>
<br>
This solve nicelly all the bug related to 8022:<br>
<br>
<ul>
<li>for 8022 itself, it means :<br>
</li>
<ul>
<li>that we can had a defaults"hostname" hook, with the logic
"if RUDDER_FQDN is defined non empty" <br>
</li>
<li>users are free to add other hooks to overide it ("if
RUDDER_FQDN key is defined in file ...., use that" etc)</li>
<li>and that can be done since Rudder 3.1 without problem,
because it won't change current behaviour</li>
</ul>
<li>for 8127 (check inventory before sending them), it changes
nothing : it's step 3</li>
<li>we also checked 8123 (update perl version), and the amount of
work to get there is huge and risky. So 8123 will go in next
major, and we are going to try to solve 8283 ASAP</li>
<li>finally, most of 4670 will be done: all the agent logic be
there, and rudder will be extended to actually use the data in
next major. <br>
</li>
</ul>
<p><br>
</p>
<p>Hope everhting is clear !<br>
</p></div><div text="#000000" bgcolor="#FFFFFF">
<div>-- <br>
<table cellpadding="0" cellspacing="2" width="380" border="0">
<tbody>
<tr>
<td colspan="2">
<hr></td>
</tr>
<tr>
<td colspan="2"><b><img alt="" src="cid:part9.7B7949AB.0E6984A1@normation.com" width="50" align="left" height="50" hspace="10"> <span>François ARMAND</span></b><br>
<span><i>Co-founder & CTO</i></span><br>
<span><a href="http://www.normation.com" target="_blank">Normation</a></span> </td>
</tr>
<tr>
<td colspan="2">
<hr></td>
</tr>
<tr>
<td colspan="2"><span><b>87 rue de Turbigo,
75003 Paris, France</b></span></td>
</tr>
<tr>
<td><span>Telephone:</span></td>
<td><span>+33 (0)1 83 62 99 23</span></td>
</tr>
<tr>
<td><span>Mobile:</span></td>
<td><span>+33 (0)6 63 37 60 55</span></td>
</tr>
<tr>
<td colspan="2">
<hr></td>
</tr>
</tbody>
</table>
</div>
</div></blockquote></div>