<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 13/05/2016 17:16, Mattyasovszky
János wrote:<br>
</div>
<blockquote
cite="mid:CAPz0yRuXOmG3024Y=OyYsGBw7FtDdDvxLsH-hporqQ6EN1CE-Q@mail.gmail.com"
type="cite">
<p dir="ltr">Hi</p>
<p dir="ltr">Just some minor questions / summaries of your
summaries :) <br>
- the node will not begin with uploading an inventory on the
initial "bootstrapping" run, only after it received its initial
promises from the policy server. </p>
</blockquote>
Yes, that's the idea: the futur embeded initial promises will be
just "download real stuff from the server". That would allow to make
evolution on these actual logic in promises (do inventory, etc) much
more easier to evolve in the future, because nothing change on nodes
embeded code. <br>
<blockquote
cite="mid:CAPz0yRuXOmG3024Y=OyYsGBw7FtDdDvxLsH-hporqQ6EN1CE-Q@mail.gmail.com"
type="cite">
<p dir="ltr"><br>
- the bootstrapping code will still have a hook to define the
Hostname, but it can be overwritten by later customer hooks?<br>
</p>
</blockquote>
<br>
Yes<br>
<br>
<blockquote
cite="mid:CAPz0yRuXOmG3024Y=OyYsGBw7FtDdDvxLsH-hporqQ6EN1CE-Q@mail.gmail.com"
type="cite">
<p dir="ltr">
-> Q: will the Hooks be executed by their names sorted
lexically? <br>
</p>
</blockquote>
<br>
Yes, and we will put default hooks to follow a 100-xxx, 200-xxx, etc
convention. We think that it will let sufficient space for future
hooks.<br>
<br>
<blockquote
cite="mid:CAPz0yRuXOmG3024Y=OyYsGBw7FtDdDvxLsH-hporqQ6EN1CE-Q@mail.gmail.com"
type="cite">
<p dir="ltr">
-> Q: will all nodes receive the same additional hook
plugins? <br>
</p>
</blockquote>
<br>
We are still thinking about that, but what we can say for sure is:<br>
<ul>
<li>before node acceptation, we don't think it is necessary to had
a generic way to make nodes download different hooks based on
some property. We may had a way to let nodes get different
initialpromises (but not sure about that), and if so, hooks will
get the same behaviour. And the user will still be able to
define hooks starting with a test to decide if the hooks should
actually run on that node. <br>
</li>
<li>after node acceptation, we have no definitive idea, but of
course, as nodes have access to dedicated things to download
(their promises, at least ;), it would make sense to let them
also have dedicated hooks, managed from Rudder (among other
things). <br>
</li>
</ul>
<br>
<blockquote
cite="mid:CAPz0yRuXOmG3024Y=OyYsGBw7FtDdDvxLsH-hporqQ6EN1CE-Q@mail.gmail.com"
type="cite">
<p dir="ltr">
-> Q: will the inventory contain the json output 1:1 or will
it be converted to xml? </p>
</blockquote>
<br>
We think it is better to actually included the real json, because
xml <-> json translation is not direct, and we believe it will
likelly create case where the final data is not the same as the
original one. <br>
<br>
<blockquote
cite="mid:CAPz0yRuXOmG3024Y=OyYsGBw7FtDdDvxLsH-hporqQ6EN1CE-Q@mail.gmail.com"
type="cite">
<p dir="ltr">// can one only use key: value pairs or more complex
data structures to extend the inventory? </p>
</blockquote>
<br>
You will be able to add real json, so whatever data structure you
want till the final format is a valid json file. <br>
<br>
<blockquote
cite="mid:CAPz0yRuXOmG3024Y=OyYsGBw7FtDdDvxLsH-hporqQ6EN1CE-Q@mail.gmail.com"
type="cite">
<p dir="ltr">(NB: Using the collected "facts" as variables is
cool, but please don't forget to also add the centrally managed
node properties as vars to the node...) </p>
</blockquote>
<br>
Yes. <br>
<br>
Hope it makes thing clearer. <br>
<br>
<blockquote
cite="mid:CAPz0yRuXOmG3024Y=OyYsGBw7FtDdDvxLsH-hporqQ6EN1CE-Q@mail.gmail.com"
type="cite">
<p dir="ltr">Thanks <br>
Janos <br>
</p>
<br>
<div class="gmail_quote">
<div dir="ltr">Francois Armand <<a moz-do-not-send="true"
href="mailto:francois.armand@normation.com">francois.armand@normation.com</a>>
ezt írta (időpont: 2016. máj. 13., P 16:59):<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>On 02/05/2016 22:13, Mattyasovszky János wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi<br>
<br>
<i>Answers Inline.<br>
</i><br>
Regards,<br>
Matya
<div><br>
<div class="gmail_quote">
<div dir="ltr">Jonathan Clarke <<a
moz-do-not-send="true"
href="mailto:jonathan.clarke@normation.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:jonathan.clarke@normation.com">jonathan.clarke@normation.com</a></a>>
ezt írta (időpont: 2016. máj. 2., H, 18:29):<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi,</div>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<div><br>
<br>
On 04/13/2016 11:48 AM, Francois wrote:<br>
</div>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div>On 13/04/2016 09:49, Nicolas Charles
wrote:<br>
</div>
<blockquote type="cite">
<div>Hello Francois, <br>
<br>
Thank you for the nice sum-up. My answers
in the text<br>
</div>
</blockquote>
</blockquote>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> +1</div>
<div text="#000000" bgcolor="#FFFFFF"><br>
<br>
<blockquote type="cite">
<blockquote type="cite">
<div> Le 07/04/2016 15:33, Francois Armand a
écrit :<br>
</div>
<blockquote type="cite">
<div>On 07/04/2016 15:08, Vincent Membré
wrote:<br>
</div>
<blockquote type="cite">
<div>Le 31/03/2016 12:37, Francois
Armand a écrit :<br>
</div>
<blockquote type="cite"> Hello, <br>
<br>
So, here goes for a summary of <a
moz-do-not-send="true"
href="http://www.rudder-project.org/redmine/issues/8022"
target="_blank"><a class="moz-txt-link-freetext" href="http://www.rudder-project.org/redmine/issues/8022">http://www.rudder-project.org/redmine/issues/8022</a></a>,
"Node's FQDN-Resolution is sometimes
invalid" and related tickets. <br>
The problem cover up several
sub-cases, which need to be addressed
systematically to achieve some result.<br>
They are: <br>
<ul>
<li>1/ the node FQDN is used for
identifying a node, and then
manage authentication and
authorization to access its
promises. If Rudder server,
CFEngine promise server, and the
node don't agree on it, the node
can't get its promises. This is a
hard problem because:</li>
<ul>
<li>FQDN is fragile. It needs a
perfectly up to date and shared
DNS environment. But "it's
always a DNS problem", what
gives an idea. <br>
</li>
<li>FQDN tools are notoriously
broken, and don't always agree
about what is the FQDN of a host</li>
<li>even in a perfectly working
and up-to-date DNS env, there
may have voluntary decision not
to have the same FQDN from the
host and the server, as
explained in #8022 ticket. <br>
</li>
</ul>
<li>2/ if the node FQDN is not
correct in the first sent
inventory, the node configuration
is delayed by one day, because
from the node point of view, the
inventory was correctly sent and
so the standard frequency for
sending inventories is applied. <br>
</li>
</ul>
<br>
The long-term solution for 1/ is to
use something else that FQDN to
identify the node - we have for
example an UUID for that. The problem
here is that it is a hard limitation
of the protocol used by cf-serverd. So
to use another identification scheme
(and why not our own
authentication/authorization), we need
to either patch cf-serverd or use a
different client-server protocol for
promises transfer. Both solutions are
open, but are out of scope.<br>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> Actually,
it is already possible to do authorizations in
cf-serverd based on the key hash of connecting
agents: <a moz-do-not-send="true"
href="https://docs.cfengine.com/docs/3.7/reference-promise-types-access.html#admit_keys"
target="_blank">https://docs.cfengine.com/docs/3.7/reference-promise-types-access.html#admit_keys</a>.
This has been possible since CFEngine 3.6 (so
works on Rudder 2.11+ agents).<br>
<br>
We could pretty easily implement this, since we
already have an inventory with nodes CFEngine
keys in. It could be opt-in as a start, while we
ensure it's reliable (as in "more reliable that
hostnames"). WDYT?</div>
<div text="#000000" bgcolor="#FFFFFF"><br>
</div>
</blockquote>
<div><br>
</div>
<div><i>This is the most welcome!</i></div>
<div><i><br>
</i></div>
<div><i>The only thing one should not forget about
is that apache is still having its allowed IP
Subnets, which are also not making one's life
too easy if you have a ton of subnets your
clients connect from.</i></div>
<div><i><br>
</i></div>
<div><i>Please also do not forget about <a
moz-do-not-send="true"
href="https://www.rudder-project.org/redmine/issues/7893"
target="_blank"><a class="moz-txt-link-freetext" href="https://www.rudder-project.org/redmine/issues/7893">https://www.rudder-project.org/redmine/issues/7893</a></a>,
which is about extending the Rudder API with the
possibility to edit the allowed subnets for the
policy servers, which one currently has to do
either in the WebGUI or over some
direct-brain-surgery out-of-band LDAP editing...</i></div>
<div><span style="line-height:1.5"><br>
</span></div>
<div><span style="line-height:1.5"> </span><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"> Meanwhile, we
can address at best 1/ and 2/ <br>
<br>
<b>For 1/, </b>we need to prevent as
much as we can to give bad FQDN and in
all case, give the user a possibility
to hook what he knows should be the
correct value. <br>
<b>- Prevent more bad FQDN:</b><b><br>
</b>=> update perl version: <a
moz-do-not-send="true"
href="http://www.rudder-project.org/redmine/issues/8123"
target="_blank"><a class="moz-txt-link-freetext" href="http://www.rudder-project.org/redmine/issues/8123">http://www.rudder-project.org/redmine/issues/8123</a></a><br>
I didn't find anything else on that
subject, compared to what we are doing
now. <br>
<br>
<b>- Let the user hook the correct
value: </b><b><br>
</b>=> <a moz-do-not-send="true"
href="http://www.rudder-project.org/redmine/issues/8022#note-20"
target="_blank">http://www.rudder-project.org/redmine/issues/8022#note-20</a><br>
Here, we still need to specify the
path convention for command and file
to look for the correct FDQN. <br>
</blockquote>
I have some remarks/questions here:<br>
<br>
* If I understand correctly the process
will be: Inventory is ran on the Node,
then inventory is modified to add data
from commands / file ? or will this be
used when runnig fusionInventory ? <br>
</blockquote>
<br>
The idea is to patch fusion inventory
plugin for Rudder to use that logic. <br>
</blockquote>
There are already a way to extend
inventories: <br>
<a moz-do-not-send="true"
href="http://fusioninventory.org/documentation/agent/additional_content.html"
target="_blank">http://fusioninventory.org/documentation/agent/additional_content.html</a><br>
I think it will be much easier and future
proof to use the FusionInventory way rather
than reimplementing it ourselves<br>
<br>
We could use this principle to create our
own task that would read data or run
external script - but maybe that's what you
were implying ?<br>
</blockquote>
<br>
No, I meant that we should evolve the logic to
fill <RUDDER><HOSTNAME>,
contribute the evolution to rudder plugin, and
patch our own until then. <br>
I'm not about using the what you linked to for
that, it's really not an extension, but the
actual "business" logic we want to implement
for Rudder <HOSTNAME>. <br>
<br>
On the other hand, what you propose seems
really nice for 4670, I updated the ticket
accordingly. <br>
<br>
</blockquote>
</div>
</blockquote>
<div><br>
</div>
<div><i>#4670 is really something very useful, as
this would allow to merge other
inventory-collecting systems into the one rudder
collects with fusioninventory, which is
currently not providing any interface to get any
user-defined data, which could be collected on
the server (like Firmware version of different
cards, SAP SID etc.)</i><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"> * In Which entry
will they be stored ? RUDDER/HOSTAME?
another one</blockquote>
<br>
Yes, RUDDER/HOSTNAME<br>
<br>
<blockquote type="cite"> * Do we want to
define Hostname only ? Would it not be
better if the solution was much more
adaptable and can modify any entry from
the inventory ? <br>
</blockquote>
<br>
It's an other ticket, #4670, linked in
#8022. HOSTNAME (FQDN, really) is
different from the general use case (at
least adding information into inventory)
because of the special importance of FQDN
in CFengine server/agent identification
protocol. So it is kind of ok to see #8022
as a bug in the existing versions, and
#4670 (or an extension of it) as a new
feature going to next versions. <br>
</blockquote>
Yes, let start with HOSTNAME in released
version<br>
<blockquote type="cite"> <br>
<blockquote type="cite"> <br>
<br>
About the path, I suggest we should put
them under /var/rudder/inventories :<br>
<br>
<ul>
<li>If we modify only hostname
(RUDDER/HOSTNAME):</li>
<ul>
<li>/var/rudder/inventories/hostname-command:
Command to execute the get the
correct hostname</li>
<li>/var/rudder/inventories/hostname-file:
File containing the path the file
containing the correct hostname</li>
</ul>
</ul>
</blockquote>
<br>
No specific feeling on that... Any idea,
other ? <br>
</blockquote>
I'm not sure it should be in /var - if these
are scripts to run to get data, or data user
should put, I feel they would live in other
folder.</blockquote>
</blockquote>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> It is
pretty unusual to configure anything in /var.
The right place for configs is /etc. (yes,
policy_server.dat is a nasty exception to this
rule). How about
/opt/rudder/etc/hooks/inventory.d/ ? This would
be a good start to having extensible inventory.<br>
<br>
(and yes, it would be desirable to have configs
in /etc, without the /opt/rudder prefix, but
we're talking about a bug fix in existing
versions here, so that it clearly out of scope)</div>
<div text="#000000" bgcolor="#FFFFFF"><br>
<br>
</div>
</blockquote>
<div><br>
</div>
<div><i>Well, you already have stuff in
/opt/rudder/etc, so adding more stuff there
would actually make more sense then opening a
third location (well, there <b>is</b> actually
/etc/sysconfig/rudder-apache, which is already a
config below /etc, so you have already stuff
laying in there...</i></div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<blockquote type="cite">Plus, if we want user
to preseed the system with these
scripts/data, before installing agent,
/var/rudder will not exist<br>
</blockquote>
</blockquote>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> This is
irrelevant, I think. If you need to install a
file in a directory, it's pretty easy to create
- in fact, just good practice.</div>
<div text="#000000" bgcolor="#FFFFFF"><br>
<br>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"> <b>For 2/, </b>we
need to prevent the sending of
inventory that will be rejected by the
server for sure. <b><br>
</b>=> <a moz-do-not-send="true"
href="http://www.rudder-project.org/redmine/issues/8127"
target="_blank">http://www.rudder-project.org/redmine/issues/8127</a><br>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> FYI, Benoît
is working on this this week. We will implement
simple checks on the most important fields using
a Perl script (since the inventory is already
run in Perl, we know it's available).<br>
<br>
Jon<br>
</div>
_______________________________________________<br>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
<br>
</div>
<div text="#000000" bgcolor="#FFFFFF"> So, we sum up
everything we have on the subject, and we have (finally)
something coherent with a consistent long term vision. So
let's start by that, it will make other decision clearer. <br>
<br>
In the long term (i.e, not in a bug correction releaes, but
can and should go in next major):<br>
<ul>
<li>nodes don't get a set of complexe initial promise.
They have only a really small one, which mostly like say
"go searching your initial promises from the policy
server at that place"</li>
<ul>
<li>this system can be use to also download a set of
initial configuration files, among which can be a set
of inventory hooks, <br>
</li>
</ul>
<li>the inventory processing is done in three steps :</li>
<ul>
<li>1/ produce an inventory file from fusion, <br>
</li>
<li>2/ run hooks from <b>/var/rudder/hooks/inventory.d/*</b>
<br>
</li>
<ul>
<li>each hook produce a json file, merged with the
previous json</li>
<li>values are overloaded is a newer hooks define the
same key</li>
<li>the resulting JSON file is check for the existence
os a "hostname" key. If it exists, it is extracted
(removed) and its value is used to overwrite
<RUDDER><HOSTNAME></li>
<li>the resulting final JSON file is added in the
inventory file, somewhere in <RUDDER> - it
could make sense to send it apport inventory.xml
(like the key part) but the gain, appart for not
yielling because there is JSON in XML, does not
seems obvious</li>
<li>REMARQUE: it would very much make sense that the
resulting JSON file is left on the node, and can be
used as a datasource for node properties in
directives etc. Or it will make sense that along
with promise download, the node download such a JSON
file, sourced for data for the remaining of the run.
<br>
</li>
</ul>
<li>3/ the final inventory final is sanity check so that
is necessary data are missing (empty hostname, etc),
it is not sent to the policy server and an error is
logged. <br>
</li>
</ul>
<li>all the content from the json file is added as
properties of the node (and will be searchable, usable
to define group, usable as variable in directives, etc).
<br>
</li>
</ul>
<br>
This solve nicelly all the bug related to 8022:<br>
<br>
<ul>
<li>for 8022 itself, it means :<br>
</li>
<ul>
<li>that we can had a defaults"hostname" hook, with the
logic "if RUDDER_FQDN is defined non empty" <br>
</li>
<li>users are free to add other hooks to overide it ("if
RUDDER_FQDN key is defined in file ...., use that"
etc)</li>
<li>and that can be done since Rudder 3.1 without
problem, because it won't change current behaviour</li>
</ul>
<li>for 8127 (check inventory before sending them), it
changes nothing : it's step 3</li>
<li>we also checked 8123 (update perl version), and the
amount of work to get there is huge and risky. So 8123
will go in next major, and we are going to try to solve
8283 ASAP</li>
<li>finally, most of 4670 will be done: all the agent
logic be there, and rudder will be extended to actually
use the data in next major. <br>
</li>
</ul>
<p><br>
</p>
<p>Hope everhting is clear !<br>
</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<div>-- <br>
<table cellpadding="0" cellspacing="2" width="380"
border="0">
<tbody>
<tr>
<td colspan="2">
<hr></td>
</tr>
<tr>
<td colspan="2"><b><img moz-do-not-send="true"
alt=""
src="cid:part9.7B7949AB.0E6984A1@normation.com"
width="50" align="left" height="50"
hspace="10"> <span>François ARMAND</span></b><br>
<span><i>Co-founder & CTO</i></span><br>
<span><a moz-do-not-send="true"
href="http://www.normation.com"
target="_blank">Normation</a></span> </td>
</tr>
<tr>
<td colspan="2">
<hr></td>
</tr>
<tr>
<td colspan="2"><span><b>87 rue de Turbigo, 75003
Paris, France</b></span></td>
</tr>
<tr>
<td><span>Telephone:</span></td>
<td><span>+33 (0)1 83 62 99 23</span></td>
</tr>
<tr>
<td><span>Mobile:</span></td>
<td><span>+33 (0)6 63 37 60 55</span></td>
</tr>
<tr>
<td colspan="2">
<hr></td>
</tr>
</tbody>
</table>
</div>
</div>
</blockquote>
</div>
</blockquote>
<br>
<p><br>
</p>
<div class="moz-signature">-- <br>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<style type="text/css"><!--
a.redlink:link { color: #1782E6; text-decoration: none; }
a.redlink:visited { color: #1782E6; text-decoration: none; }
.sig { font-family: 'Century Gothic', CenturyGothic, AppleGothic, sans-serif; font-size: small; }
.sigsmall { font-family: 'Century Gothic', CenturyGothic, AppleGothic, sans-serif; font-size: x-small; }
--></style>
<table cellpadding="0" cellspacing="2" width="380" border="0">
<tbody>
<tr>
<td colspan="2">
<hr></td>
</tr>
<tr>
<td colspan="2"><b><img alt=""
src="cid:part12.E70CA480.C7867D10@normation.com"
width="50" align="left" height="50" hspace="10"> <span
class="sig">François ARMAND</span></b><br>
<span class="sig"><i>Co-founder & CTO</i></span><br>
<span class="sig"><a class="redlink"
href="http://www.normation.com">Normation</a></span> </td>
</tr>
<tr>
<td colspan="2">
<hr></td>
</tr>
<tr>
<td colspan="2"><span class="sigsmall"><b>87 rue de Turbigo,
75003 Paris, France</b></span></td>
</tr>
<tr>
<td><span class="sigsmall">Telephone:</span></td>
<td><span class="sigsmall">+33 (0)1 83 62 99 23</span></td>
</tr>
<tr>
<td><span class="sigsmall">Mobile:</span></td>
<td><span class="sigsmall">+33 (0)6 63 37 60 55</span></td>
</tr>
<tr>
<td colspan="2">
<hr></td>
</tr>
</tbody>
</table>
</div>
</body>
</html>