[rudder-dev] Having correct FQDN in (first) inventory
Francois Armand
francois.armand at normation.com
Fri May 13 17:44:34 CEST 2016
On 13/05/2016 17:16, Mattyasovszky János wrote:
>
> Hi
>
> Just some minor questions / summaries of your summaries :)
> - the node will not begin with uploading an inventory on the initial
> "bootstrapping" run, only after it received its initial promises from
> the policy server.
>
Yes, that's the idea: the futur embeded initial promises will be just
"download real stuff from the server". That would allow to make
evolution on these actual logic in promises (do inventory, etc) much
more easier to evolve in the future, because nothing change on nodes
embeded code.
>
>
> - the bootstrapping code will still have a hook to define the
> Hostname, but it can be overwritten by later customer hooks?
>
Yes
> -> Q: will the Hooks be executed by their names sorted lexically?
>
Yes, and we will put default hooks to follow a 100-xxx, 200-xxx, etc
convention. We think that it will let sufficient space for future hooks.
> -> Q: will all nodes receive the same additional hook plugins?
>
We are still thinking about that, but what we can say for sure is:
* before node acceptation, we don't think it is necessary to had a
generic way to make nodes download different hooks based on some
property. We may had a way to let nodes get different
initialpromises (but not sure about that), and if so, hooks will get
the same behaviour. And the user will still be able to define hooks
starting with a test to decide if the hooks should actually run on
that node.
* after node acceptation, we have no definitive idea, but of course,
as nodes have access to dedicated things to download (their
promises, at least ;), it would make sense to let them also have
dedicated hooks, managed from Rudder (among other things).
> -> Q: will the inventory contain the json output 1:1 or will it be
> converted to xml?
>
We think it is better to actually included the real json, because xml
<-> json translation is not direct, and we believe it will likelly
create case where the final data is not the same as the original one.
> // can one only use key: value pairs or more complex data structures
> to extend the inventory?
>
You will be able to add real json, so whatever data structure you want
till the final format is a valid json file.
> (NB: Using the collected "facts" as variables is cool, but please
> don't forget to also add the centrally managed node properties as vars
> to the node...)
>
Yes.
Hope it makes thing clearer.
> Thanks
> Janos
>
>
> Francois Armand <francois.armand at normation.com
> <mailto:francois.armand at normation.com>> ezt írta (időpont: 2016. máj.
> 13., P 16:59):
>
> On 02/05/2016 22:13, Mattyasovszky János wrote:
>> Hi
>>
>> /Answers Inline.
>> /
>> Regards,
>> Matya
>>
>> Jonathan Clarke <jonathan.clarke at normation.com
>> <mailto:jonathan.clarke at normation.com>> ezt írta (időpont: 2016.
>> máj. 2., H, 18:29):
>>
>> Hi,
>>
>>
>> On 04/13/2016 11:48 AM, Francois wrote:
>>> On 13/04/2016 09:49, Nicolas Charles wrote:
>>>> Hello Francois,
>>>>
>>>> Thank you for the nice sum-up. My answers in the text
>>
>> +1
>>
>>
>>>> Le 07/04/2016 15:33, Francois Armand a écrit :
>>>>> On 07/04/2016 15:08, Vincent Membré wrote:
>>>>>> Le 31/03/2016 12:37, Francois Armand a écrit :
>>>>>>> Hello,
>>>>>>>
>>>>>>> So, here goes for a summary of
>>>>>>> http://www.rudder-project.org/redmine/issues/8022,
>>>>>>> "Node's FQDN-Resolution is sometimes invalid" and
>>>>>>> related tickets.
>>>>>>> The problem cover up several sub-cases, which need to be
>>>>>>> addressed systematically to achieve some result.
>>>>>>> They are:
>>>>>>>
>>>>>>> * 1/ the node FQDN is used for identifying a node, and
>>>>>>> then manage authentication and authorization to
>>>>>>> access its promises. If Rudder server, CFEngine
>>>>>>> promise server, and the node don't agree on it, the
>>>>>>> node can't get its promises. This is a hard problem
>>>>>>> because:
>>>>>>> o FQDN is fragile. It needs a perfectly up to date
>>>>>>> and shared DNS environment. But "it's always a
>>>>>>> DNS problem", what gives an idea.
>>>>>>> o FQDN tools are notoriously broken, and don't
>>>>>>> always agree about what is the FQDN of a host
>>>>>>> o even in a perfectly working and up-to-date DNS
>>>>>>> env, there may have voluntary decision not to
>>>>>>> have the same FQDN from the host and the server,
>>>>>>> as explained in #8022 ticket.
>>>>>>> * 2/ if the node FQDN is not correct in the first sent
>>>>>>> inventory, the node configuration is delayed by one
>>>>>>> day, because from the node point of view, the
>>>>>>> inventory was correctly sent and so the standard
>>>>>>> frequency for sending inventories is applied.
>>>>>>>
>>>>>>>
>>>>>>> The long-term solution for 1/ is to use something else
>>>>>>> that FQDN to identify the node - we have for example an
>>>>>>> UUID for that. The problem here is that it is a hard
>>>>>>> limitation of the protocol used by cf-serverd. So to use
>>>>>>> another identification scheme (and why not our own
>>>>>>> authentication/authorization), we need to either patch
>>>>>>> cf-serverd or use a different client-server protocol for
>>>>>>> promises transfer. Both solutions are open, but are out
>>>>>>> of scope.
>>
>> Actually, it is already possible to do authorizations in
>> cf-serverd based on the key hash of connecting agents:
>> https://docs.cfengine.com/docs/3.7/reference-promise-types-access.html#admit_keys.
>> This has been possible since CFEngine 3.6 (so works on Rudder
>> 2.11+ agents).
>>
>> We could pretty easily implement this, since we already have
>> an inventory with nodes CFEngine keys in. It could be opt-in
>> as a start, while we ensure it's reliable (as in "more
>> reliable that hostnames"). WDYT?
>>
>>
>> /This is the most welcome!/
>> /
>> /
>> /The only thing one should not forget about is that apache is
>> still having its allowed IP Subnets, which are also not making
>> one's life too easy if you have a ton of subnets your clients
>> connect from./
>> /
>> /
>> /Please also do not forget about
>> https://www.rudder-project.org/redmine/issues/7893, which is
>> about extending the Rudder API with the possibility to edit the
>> allowed subnets for the policy servers, which one currently has
>> to do either in the WebGUI or over some direct-brain-surgery
>> out-of-band LDAP editing.../
>>
>>
>>>>>>> Meanwhile, we can address at best 1/ and 2/
>>>>>>>
>>>>>>> *For 1/, *we need to prevent as much as we can to give
>>>>>>> bad FQDN and in all case, give the user a possibility to
>>>>>>> hook what he knows should be the correct value.
>>>>>>> *- Prevent more bad FQDN:**
>>>>>>> *=> update perl version:
>>>>>>> http://www.rudder-project.org/redmine/issues/8123
>>>>>>> I didn't find anything else on that subject, compared to
>>>>>>> what we are doing now.
>>>>>>>
>>>>>>> *- Let the user hook the correct value: **
>>>>>>> *=>
>>>>>>> http://www.rudder-project.org/redmine/issues/8022#note-20
>>>>>>> Here, we still need to specify the path convention for
>>>>>>> command and file to look for the correct FDQN.
>>>>>> I have some remarks/questions here:
>>>>>>
>>>>>> * If I understand correctly the process will be:
>>>>>> Inventory is ran on the Node, then inventory is modified
>>>>>> to add data from commands / file ? or will this be used
>>>>>> when runnig fusionInventory ?
>>>>>
>>>>> The idea is to patch fusion inventory plugin for Rudder to
>>>>> use that logic.
>>>> There are already a way to extend inventories:
>>>> http://fusioninventory.org/documentation/agent/additional_content.html
>>>> I think it will be much easier and future proof to use the
>>>> FusionInventory way rather than reimplementing it ourselves
>>>>
>>>> We could use this principle to create our own task that
>>>> would read data or run external script - but maybe that's
>>>> what you were implying ?
>>>
>>> No, I meant that we should evolve the logic to fill
>>> <RUDDER><HOSTNAME>, contribute the evolution to rudder
>>> plugin, and patch our own until then.
>>> I'm not about using the what you linked to for that, it's
>>> really not an extension, but the actual "business" logic we
>>> want to implement for Rudder <HOSTNAME>.
>>>
>>> On the other hand, what you propose seems really nice for
>>> 4670, I updated the ticket accordingly.
>>>
>>
>> /#4670 is really something very useful, as this would allow to
>> merge other inventory-collecting systems into the one rudder
>> collects with fusioninventory, which is currently not providing
>> any interface to get any user-defined data, which could be
>> collected on the server (like Firmware version of different
>> cards, SAP SID etc.)/
>>
>>>>>> * In Which entry will they be stored ? RUDDER/HOSTAME?
>>>>>> another one
>>>>>
>>>>> Yes, RUDDER/HOSTNAME
>>>>>
>>>>>> * Do we want to define Hostname only ? Would it not be
>>>>>> better if the solution was much more adaptable and can
>>>>>> modify any entry from the inventory ?
>>>>>
>>>>> It's an other ticket, #4670, linked in #8022. HOSTNAME
>>>>> (FQDN, really) is different from the general use case (at
>>>>> least adding information into inventory) because of the
>>>>> special importance of FQDN in CFengine server/agent
>>>>> identification protocol. So it is kind of ok to see #8022
>>>>> as a bug in the existing versions, and #4670 (or an
>>>>> extension of it) as a new feature going to next versions.
>>>> Yes, let start with HOSTNAME in released version
>>>>>
>>>>>>
>>>>>>
>>>>>> About the path, I suggest we should put them under
>>>>>> /var/rudder/inventories :
>>>>>>
>>>>>> * If we modify only hostname (RUDDER/HOSTNAME):
>>>>>> o /var/rudder/inventories/hostname-command: Command
>>>>>> to execute the get the correct hostname
>>>>>> o /var/rudder/inventories/hostname-file: File
>>>>>> containing the path the file containing the
>>>>>> correct hostname
>>>>>>
>>>>>
>>>>> No specific feeling on that... Any idea, other ?
>>>> I'm not sure it should be in /var - if these are scripts to
>>>> run to get data, or data user should put, I feel they would
>>>> live in other folder.
>>
>> It is pretty unusual to configure anything in /var. The right
>> place for configs is /etc. (yes, policy_server.dat is a nasty
>> exception to this rule). How about
>> /opt/rudder/etc/hooks/inventory.d/ ? This would be a good
>> start to having extensible inventory.
>>
>> (and yes, it would be desirable to have configs in /etc,
>> without the /opt/rudder prefix, but we're talking about a bug
>> fix in existing versions here, so that it clearly out of scope)
>>
>>
>>
>> /Well, you already have stuff in /opt/rudder/etc, so adding more
>> stuff there would actually make more sense then opening a third
>> location (well, there *is* actually /etc/sysconfig/rudder-apache,
>> which is already a config below /etc, so you have already stuff
>> laying in there.../
>>
>>>> Plus, if we want user to preseed the system with these
>>>> scripts/data, before installing agent, /var/rudder will not
>>>> exist
>>
>> This is irrelevant, I think. If you need to install a file in
>> a directory, it's pretty easy to create - in fact, just good
>> practice.
>>
>>
>>>>>>> *For 2/, *we need to prevent the sending of inventory
>>>>>>> that will be rejected by the server for sure. *
>>>>>>> *=> http://www.rudder-project.org/redmine/issues/8127
>>
>> FYI, Benoît is working on this this week. We will implement
>> simple checks on the most important fields using a Perl
>> script (since the inventory is already run in Perl, we know
>> it's available).
>>
>> Jon
>> _______________________________________________
>>
>
>
> So, we sum up everything we have on the subject, and we have
> (finally) something coherent with a consistent long term vision.
> So let's start by that, it will make other decision clearer.
>
> In the long term (i.e, not in a bug correction releaes, but can
> and should go in next major):
>
> * nodes don't get a set of complexe initial promise. They have
> only a really small one, which mostly like say "go searching
> your initial promises from the policy server at that place"
> o this system can be use to also download a set of initial
> configuration files, among which can be a set of inventory
> hooks,
> * the inventory processing is done in three steps :
> o 1/ produce an inventory file from fusion,
> o 2/ run hooks from */var/rudder/hooks/inventory.d/**
> + each hook produce a json file, merged with the
> previous json
> + values are overloaded is a newer hooks define the same key
> + the resulting JSON file is check for the existence os
> a "hostname" key. If it exists, it is extracted
> (removed) and its value is used to overwrite
> <RUDDER><HOSTNAME>
> + the resulting final JSON file is added in the
> inventory file, somewhere in <RUDDER> - it could make
> sense to send it apport inventory.xml (like the key
> part) but the gain, appart for not yielling because
> there is JSON in XML, does not seems obvious
> + REMARQUE: it would very much make sense that the
> resulting JSON file is left on the node, and can be
> used as a datasource for node properties in directives
> etc. Or it will make sense that along with promise
> download, the node download such a JSON file, sourced
> for data for the remaining of the run.
> o 3/ the final inventory final is sanity check so that is
> necessary data are missing (empty hostname, etc), it is
> not sent to the policy server and an error is logged.
> * all the content from the json file is added as properties of
> the node (and will be searchable, usable to define group,
> usable as variable in directives, etc).
>
>
> This solve nicelly all the bug related to 8022:
>
> * for 8022 itself, it means :
> o that we can had a defaults"hostname" hook, with the logic
> "if RUDDER_FQDN is defined non empty"
> o users are free to add other hooks to overide it ("if
> RUDDER_FQDN key is defined in file ...., use that" etc)
> o and that can be done since Rudder 3.1 without problem,
> because it won't change current behaviour
> * for 8127 (check inventory before sending them), it changes
> nothing : it's step 3
> * we also checked 8123 (update perl version), and the amount of
> work to get there is huge and risky. So 8123 will go in next
> major, and we are going to try to solve 8283 ASAP
> * finally, most of 4670 will be done: all the agent logic be
> there, and rudder will be extended to actually use the data in
> next major.
>
>
> Hope everhting is clear !
>
> --
> ------------------------------------------------------------------------
> *François ARMAND*
> /Co-founder & CTO/
> Normation <http://www.normation.com>
> ------------------------------------------------------------------------
> *87 rue de Turbigo, 75003 Paris, France*
> Telephone: +33 (0)1 83 62 99 23
> Mobile: +33 (0)6 63 37 60 55
> ------------------------------------------------------------------------
>
--
------------------------------------------------------------------------
*François ARMAND*
/Co-founder & CTO/
Normation <http://www.normation.com>
------------------------------------------------------------------------
*87 rue de Turbigo, 75003 Paris, France*
Telephone: +33 (0)1 83 62 99 23
Mobile: +33 (0)6 63 37 60 55
------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.rudder-project.org/pipermail/rudder-dev/attachments/20160513/f19d453a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logo-square.gif
Type: image/gif
Size: 1036 bytes
Desc: not available
URL: <http://www.rudder-project.org/pipermail/rudder-dev/attachments/20160513/f19d453a/attachment-0001.gif>
More information about the rudder-dev
mailing list