[rudder-dev] Having correct FQDN in (first) inventory
Mattyasovszky János
mail at matya.hu
Fri May 13 17:16:51 CEST 2016
Hi
Just some minor questions / summaries of your summaries :)
- the node will not begin with uploading an inventory on the initial
"bootstrapping" run, only after it received its initial promises from the
policy server.
- the bootstrapping code will still have a hook to define the Hostname,
but it can be overwritten by later customer hooks?
-> Q: will the Hooks be executed by their names sorted lexically?
-> Q: will all nodes receive the same additional hook plugins?
-> Q: will the inventory contain the json output 1:1 or will it be
converted to xml? // can one only use key: value pairs or more complex data
structures to extend the inventory?
(NB: Using the collected "facts" as variables is cool, but please don't
forget to also add the centrally managed node properties as vars to the
node...)
Thanks
Janos
Francois Armand <francois.armand at normation.com> ezt írta (időpont: 2016.
máj. 13., P 16:59):
> On 02/05/2016 22:13, Mattyasovszky János wrote:
>
> Hi
>
>
> *Answers Inline. *
> Regards,
> Matya
>
> Jonathan Clarke <jonathan.clarke at normation.com> ezt írta (időpont: 2016.
> máj. 2., H, 18:29):
>
>> Hi,
>>
>>
>> On 04/13/2016 11:48 AM, Francois wrote:
>>
>> On 13/04/2016 09:49, Nicolas Charles wrote:
>>
>> Hello Francois,
>>
>> Thank you for the nice sum-up. My answers in the text
>>
>>
>> +1
>>
>>
>> Le 07/04/2016 15:33, Francois Armand a écrit :
>>
>> On 07/04/2016 15:08, Vincent Membré wrote:
>>
>> Le 31/03/2016 12:37, Francois Armand a écrit :
>>
>> Hello,
>>
>> So, here goes for a summary of
>> http://www.rudder-project.org/redmine/issues/8022, "Node's
>> FQDN-Resolution is sometimes invalid" and related tickets.
>> The problem cover up several sub-cases, which need to be addressed
>> systematically to achieve some result.
>> They are:
>>
>> - 1/ the node FQDN is used for identifying a node, and then manage
>> authentication and authorization to access its promises. If Rudder server,
>> CFEngine promise server, and the node don't agree on it, the node can't get
>> its promises. This is a hard problem because:
>> - FQDN is fragile. It needs a perfectly up to date and shared DNS
>> environment. But "it's always a DNS problem", what gives an idea.
>> - FQDN tools are notoriously broken, and don't always agree about
>> what is the FQDN of a host
>> - even in a perfectly working and up-to-date DNS env, there may
>> have voluntary decision not to have the same FQDN from the host and the
>> server, as explained in #8022 ticket.
>> - 2/ if the node FQDN is not correct in the first sent inventory,
>> the node configuration is delayed by one day, because from the node point
>> of view, the inventory was correctly sent and so the standard frequency for
>> sending inventories is applied.
>>
>>
>> The long-term solution for 1/ is to use something else that FQDN to
>> identify the node - we have for example an UUID for that. The problem here
>> is that it is a hard limitation of the protocol used by cf-serverd. So to
>> use another identification scheme (and why not our own
>> authentication/authorization), we need to either patch cf-serverd or use a
>> different client-server protocol for promises transfer. Both solutions are
>> open, but are out of scope.
>>
>>
>> Actually, it is already possible to do authorizations in cf-serverd based
>> on the key hash of connecting agents:
>> https://docs.cfengine.com/docs/3.7/reference-promise-types-access.html#admit_keys.
>> This has been possible since CFEngine 3.6 (so works on Rudder 2.11+ agents).
>>
>> We could pretty easily implement this, since we already have an inventory
>> with nodes CFEngine keys in. It could be opt-in as a start, while we ensure
>> it's reliable (as in "more reliable that hostnames"). WDYT?
>>
>>
> *This is the most welcome!*
>
> *The only thing one should not forget about is that apache is still having
> its allowed IP Subnets, which are also not making one's life too easy if
> you have a ton of subnets your clients connect from.*
>
> *Please also do not forget
> about https://www.rudder-project.org/redmine/issues/7893
> <https://www.rudder-project.org/redmine/issues/7893>
> <https://www.rudder-project.org/redmine/issues/7893>, which is about
> extending the Rudder API with the possibility to edit the allowed subnets
> for the policy servers, which one currently has to do either in the WebGUI
> or over some direct-brain-surgery out-of-band LDAP editing...*
>
>
>
>> Meanwhile, we can address at best 1/ and 2/
>>
>> *For 1/, *we need to prevent as much as we can to give bad FQDN and in
>> all case, give the user a possibility to hook what he knows should be the
>> correct value.
>> *- Prevent more bad FQDN:*
>> => update perl version: http://www.rudder-project.org/redmine/issues/8123
>> I didn't find anything else on that subject, compared to what we are
>> doing now.
>>
>> *- Let the user hook the correct value: *
>> => http://www.rudder-project.org/redmine/issues/8022#note-20
>> Here, we still need to specify the path convention for command and file
>> to look for the correct FDQN.
>>
>> I have some remarks/questions here:
>>
>> * If I understand correctly the process will be: Inventory is ran on the
>> Node, then inventory is modified to add data from commands / file ? or will
>> this be used when runnig fusionInventory ?
>>
>>
>> The idea is to patch fusion inventory plugin for Rudder to use that
>> logic.
>>
>> There are already a way to extend inventories:
>> http://fusioninventory.org/documentation/agent/additional_content.html
>> I think it will be much easier and future proof to use the
>> FusionInventory way rather than reimplementing it ourselves
>>
>> We could use this principle to create our own task that would read data
>> or run external script - but maybe that's what you were implying ?
>>
>>
>> No, I meant that we should evolve the logic to fill <RUDDER><HOSTNAME>,
>> contribute the evolution to rudder plugin, and patch our own until then.
>> I'm not about using the what you linked to for that, it's really not an
>> extension, but the actual "business" logic we want to implement for Rudder
>> <HOSTNAME>.
>>
>> On the other hand, what you propose seems really nice for 4670, I updated
>> the ticket accordingly.
>>
>>
> *#4670 is really something very useful, as this would allow to merge other
> inventory-collecting systems into the one rudder collects with
> fusioninventory, which is currently not providing any interface to get any
> user-defined data, which could be collected on the server (like Firmware
> version of different cards, SAP SID etc.)*
>
>
>> * In Which entry will they be stored ? RUDDER/HOSTAME? another one
>>
>>
>> Yes, RUDDER/HOSTNAME
>>
>> * Do we want to define Hostname only ? Would it not be better if the
>> solution was much more adaptable and can modify any entry from the
>> inventory ?
>>
>>
>> It's an other ticket, #4670, linked in #8022. HOSTNAME (FQDN, really) is
>> different from the general use case (at least adding information into
>> inventory) because of the special importance of FQDN in CFengine
>> server/agent identification protocol. So it is kind of ok to see #8022 as a
>> bug in the existing versions, and #4670 (or an extension of it) as a new
>> feature going to next versions.
>>
>> Yes, let start with HOSTNAME in released version
>>
>>
>>
>>
>> About the path, I suggest we should put them under
>> /var/rudder/inventories :
>>
>>
>> - If we modify only hostname (RUDDER/HOSTNAME):
>> - /var/rudder/inventories/hostname-command: Command to execute the
>> get the correct hostname
>> - /var/rudder/inventories/hostname-file: File containing the path
>> the file containing the correct hostname
>>
>>
>> No specific feeling on that... Any idea, other ?
>>
>> I'm not sure it should be in /var - if these are scripts to run to get
>> data, or data user should put, I feel they would live in other folder.
>>
>>
>> It is pretty unusual to configure anything in /var. The right place for
>> configs is /etc. (yes, policy_server.dat is a nasty exception to this
>> rule). How about /opt/rudder/etc/hooks/inventory.d/ ? This would be a good
>> start to having extensible inventory.
>>
>> (and yes, it would be desirable to have configs in /etc, without the
>> /opt/rudder prefix, but we're talking about a bug fix in existing versions
>> here, so that it clearly out of scope)
>>
>>
>>
> *Well, you already have stuff in /opt/rudder/etc, so adding more stuff
> there would actually make more sense then opening a third location (well,
> there is actually /etc/sysconfig/rudder-apache, which is already a config
> below /etc, so you have already stuff laying in there...*
>
>
>> Plus, if we want user to preseed the system with these scripts/data,
>> before installing agent, /var/rudder will not exist
>>
>>
>> This is irrelevant, I think. If you need to install a file in a
>> directory, it's pretty easy to create - in fact, just good practice.
>>
>>
>> *For 2/, *we need to prevent the sending of inventory that will be
>> rejected by the server for sure.
>> => http://www.rudder-project.org/redmine/issues/8127
>>
>>
>> FYI, Benoît is working on this this week. We will implement simple checks
>> on the most important fields using a Perl script (since the inventory is
>> already run in Perl, we know it's available).
>>
>> Jon
>> _______________________________________________
>>
>
>
> So, we sum up everything we have on the subject, and we have (finally)
> something coherent with a consistent long term vision. So let's start by
> that, it will make other decision clearer.
>
> In the long term (i.e, not in a bug correction releaes, but can and should
> go in next major):
>
> - nodes don't get a set of complexe initial promise. They have only a
> really small one, which mostly like say "go searching your initial promises
> from the policy server at that place"
> - this system can be use to also download a set of initial
> configuration files, among which can be a set of inventory hooks,
> - the inventory processing is done in three steps :
> - 1/ produce an inventory file from fusion,
> - 2/ run hooks from */var/rudder/hooks/inventory.d/**
> - each hook produce a json file, merged with the previous json
> - values are overloaded is a newer hooks define the same key
> - the resulting JSON file is check for the existence os a
> "hostname" key. If it exists, it is extracted (removed) and its value is
> used to overwrite <RUDDER><HOSTNAME>
> - the resulting final JSON file is added in the inventory file,
> somewhere in <RUDDER> - it could make sense to send it apport inventory.xml
> (like the key part) but the gain, appart for not yielling because there is
> JSON in XML, does not seems obvious
> - REMARQUE: it would very much make sense that the resulting
> JSON file is left on the node, and can be used as a datasource for node
> properties in directives etc. Or it will make sense that along with promise
> download, the node download such a JSON file, sourced for data for the
> remaining of the run.
> - 3/ the final inventory final is sanity check so that is
> necessary data are missing (empty hostname, etc), it is not sent to the
> policy server and an error is logged.
> - all the content from the json file is added as properties of the
> node (and will be searchable, usable to define group, usable as variable in
> directives, etc).
>
>
> This solve nicelly all the bug related to 8022:
>
>
> - for 8022 itself, it means :
> - that we can had a defaults"hostname" hook, with the logic "if
> RUDDER_FQDN is defined non empty"
> - users are free to add other hooks to overide it ("if RUDDER_FQDN
> key is defined in file ...., use that" etc)
> - and that can be done since Rudder 3.1 without problem, because it
> won't change current behaviour
> - for 8127 (check inventory before sending them), it changes nothing :
> it's step 3
> - we also checked 8123 (update perl version), and the amount of work
> to get there is huge and risky. So 8123 will go in next major, and we are
> going to try to solve 8283 ASAP
> - finally, most of 4670 will be done: all the agent logic be there,
> and rudder will be extended to actually use the data in next major.
>
>
> Hope everhting is clear !
> --
> ------------------------------
> * François ARMAND*
> *Co-founder & CTO*
> Normation <http://www.normation.com>
> ------------------------------
> *87 rue de Turbigo, 75003 Paris, France*
> Telephone: +33 (0)1 83 62 99 23
> Mobile: +33 (0)6 63 37 60 55
> ------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.rudder-project.org/pipermail/rudder-dev/attachments/20160513/47a1b881/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logo-square.gif
Type: image/gif
Size: 1036 bytes
Desc: not available
URL: <http://www.rudder-project.org/pipermail/rudder-dev/attachments/20160513/47a1b881/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logo-square.gif
Type: image/gif
Size: 1036 bytes
Desc: not available
URL: <http://www.rudder-project.org/pipermail/rudder-dev/attachments/20160513/47a1b881/attachment-0003.gif>
More information about the rudder-dev
mailing list