[rudder-users] Compliance reports

Fri Mar 30 12:50:37 CEST 2018

Francois,

I'll absolutely let you know how I get on. And if I get anything working
well, I will publish an article on my blog.

Rob

--
*Rob Pomeroy*, CISSP <https://www.isc2.org/cissp>, Solicitor
<http://solicitors.lawsociety.org.uk/person/250541/robert-john-pomeroy> *| *
rob at pomeroy.me
My novel <https://www.smashwords.com/books/view/78386> | LinkedIn
<http://www.linkedin.com/in/robpomeroy> | Personal blog <http://pomeroy.me/>
 | Geek & Dummy <http://geekanddummy.com/>

On 30 March 2018 at 10:58, Francois Armand <francois.armand at normation.com>
wrote:

> Hey, we would love to know more about your integration with check_mk, and
> why not help you build a Rudder plugin for that. Nowadays, we are trying to
> expend the number of integration plugins, and along with CMDB like iTop,
> monitoring software are the main proponent (we have a working plugin for
> Centreon for ex).
> In particular, we would love to know if you are missing things in Rudder
> integration api to reach your goal - and a compliance hook will most
> likelly be one that list (see https://www.rudder-project.
> org/redmine/issues/11221 or perhaps even a full fledge event notification
> system).
>
> Cheers,
>
> On 30/03/2018 11:44, Rob Pomeroy wrote:
>
> Hi Francois,
>
> Thank you for your very complete answers! Yes, I see what you mean about
> the semantic/race condition on compliance state.
>
> I'll think about how best to express my change request for an improved
> search interface.
>
> In the meantime I need to stop being lazy and just take the plunge with
> the API! My ultimate aim is to bring Rudder status into Check_MK, so I have
> a single pane of glass for monitoring and alerting. (We then integrate
> Check_MK with PagerDuty so we can have the delight of being woken up in the
> middle of the night when a disk fills.) I may also send events into Graylog.
>
> Merci beaucoup!
>
> Rob
>
>
> --
> *Rob Pomeroy*, CISSP <https://www.isc2.org/cissp>, Solicitor
> <http://solicitors.lawsociety.org.uk/person/250541/robert-john-pomeroy>
> *| *rob at pomeroy.me
> My novel <https://www.smashwords.com/books/view/78386> | LinkedIn
> <http://www.linkedin.com/in/robpomeroy> | Personal blog
> <http://pomeroy.me/> | Geek & Dummy <http://geekanddummy.com/>
>
> On 30 March 2018 at 10:26, Francois Armand <francois.armand at normation.com>
> wrote:
>
>> On 29/03/2018 15:59, Rob Pomeroy wrote:
>>
>> Hi,
>>
>> This probably should be raised as a feature request/issue - but I wanted
>> the views of those on this list first? (Also when I looked in the issue
>> log, there are many pages of issues there, so no point adding another one
>> if it wouldn't be a priority.) :-)
>>
>> I'd really like to be able to search by the compliant state of nodes.
>> It'd be handy to be able to pull out just those nodes that require
>> attention. I have dozens and dozens of managed nodes now and it's getting
>> harder to find those that need to be fixed.
>>
>> So two things I'd be interested in:
>>
>>
>>    1. A search/dynamic group based on whether a not a node is in a
>>    compliant state.
>>    2. A way of finding nodes based on a period between today's date and
>>    the last inventory received. You can of course search based on a fixed
>>    date, but I'm not sure what, if any, functions can be used in the search
>>    box. Something like last inventory date > now() - 1 day.
>>
>>
>> I couldn't find anything like this in the online documentation or the
>> interface. But perhaps I've overlooked something that already exists?
>>
>> Cheers,
>>
>> Rob
>>
>> -
>>
>>
>> Hello Rob,
>>
>> Firstly, just to be sure you didn't miss that: you can sort nodes in the
>> "all node" list by compliance (click on the "Compliance" column header to
>> do so), so that help put the one with red on top.
>>
>> Now, for your requests:
>>
>> 1/ This one is interesting, but it's hard to get right in the "group"
>> workflow. First, there is the semantic problem about what is the
>> "compliance" number you want to match on. Global compliance ? One state in
>> particular ? It can be sorted out, but with the current group search UI,
>> our tests make it cumbersome.
>> But then, the real problem is in the feedback loop and
>> hysteris/instabilities it may introduce in your system. Imagine you build
>> such a group and make a rule on them. Now, the rule depends on its outcome,
>> with a propagation travel time to take in account. Node may enter and get
>> out fast ("fast" as in "not many tickes", considering a "tick" at rudder
>> scale to be 5min). So that will multiply policy generation, but the time
>> they reach the nodes, perhaps the dyn group already change again...
>>
>> So. To sum up, it is a not obvious problem to sort out cleanly, and we
>> are not yet at the point where we can go there.
>>
>> So, in the meantime, we are solving the much more easier problem of
>> getting nice custom reports on compliance (read-only, no feedback loop,
>> nice graphs :). It will be through a "compliance report" plugin that allows
>> to build custom reports with history of compliance evolution on selected
>> nodes/groups/rules/etc. It will be ready in the very short term (coming
>> weeks) and part of a commercial offer for Rudder. If you are interested to
>> learn more, I can put you in touch with the relevant person.
>>
>> And in a hacky-already-work way of doing things, you can use API + jq (
>> https://stedolan.github.io/jq) to do what you want. For example, to get
>> all nodes with global compliance < 80%, you can do:
>>
>> % curl -k -H "X-API-Token: xxxx" -H "Content-Type: application/json" -X
>> GET 'https://....rudder/api/latest/compliance/nodes?level=1' |* jq
>> '.data.nodes | sort_by(.compliance) | map(select(.compliance < 80))' *
>> [
>> {
>>   "id": "717b63d1-01fe-4d4f-a7e5-cfb7c0d47b4f",
>>   "name": "debian-8-64.labo.normation.com",
>>   "compliance": 0,
>>   "mode": "full-compliance",
>>   "complianceDetails": {
>>     "unexpectedMissingComponent": 49.14,
>>     "unexpectedUnknownComponent": 50.86
>>   }
>> }
>> {
>>   "id": "0c846655-cb06-486f-ace4-eaeb11372097",
>>   "name": "centos-7-64.labo.normation.com",
>>   "compliance": 69.57,
>>   "mode": "full-compliance",
>>   "complianceDetails": {
>>     "successAlreadyOK": 30.43,
>>     "successNotApplicable": 34.78,
>>     "error": 30.43,
>>     "successRepaired": 4.35
>>   }
>> }
>> ...
>> ]
>>
>>
>> 2/ There is no particular function in the search box appart from "is:"
>> (for rule, node, group, parameter, directive, rule) and "in:" (for
>> attributes), as explained in the doc here: https://orchestrateur-4.labo.n
>> ormation.com/rudder-doc/search-nodes.html . Having a more complexe
>> search language for the quick search would be nice. Perhaps could you mind
>> opening an user story for that?
>>
>> In the meantime, you can again rely on APIs (yeah, that's our swiss army
>> knife ;). It will fullfill what I understand of your need, more or less.
>> More or less because medling with date format and date difference is a
>> nightmare in any language, put it's even worse in the middle of a jq
>> directive. So in place of "selected range date", I give you "sorted, and
>> select a subsection of the resulting array".
>>
>> This command will give you the 3 nodes with the oldest last inventory
>> date (you can choose any slice in the resulting array, of course):
>>
>> % curl -k -H "X-API-Token: dTxvl4eL8p3YqvwefVbaJLdy8DyEt7Vw" -H
>> "Content-Type: application/json" -X GET 'https://orchestrateur-4.labo.
>> normation.com/rudder/api/latest/nodes?include=minimal,lastInventoryDate'
>> | *jq '.data.nodes | sort_by(.lastInventoryDate) | .[0:3]'*
>>
>> [
>>   {
>>     "id": "fc846655-cb06-486f-ace4-eaeb11372097",
>>     "hostname": "sovma136",
>>     "status": "accepted",
>>     "lastInventoryDate": "2016-12-05 15:12"
>>   },
>>   {
>>     "id": "8b168194-c0b4-41ab-b2b5-9571a8906d59",
>>     "hostname": "debian-5-64.labo.normation.com",
>>     "status": "accepted",
>>     "lastInventoryDate": "2017-02-01 17:06"
>>   },
>>   {
>>     "id": "94b6d33d-a23e-46b9-b5f8-971751bebcbb",
>>     "hostname": "ubuntu-16-04-64",
>>     "status": "accepted",
>>     "lastInventoryDate": "2018-03-30 02:02"
>>   }
>> ]
>>
>>
>> And here come the link toward jq manual: https://stedolan.github.io/jq/
>> manual
>> jq is globally hard, but you can get a lots of things done with it.
>>
>> Hope it helps,
>>
>> --
>>
>>
>> ------------------------------
>> * François ARMAND*
>> *Co-founder & CTO*
>> Normation <http://www.normation.com>
>> ------------------------------
>> *87 rue de Turbigo, 75003 Paris, France*
>> Telephone: +33 (0)1 83 62 99 23 <+33%201%2083%2062%2099%2023>
>> Mobile: +33 (0)6 63 37 60 55 <+33%206%2063%2037%2060%2055>
>> ------------------------------
>>
>
>
> --
>
>
> ------------------------------
> * François ARMAND*
> *Co-founder & CTO*
> Normation <http://www.normation.com>
> ------------------------------
> *87 rue de Turbigo, 75003 Paris, France*
> Telephone: +33 (0)1 83 62 99 23 <+33%201%2083%2062%2099%2023>
> Mobile: +33 (0)6 63 37 60 55 <+33%206%2063%2037%2060%2055>
> ------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20180330/c3e073c2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logo-square.gif
Type: image/gif
Size: 1036 bytes
Desc: not available
URL: <http://www.rudder-project.org/pipermail/rudder-users/attachments/20180330/c3e073c2/attachment-0001.gif>