[rudder-dev] Relay API
Benoit Peccatte
benoit.peccatte at normation.com
Sun Nov 6 14:15:10 CET 2016
Le 05/11/2016 à 22:30, Janos Mattyasovszky a écrit :
> Hi @all,
>
> I find the idea very mind-stimulating, I cannot even think about the
> possibilities this will enable ;-]
> I had some question / remarks:
>>
>> 1. Remote-run:
>>
> *
> As far I have understood, this remote-run api will be available on
> the relays themselves but only accessible from their policy
> server, right? The root server will then provide the capability to
> run the agent on the given Node (by UUID) by propagating the task
> to the relay the node(s) reside on and "proxy" the action?
>
Yes.
> *
> Will it behave like the regular api-system with
> one-token-all-access, or is there the possibility to somehow
> restrict actions, like maybe assign a token to a group and/or
> limit the cfengine class to be set? This would allow to delegate
> defined actions to "external" parties, who should not directly
> have access to the Rudder UI, but like they are responsible for a
> group of nodes, and I can delegate the task of patching the system
> and reboot it afterwards by giving them an API and a Key.
>
The rudder server API will be token based like all other ones, which
means it will have the same limits.
Having better right management on them is a goal that will be done in
another task.
> *
> On the "async"-kind of run: have you thought about creating a
> "Task-ID", so when you trigger the action, get a token for that
> run, and then can poll the action with some delay for it's outcome
> (return code) and the output? This would make longer-running tasks
> (like patching systems) possible without having to think about
> http timeouts and similar stuff...
>
It comes with some new problems too. Where, for how long and for who do
we keep the output ?
Timeout are not really a problem, you can easily have long running http
connections, it is a matter of proper configuration.
I understand that can be a questioning for people having a proxy with
timeout in them but i don't expect agent runs to last longer than a http
timeout.
>> 2. Share Files
>>
> Some suggestions:
>
> *
> A good idea is to maybe establish the ability to limit / turn off
> this behavior on given relays, like in a DMZ, to limit the
> possible attach vectors on intrusions.
>
That is a good idea, how would you expect it to be configured ?
A per relay setting like there is a per relay allowed network ?
> Some questions I still have, I could not completely follow the logic
> of sharing Node-Alice sharing a file with Node-Bob:
>
> * Does the root server propagate the forwarded files to downstream
> relays ? If example the topology is N1 ==> R1 ==> Root <== R2 <==
> N2, and N1 wants to share with N2, then R1 forwards to root, but
> root then needs to give R2 the file to make if available for N2...
>
Indeed, and it works because the root server know how to make a file
available for N2. Like withe the share directory, the structure is a
tree with a directoryfor N2 below the directory for R2. So R2 will
download its shared-files-nodes directory as if it was for itself and
put it in the same place where N2 will download it.
> *
> The signature of the File is there to authenticate the node? Is
> this done by signing the file with the private key of the node to
> generate the hash?
>
Yes
> *
> Where does the "file_id" come from? Is that the hash of the file,
> or something Node-A determines to be used as ID?
>
It is a free identifier provided by the user of ncf. The goal is to make
it easy to share data just by knowing a common identifier on both sides.
So you share with "my_own_id" on node A and you download "my_own_id" on
node B.
> * What happens if the Server return 404 for an uploaded file? Best
> would be if then the relay would discard that file from being
> retried, as otherwise you'd be hammering with uploads until
> someone removes that by hand.
>
Yes. The relay will only retry if there is a network error or a server
error (5xx).
> * Will there be a DELETE behavior to delete files? Would really make
> sense. Who should be able to delete those files? The sender, or
> also the receiver?
>
There is not yet, but there is a TTL argument to the sharedfile_to_node
to make sure a file doesn't stay here forever.
> * Do you plan to implement also anything to make these uploads /
> shared files visible in the GUI? Possibly also with a Delete
> button for the files? That could be resolved with remote agent
> runs on the relay by setting a delete_file_<hash> class that would
> result in that particular file being removed from the relay. It
> only gets tricky when you have chained up multiple relays :-)
>
The plan is currently to enable communication between nodes, so we take
th shortest path and we don't push up to the server if we don't need to.
But it will be possible possible one day to transmit info to the server
if it seems useful.
> So much for the first round of thinking, later things might follow.
>
> Have a nice sunday,
> Matya
>
>> -------- Original Message --------
>> Subject: [rudder-dev] Relay API
>> Local Time: 4. November 2016 5:38 PM
>> UTC Time: 4. November 2016 16:38
>> From: benoit.peccatte at normation.com
>> To: rudder-dev at lists.rudder-project.org
>> <rudder-dev at lists.rudder-project.org>
>>
>>
>> Hello,
>>
>> One of Rudder 4.1 new features will be relay APIs. This is the first
>> attempt to describe it.
>>
>>
>> There are currently 2 API entries we want to add: remote-run and
>> share-files.
>> They will both be under /rudder/relay-api itself under
>> https://<server>:<port>/ like the current api is.
>> /rudder is the common root for all rudder service
>> /relay-api is different from existing api to avoid conflicts with
>> them when it will be installed on the server
>>
>>
>>
>> 1. Remote-run:
>>
>> The goal is to make a given relay call "rudder remote run" on one of
>> its attached node
>> The API will be under /rudder/relay-api/remote-run
>> GET remote-run/node/<node-uuid>
>> GET remote-run/all
>> GET remote-run/nodes
>>
>> Parameters:
>> - output = keep / discard : to keep the output of the remote-run call
>> or discard its content
>> - async = yes / no : yes to ignore the return code of the call and
>> return immediately, no to wait until the end of the call and get the
>> return code
>> - classes = XXX : list of cfengine classes to set during the remote call
>> - nodes = uuid,... : list of uuid to call in the "/nodes" case
>>
>> Behavior:
>> - Loop on all nodes
>> - Find its hostname from its uuid in a matching file created by
>> promise generation on the server
>> - The call is descending, so we don't care about host that do not exist
>> - The call is descending, so we will only accept calls from the
>> policy server
>> - Call rudder remote
>> - prefix the remote output lines with <uuid>: to make sure the caller
>> can parse output during async call on multiple nodes
>> - surround the output with json format lines and include return code,
>> duration and stderr (-> we should escape the output for use within a
>> json string)
>>
>>
>>
>> 2. Share Files
>>
>>
>> The goal is to share files between an agent and another one via their
>> policy server.
>> The API will be under /rudder/relay-api/shared-files
>> POST shared-files/node/<target_uuid>/<file_id> (share a file
>> content with the target uuid using a given file id)
>> HEAD shared-files/node/<target_uuid>/<file_id> (ask if a file with
>> the given file id is already shared with the given target id)
>>
>>
>>
>> Parameters:
>> - source-uuid: the uuid of the node sharing a file with the target uuid
>> - signature: the file's signature in case of a POST, to authenticate
>> the source node
>> - hash: the file's hash in case of a HEAD to know is the version
>> already present matches
>>
>>
>> POST Behavior:
>> - if the target is known by the local relay
>> -> validate the signature
>> -> store the file in
>> /var/rudder/shared-files-nodes/[..<relay-uuid>..]/<target_uuid>/<source_uuid>/<file_id>
>> -> store the metadata (including, date, hash and signature) in <the
>> same path>.medata
>> - if the target in not known and we are not the root server
>> -> store the file in a temporary directory
>> -> try to send the file to the relay server
>> -> do nt remove the file and try again as long as there is a fatal
>> error (code >= 500, network error)
>> - if the target in not known and we are the root server
>> -> ignore the file and return 404
>>
>> HEAD behavior:
>> - If the file exists in /var/rudder/shared-files-nodes... with the
>> same hash, return 200
>> - If it doesn't, return 404
>>
>> 3. Using this API from ncf
>> We will create 2 new generic methods in ncf:
>> - sharedfile_to_node(target_uuid, file_id, file_path, ttl) where
>> ttl is infinite by default
>> - sharedfile_from_node(source_uuid, file_id, file_path)
>>
>> The first one will call HEAD on the shared-files API and if is gets a
>> 404, call POST to send the content.
>>
>> The second one will just download the file using regular cfengine
>> protocol.
>>
>>
>>
>> That's all folks !
>> Any comment ?
>>
>> --
>>
>> ------------------------------------------------------------------------
>>
>> Logo Normation Benoît Peccatte
>> Architecte
>> Normation <http://www.normation.com>
>> ------------------------------------------------------------------------
>>
>> 87, Rue de Turbigo, 75003 Paris, France
>> Phone:
>> +33 (0)1 85 08 48 96
>> ------------------------------------------------------------------------
>>
>
>
>
> _______________________________________________
> rudder-dev mailing list
> rudder-dev at lists.rudder-project.org
> http://www.rudder-project.org/mailman/listinfo/rudder-dev
--
------------------------------------------------------------------------
Logo Normation Benoît Peccatte
Architecte
Normation <http://www.normation.com>
------------------------------------------------------------------------
87, Rue de Turbigo, 75003 Paris, France
Phone: +33 (0)1 85 08 48 96
------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.rudder-project.org/pipermail/rudder-dev/attachments/20161106/396b0b0f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logo-square3.gif
Type: image/gif
Size: 1036 bytes
Desc: not available
URL: <http://www.rudder-project.org/pipermail/rudder-dev/attachments/20161106/396b0b0f/attachment-0001.gif>
More information about the rudder-dev
mailing list