Project

General

Profile

User story #9707

Add a Relay API: share files between nodes, launch run on remote run behind relay

Added by Alexis MOUSSET 12 months ago. Updated 10 months ago.

Status:
Released
Priority:
N/A
Assignee:
-
Category:
API
Target version:
Target version (plugin):
Suggestion strength:
User visibility:
Effort required:
Pull Request:

Description

One of Rudder 4.1 new features will be relay APIs. This is the first attempt to describe it.

There are currently 2 API entries we want to add: remote-run and share-files.
They will both be under /rudder/relay-api itself under https://<server>:<port>/ like the current api is.
/rudder is the common root for all rudder service
/relay-api is different from existing api to avoid conflicts with them when it will be installed on the server

1. Remote-run:
The goal is to make a given relay call "rudder remote run" on one of its attached node
The API will be under /rudder/relay-api/remote-run
GET remote-run/node/<node-uuid>
GET remote-run/all
GET remote-run/nodes

Parameters:
- output = keep / discard : to keep the output of the remote-run call or discard its content
- async = yes / no : yes to ignore the return code of the call and return immediately, no to wait until the end of the call and get the return code
- classes = XXX : list of cfengine classes to set during the remote call
- nodes = uuid,... : list of uuid to call in the "/nodes" case

Behavior:
- Loop on all nodes
- Find its hostname from its uuid in a matching file created by promise generation on the server
- The call is descending, so we don't care about host that do not exist
- The call is descending, so we will only accept calls from the policy server
- Call rudder remote
- prefix the remote output lines with <uuid>: to make sure the caller can parse output during async call on multiple nodes
- surround the output with json format lines and include return code, duration and stderr (-> we should escape the output for use within a json string)

2. Share Files

The goal is to share files between an agent and another one via their policy server.
The API will be under /rudder/relay-api/shared-files
POST shared-files/node/<target_uuid>/<file_id> (share a file content with the target uuid using a given file id)
HEAD shared-files/node/<target_uuid>/<file_id> (ask if a file with the given file id is already shared with the given target id)

Parameters:
- source-uuid: the uuid of the node sharing a file with the target uuid
- signature: the file's signature in case of a POST, to authenticate the source node
- hash: the file's hash in case of a HEAD to know is the version already present matches
POST Behavior:
- if the target is known by the local relay
> validate the signature
> store the file in /var/rudder/shared-files-nodes/[..<relay-uuid>..]/<target_uuid>/<source_uuid>/<file_id>
> store the metadata (including, date, hash and signature) in <the same path>.medata
- if the target in not known and we are not the root server
> store the file in a temporary directory
> try to send the file to the relay server
> do nt remove the file and try again as long as there is a fatal error (code >= 500, network error)
- if the target in not known and we are the root server
> ignore the file and return 404

HEAD behavior:
- If the file exists in /var/rudder/shared-files-nodes... with the same hash, return 200
- If it doesn't, return 404

3. Using this API from ncf
We will create 2 new generic methods in ncf:
- sharedfile_to_node(target_uuid, file_id, file_path, ttl) where ttl is infinite by default
- sharedfile_from_node(source_uuid, file_id, file_path)

The first one will call HEAD on the shared-files API and if is gets a 404, call POST to send the content.

The second one will just download the file using regular cfengine protocol.


Subtasks

Architecture #9708: Create a relay API for shared-filesReleasedAlexis MOUSSET

User story #9709: Add the path of files shared between nodes to cf-serverd configurationReleasedAlexis MOUSSET

User story #9710: Generate a file in relays promises that contain the map of nodes and relays below itReleasedAlexis MOUSSET

User story #9715: Install rudder-relay on rudder root policy serverRejected

Architecture #9729: New system variablesReleasedVincent MEMBRÉ

Bug #9847: StackOverflowError on policy generationReleasedAlexis MOUSSET

Architecture #9730: Package the new relay-apiReleasedAlexis MOUSSET

User story #9732: Create a relay API for remote runReleasedBenoît PECCATTE

Bug #9825: Wrong path in relay-api packagingReleasedBenoît PECCATTE

Bug #9828: The relay package must depend on python dev package to be able to buildReleasedAlexis MOUSSET

Bug #9830: The relay package doesn build on 4.1ReleasedAlexis MOUSSET

Bug #9838: Old debian call libpython-dev python-devReleasedAlexis MOUSSET

Bug #9840: Old debian call libpython-dev python-devReleasedAlexis MOUSSET

Bug #9852: the relay tries to packages configuration in /etc/httpdReleasedNicolas CHARLES

Bug #9889: Remove common conf between relay and webapp from webapp packageReleasedBenoît PECCATTE

Bug #9907: Wrong file name in relay packageReleasedBenoît PECCATTE

Bug #9908: Apache conf of webapps were movedReleasedBenoît PECCATTE

Bug #9909: Wrong filename in deb packagingReleasedBenoît PECCATTE

Bug #9911: Wrong filename in rpm packagingReleasedBenoît PECCATTE

Bug #9912: Wrong filename in apache conf packagingReleasedBenoît PECCATTE

Bug #9915: Dependency issue in 4.1 relay packageReleasedBenoît PECCATTE

User story #9890: Remove relay specific conf from rudder apache config filesReleasedBenoît PECCATTE

Bug #9895: relay package fail to build on sles 11ReleasedAlexis MOUSSET

Bug #9896: relay package fail to build on sles 11ReleasedAlexis MOUSSET

Bug #9902: relay package fail to build on sles 11ReleasedAlexis MOUSSET

User story #9904: Add apache configuration for relay-apiReleasedBenoît PECCATTE

User story #9906: Generate apache conf authorizing connection from policy serverReleasedBenoît PECCATTE

Bug #9923: flask doesn't build on sles11ReleasedAlexis MOUSSET

Bug #9925: Broken postinst in relay packageReleasedBenoît PECCATTE

Bug #9941: Cannot install rudder-server-relay on CentOSReleasedBenoît PECCATTE

Bug #9951: rpm build fail on centos7ReleasedAlexis MOUSSET

Bug #9954: rpm build fail on centos7ReleasedAlexis MOUSSET

Bug #9959: rpm build fail on centos7ReleasedAlexis MOUSSET

Bug #9952: Missing relay-api.wsgi on relaysRejected

Bug #9956: Relay apache conf does not allow accessing the apiReleasedBenoît PECCATTE

User story #9957: Copy nodeslist.json file outside of inputs dirReleasedBenoît PECCATTE

User story #9961: Use nodeslist from /opt/rudder/etcReleasedBenoît PECCATTE

User story #9965: Set the right permissions on nodeslist.jsonReleasedBenoît PECCATTE

User story #9958: Nodeslist file is empty on a root server with one nodeReleasedNicolas CHARLES

User story #9968: Add a CRON task to call cleanup.shReleasedBenoît PECCATTE

Bug #9969: cron.d is not created in .specReleasedBenoît PECCATTE

Bug #9970: old a2ensite does not append .conf automaticallyReleasedBenoît PECCATTE

Bug #9971: Missing permission for wsgiReleasedBenoît PECCATTE

Bug #9973: Missing fix for certificates in relay packageReleasedBenoît PECCATTE

Bug #9975: We sometimes write a wrong require statement in apache 2.4 confReleasedBenoît PECCATTE

Bug #9977: Wrong place for wsgi conf in deb packagesReleasedBenoît PECCATTE

User story #9983: The rudder user should be in the rudder groupReleasedBenoît PECCATTE

Bug #9985: Error when executiong remote runReleasedBenoît PECCATTE

Bug #9986: Relay api conf is not enabled after installReleasedBenoît PECCATTE

Bug #9988: file /opt/rudder/share/relay-api/cleanup.sh is not executable, lots of cron mail about itReleasedAlexis MOUSSET

Bug #9989: Relay api runs with www-data user instead of rudderReleasedAlexis MOUSSET

Bug #9990: Broken remote run in 4.1ReleasedBenoît PECCATTE

Bug #9991: Syntax error in parent ticketReleasedBenoît PECCATTE

User story #9992: Use sudo to execute rudder remote run in the APIReleasedBenoît PECCATTE

Bug #10002: Broken RPM packaging of sudo confReleasedBenoît PECCATTE


Related issues

Related to ncf - User story #9711: Add generic methods to share files between nodes using rudder relay API Released
Related to Rudder - Bug #9913: Add SELinux config to relay package Released
Related to Rudder - User story #9914: Apache configuration files status is inconsistent in packages New

History

#1 Updated by Alexis MOUSSET 12 months ago

  • Tracker changed from Bug to User story
  • Description updated (diff)
  • Category set to API

#2 Updated by Alexis MOUSSET 12 months ago

  • Description updated (diff)

#3 Updated by Alexis MOUSSET 12 months ago

  • Related to User story #9711: Add generic methods to share files between nodes using rudder relay API added

#4 Updated by Alexis MOUSSET 11 months ago

  • Related to Bug #9913: Add SELinux config to relay package added

#5 Updated by Alexis MOUSSET 11 months ago

  • Related to User story #9914: Apache configuration files status is inconsistent in packages added

#6 Updated by Vincent MEMBRÉ 10 months ago

  • Target version changed from 4.1.0~beta1 to 4.1.0~beta2

#7 Updated by Vincent MEMBRÉ 10 months ago

  • Target version changed from 4.1.0~beta2 to 4.1.0~beta1

#8 Updated by Vincent MEMBRÉ 10 months ago

  • Status changed from New to Pending technical review

#9 Updated by Vincent MEMBRÉ 10 months ago

  • Status changed from Pending technical review to 12

#10 Updated by Vincent MEMBRÉ 10 months ago

  • Status changed from 12 to Pending release

#11 Updated by Vincent MEMBRÉ 10 months ago

  • Subject changed from Add an HTTP API on the relays to Add a Relay API: share files between nodes, launch run on remote run behind relay

#12 Updated by Vincent MEMBRÉ 10 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 4.1.0 which was released today.

Also available in: Atom PDF