User story #2738
closedHandle big files payloads in the Enforce a file content Technique
Description
CFengine is unable to handle too big files payloads in the Enforce a file content Technique.
In Rudder 3.0, the limit is at 16ko and the error is at the promise validation time
The error message is:
input buffer overflow, can't enlarge buffer because scanner uses REJECT
Before, the limit is at 4ko ou 8ko, and for example, when trying to apply a backup-manager configuration using it, there is a deployment error, outputting this:
Expansion overflow constructing string. Increase CF_EXPANDSIZE macro. Tried to add " # Should we purge only archives built with BM_ARCHIVE_PREFIX export BM_ARCHIVE_STRICTPURGE="true" # You may want to nice the commands run for archive-creation # (Recommanded for desktop users.) # Choose a nice level from -20 (most favorable scheduling) to 19 (least favorable). export BM_ARCHIVE_NICE_LEVEL="10" # The backup method to use. # Available methods are: # - tarball # - tarball-incremental # - mysql # - svn # - pipe # - none # If you don't want to use any backup method (you don't want to # build archives) then choose "none" export BM_ARCHIVE_METHOD="tarball-incremental" ############################################################## # Encryption - because you cannot trust the place your # archives are ############################################################## # If you want to encrypt your archives locally, Backup Manager # can use GPG while building the archive (so the archive is never # written to the disk without being encrypted. # Note: this feature is only possible with the following archive types: # tar, tar.gz, tar.bz2 # Uncomment the following line if you want to enable encryption # available method: gpg export BM_ENCRYPTION_METHOD="gpg" # The encryption will be made using a GPG ID # Examples: # export BM_ENCRYPTION_RECIPIENT="" # export BM_ENCRYPTION_RECIPIENT="" export BM_ENCRYPTION_RECIPIENT="0C60B2BA" ############################################################## # Section "TARBALL" # - Backup method: tarball ############################################################# # Archive filename format # long : host-full-path-to-folder.tar.gz # short : parentfolder.tar.gz export BM_TARBALL_NAMEFORMAT="long" # Type of archives # Available types are: # tar, tar.gz, tar.bz2, tar.lz, dar, zip. # Make sure to satisfy the appropriate dependencies # (bzip2, dar, lzma, ...). export BM_TARBALL_FILETYPE="tar.bz2" # You can choose to build archives remotely over SSH. # You will then need to fill the BM_UPLOAD_SSH variables # (BM_UPLOAD_SSH_HOSTS, BM_UPLOAD_SSH_USER, BM_UPLOAD_SSH_KEY). # If this boolean is set to true, archive will be saved locally (in # BM_REPOSITORY_ROOT but will be built by the remote host). # Thus, BM_TARBALL_DIRECTORIES will be used to backup remote directories. # Those archive will be prefixed with the remote host name. export BM_TARBALL_OVER_SSH="false" # Do you want to dereference the files pointed by symlinks ? # enter true or false (true can lead to huge archives, be careful). export BM_TARBALL_DUMPSYMLINKS="false" # Targets to backup # You can use two different variables for defining the targets of # your backups, either a simple space-separated list (BM_TARBALL_DIRECTORIES) # or an array (BM_TARBALL_TARGETS[]). # Use the first one for simple path that doesn't contain spaces in their name. # Use the former if you want to specify paths to backups with spaces. # It's recommanded to use BM_TARBALL_TARGETS[] though. # Warning! You *must not* use both variables at the same time. # NOTE: The Debian package will only update BM_TARBALL_DIRECTORIES # Paths without spaces in their name: export BM_TARBALL_DIRECTORIES="/etc /home/* /root /var/log" # If one or more of the targets contain a space, use the array: # declare -a BM_TARBALL_TARGETS # BM_TARBALL_TARGETS[0]="/etc" # BM_TARBALL_TARGETS[1]="/boot" # export BM_TARBALL_TARGETS # Files to exclude when generating tarballs, you can put absolute # or relative paths, Bash wildcards are possible. export BM_TARBALL_BLACKLIST="/var/archives" # With the "dar" filetype, you can choose a maximum slice limit. export BM_TARBALL_SLICESIZE="1000M" # Extra options to append to the tarball generation # (take care to what you do; this will be silently added to the # command line.) export BM_TARBALL_EXTRA_OPTIONS="" ############################################################## # The tarball-incremental method uses the same keys as the # tarball method, plus two others. ######################################################## Fatal cfengine error: Can't expand varstring [2012-07-31 12:15:21] ERROR com.normation.rudder.services.policies.RudderCf3PromisesFileWriterServiceImpl - The generated promises at /var/rudder/share/713c52d2-78d8-4dbe-87be-21457496de32.new/rules/cfengine-community are invalid
I do think this is because CFengine is unable to handle such big variables unless compiled with a larger CF_EXPANDSIZE. But in that case, it is better to use a file template, I think.
Updated by Jonathan CLARKE over 11 years ago
- Target version changed from 2.4.0~beta4 to 2.4.0~rc1
Updated by Nicolas PERRON over 11 years ago
- Target version changed from 2.4.0~rc1 to 2.4.0~rc2
Updated by Nicolas PERRON over 11 years ago
- Target version changed from 2.4.0~rc2 to 2.4.0
Updated by Nicolas PERRON over 11 years ago
- Target version changed from 2.4.0 to 61
Updated by François ARMAND over 11 years ago
- Target version changed from 61 to 2.4.2
Updated by Nicolas PERRON over 11 years ago
- Target version changed from 2.4.2 to 2.4.3
Updated by Matthieu CERDA about 11 years ago
- Target version changed from 2.4.3 to 2.4.4
Updated by Jonathan CLARKE about 11 years ago
- Project changed from Rudder to 24
- Category deleted (
Techniques)
Updated by Nicolas CHARLES about 11 years ago
- Target version changed from 2.4.4 to 2.6.0~beta1
This has been fixed in later version of CFEngine; i can't really find out when, but with a CFEngine 3.4.1, it doesn't happen
I suspect that https://github.com/cfengine/core/pull/200/ is part of the solution
So by using 3.4.2, this should be automatically fixed
Updated by Nicolas PERRON about 11 years ago
- Target version changed from 2.6.0~beta1 to 2.6.0~rc1
Now that we use CFEngine 3.4.4 on Rudder 2.6, shouldn't we consider this bug as resolved (unless this is a 2.3, 2.4 or 2.5 bug) ?
Updated by Matthieu CERDA almost 11 years ago
- Target version changed from 2.6.0~rc1 to 2.6.0
Updated by Jonathan CLARKE almost 11 years ago
- Target version changed from 2.6.0 to 2.6.1
Updated by Nicolas PERRON almost 11 years ago
- Target version changed from 2.6.1 to 2.6.2
Updated by Nicolas PERRON almost 11 years ago
- Target version changed from 2.6.2 to 2.6.3
Updated by Nicolas PERRON over 10 years ago
- Target version changed from 2.6.3 to 2.6.4
Updated by Nicolas PERRON over 10 years ago
- Target version changed from 2.6.4 to 2.6.5
Updated by Nicolas PERRON over 10 years ago
- Target version changed from 2.6.5 to 2.6.6
Updated by Nicolas PERRON over 10 years ago
- Target version changed from 2.6.6 to 2.6.7
Updated by Nicolas PERRON over 10 years ago
- Target version changed from 2.6.7 to 2.6.8
Updated by Nicolas PERRON over 10 years ago
- Target version changed from 2.6.8 to 2.6.9
Updated by Nicolas PERRON over 10 years ago
- Target version changed from 2.6.9 to 2.6.10
Updated by Nicolas PERRON over 10 years ago
It seems to be a duplicate of #3574
Updated by Vincent MEMBRÉ over 10 years ago
- Target version changed from 2.6.10 to 2.6.11
Updated by Vincent MEMBRÉ about 10 years ago
- Target version changed from 2.6.11 to 2.6.12
Updated by Vincent MEMBRÉ about 10 years ago
- Target version changed from 2.6.12 to 2.6.13
Updated by Vincent MEMBRÉ almost 10 years ago
- Target version changed from 2.6.13 to 2.6.14
Updated by Jonathan CLARKE almost 10 years ago
- Target version changed from 2.6.14 to 2.6.16
Updated by Jonathan CLARKE almost 10 years ago
- Target version changed from 2.6.16 to 2.6.17
Updated by Nicolas PERRON over 9 years ago
- Target version changed from 2.6.17 to 2.6.18
Updated by Matthieu CERDA over 9 years ago
- Target version changed from 2.6.18 to 2.6.19
Updated by Vincent MEMBRÉ over 9 years ago
- Target version changed from 2.6.19 to 2.6.20
Updated by Nicolas CHARLES about 9 years ago
- Target version changed from 2.6.20 to 3.0.0
Just so that we keep track on this; CFEngine is tackling the issue of removing the 4K barrier, one step at a time.
This should work correctly on Rudder 3.0 (see ticket https://dev.cfengine.com/issues/2912)
Updated by François ARMAND about 9 years ago
Could someone test it in Rudder 3.0 so that we know if it works (or update the bug on CFEngine side ?) Nicolas ?
Updated by François ARMAND about 9 years ago
- Assignee changed from Nicolas CHARLES to Benoît PECCATTE
Updated by François ARMAND about 9 years ago
- Assignee changed from Benoît PECCATTE to Nicolas CHARLES
Updated by Nicolas CHARLES about 9 years ago
In 3.0, the limit is at 16ko, the error is at the promise validation time
The error message is:
input buffer overflow, can't enlarge buffer because scanner uses REJECT
Updated by François ARMAND about 9 years ago
- Description updated (diff)
OK, I'm updating the description accordingly.
Thanks
Updated by François ARMAND about 9 years ago
- Target version changed from 3.0.0 to 2.10.11
Updated by Vincent MEMBRÉ about 9 years ago
- Target version changed from 2.10.11 to 2.10.12
Updated by Benoît PECCATTE about 9 years ago
- Project changed from 24 to Rudder
- Category set to Techniques
Updated by Vincent MEMBRÉ about 9 years ago
- Target version changed from 2.10.12 to 2.10.13
Updated by Vincent MEMBRÉ about 9 years ago
- Target version changed from 2.10.13 to 2.10.14
Updated by Vincent MEMBRÉ almost 9 years ago
- Target version changed from 2.10.14 to 2.10.15
Updated by Vincent MEMBRÉ almost 9 years ago
- Target version changed from 2.10.15 to 2.10.16
Updated by Vincent MEMBRÉ over 8 years ago
- Target version changed from 2.10.16 to 2.10.17
Updated by Vincent MEMBRÉ over 8 years ago
- Target version changed from 2.10.17 to 2.10.18
Updated by Vincent MEMBRÉ over 8 years ago
- Target version changed from 2.10.18 to 2.10.19
Updated by Vincent MEMBRÉ over 8 years ago
- Target version changed from 2.10.19 to 2.10.20
Updated by Vincent MEMBRÉ over 8 years ago
- Target version changed from 2.10.20 to 2.11.18
Updated by Vincent MEMBRÉ about 8 years ago
- Target version changed from 2.11.18 to 2.11.19
Updated by Vincent MEMBRÉ about 8 years ago
- Target version changed from 2.11.19 to 2.11.20
Updated by Vincent MEMBRÉ about 8 years ago
- Target version changed from 2.11.20 to 2.11.21
Updated by Vincent MEMBRÉ almost 8 years ago
- Target version changed from 2.11.21 to 2.11.22
Updated by Vincent MEMBRÉ almost 8 years ago
- Target version changed from 2.11.22 to 2.11.23
Updated by Vincent MEMBRÉ over 7 years ago
- Target version changed from 2.11.23 to 2.11.24
Updated by Vincent MEMBRÉ over 7 years ago
- Target version changed from 2.11.24 to 308
Updated by Vincent MEMBRÉ over 7 years ago
- Target version changed from 308 to 3.1.14
Updated by Vincent MEMBRÉ over 7 years ago
- Target version changed from 3.1.14 to 3.1.15
Updated by Vincent MEMBRÉ over 7 years ago
- Target version changed from 3.1.15 to 3.1.16
Updated by Vincent MEMBRÉ over 7 years ago
- Target version changed from 3.1.16 to 3.1.17
Updated by Vincent MEMBRÉ over 7 years ago
- Target version changed from 3.1.17 to 3.1.18
Updated by Vincent MEMBRÉ about 7 years ago
- Target version changed from 3.1.18 to 3.1.19
Updated by Benoît PECCATTE about 7 years ago
- Tracker changed from Bug to User story
- Subject changed from CFengine is unable to handle too big files payloads in the Enforce a file content Technique to Handle big files payloads in the Enforce a file content Technique
Updated by Vincent MEMBRÉ about 7 years ago
- Target version changed from 3.1.19 to 3.1.20
Updated by Vincent MEMBRÉ almost 7 years ago
- Target version changed from 3.1.20 to 3.1.21
Updated by Vincent MEMBRÉ almost 7 years ago
- Target version changed from 3.1.21 to 3.1.22
Updated by Vincent MEMBRÉ over 6 years ago
- Target version changed from 3.1.22 to 3.1.23
Updated by Vincent MEMBRÉ over 6 years ago
- Target version changed from 3.1.23 to 3.1.24
Updated by Vincent MEMBRÉ over 6 years ago
- Target version changed from 3.1.24 to 3.1.25
Updated by Benoît PECCATTE over 6 years ago
- Target version changed from 3.1.25 to 4.1.9
Updated by Vincent MEMBRÉ over 6 years ago
- Target version changed from 4.1.9 to 4.1.10
Updated by Vincent MEMBRÉ about 6 years ago
- Target version changed from 4.1.10 to 4.1.11
Updated by Vincent MEMBRÉ about 6 years ago
- Target version changed from 4.1.11 to 4.1.12
Updated by Vincent MEMBRÉ almost 6 years ago
- Target version changed from 4.1.12 to 4.1.13
Updated by Vincent MEMBRÉ almost 6 years ago
- Target version changed from 4.1.13 to 4.1.14
Updated by Benoît PECCATTE over 5 years ago
- Target version changed from 4.1.14 to 4.1.15
Updated by Vincent MEMBRÉ over 5 years ago
- Target version changed from 4.1.15 to 4.1.16
Updated by Vincent MEMBRÉ over 5 years ago
- Target version changed from 4.1.16 to 4.1.17
Updated by Vincent MEMBRÉ over 5 years ago
- Target version changed from 4.1.17 to 4.1.18
Updated by Vincent MEMBRÉ over 5 years ago
- Target version changed from 4.1.18 to 4.1.19
Updated by Alexis Mousset about 5 years ago
- Target version changed from 4.1.19 to 4.1.20
Updated by Nicolas CHARLES about 5 years ago
- Status changed from Discussion to Rejected
This is corrected now, large payload are supported