Project

General

Profile

Actions
Copy link

Bug #12702

closed

Method copyResourceFile is quite inefficient

Added by Nicolas CHARLES almost 6 years ago. Updated over 5 years ago.

Status:
Released
Priority:
N/A
Assignee:
Nicolas CHARLES
Category:
Performance and scalability
Target version:
4.1.13
Pull Request:
https://github.com/Normation/rudder/p...
Severity:
Minor - inconvenience | misleading | easy workaround
UX impact:
User visibility:
Operational - other Techniques | Technique editor | Rudder settings
Effort required:
Priority:
32
Name check:
Fix check:
Regression:

Description

Method copyResourceFile is used to write static files (which don't change from what is on FS)
On my test system, with 1500 nodes, it takes about 10s out of 42s of writing file policy during a generation. Given the low scope of this function, there is probably way to do better

  /**
   * Copy a resource file from a technique to the node promises directory
   */
  private[this] def copyResourceFile

the only ressources I have are

TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,environment-variables.cf),common/1.0/environment-variables.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-nova-cron),common/cron/rudder-agent-nova-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,check-zypper.cf),common/1.0/check-zypper.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,properties.cf),common/1.0/properties.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-agent-community-cron),common/cron/rudder-agent-community-cron,false)
TechniqueFile(TechniqueResourceIdByName(common/1.0,rudder-stdlib.cf),common/1.0/rudder-stdlib.cf,true)
TechniqueFile(TechniqueResourceIdByName(common/1.0,minicurl),common/utilities/minicurl,false)

so about 7 or 8 files, of 1500 nodes, so less than 15000 written file in total

Targetting to 4.1 as it is quite impacting for user with large installation

Actions
Copy link
 #1

Updated by Nicolas CHARLES almost 6 years ago

Note:
It's ALWAYS the same files, so we could look it up only once
We could even hardlink or somthing

Actions
Copy link
 #2

Updated by Nicolas CHARLES almost 6 years ago

Ok, it seems the issue is that we read each time the technique content, and that is very expensive:

Tests metrics for resources creations

Creating the resource file, accessing technique repository, no content written in resource file: 12503 ms

    val destination = {
      val out = reportIdToReplace match {
        case None     => file.outPath
        case Some(id) => file.outPath.replaceAll(Policy.TAG_OF_RUDDER_MULTI_POLICY, id.getRudderUniqueId)
      }
      new File(rulePath+"/"+out)
    }

techniqueRepository.getFileContent(file.id) { optStream =>
      optStream match {
        case None => Failure(s"Can not open the technique resource file ${file.id} for reading")
        case Some(s) =>
          try {
            //FileUtils.copyInputStreamToFile(s, destination)
            Full(destination.getAbsolutePath)

Creating the resource file, not accessing the technique repository at all, no content in it: 2757 ms

    val destination = {
      val out = reportIdToReplace match {
        case None     => file.outPath
        case Some(id) => file.outPath.replaceAll(Policy.TAG_OF_RUDDER_MULTI_POLICY, id.getRudderUniqueId)
      }
      new File(rulePath+"/"+out)
    }
    Full(destination.getAbsolutePath)
    /*

[2018-05-25 21:42:43] DEBUG com.normation.rudder.services.policies.PromiseGenerationServiceImpl - Write node configurations : 2757 ms
Actions
Copy link
 #4

Updated by Benoît PECCATTE almost 6 years ago

  • Severity set to Minor - inconvenience | misleading | easy workaround
  • User visibility changed from Infrequent - complex configurations | third party integrations to Operational - other Techniques | Technique editor | Rudder settings
  • Priority changed from 0 to 32
Actions
Copy link
 #5

Updated by Nicolas CHARLES almost 6 years ago

  • Status changed from New to In progress
Actions
Copy link
 #6

Updated by Nicolas CHARLES almost 6 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/1973
Actions
Copy link
 #7

Updated by Nicolas CHARLES almost 6 years ago

It's roughtly 20-30% performance increase - depending on type of Directives (Directives based on Techniques created by technique editor benefit more from this change)

Actions
Copy link
 #8

Updated by Rudder Quality Assistant almost 6 years ago

  • Status changed from Pending technical review to Discussion
  • Assignee changed from François ARMAND to Nicolas CHARLES
Actions
Copy link
 #9

Updated by Nicolas CHARLES almost 6 years ago

  • Status changed from Discussion to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
Actions
Copy link
 #10

Updated by Rudder Quality Assistant almost 6 years ago

  • Assignee changed from François ARMAND to Nicolas CHARLES
Actions
Copy link
 #11

Updated by Nicolas CHARLES almost 6 years ago

  • Status changed from Pending technical review to Pending release
Actions
Copy link
 #12

Updated by Vincent MEMBRÉ over 5 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 4.1.13, 4.2.7 and 4.3.3 which were released today.

Actions
Copy link

Also available in: Atom PDF