HCL Commerce Version 9.1.14.0 or later

Troubleshooting: Numerous NiFi "unable to write flowfile" messages

After deploying Version 9.1.14 updates, a large number of NiFi interface warnings are generated, stating that the system is unable to write flowfile contents due to archive file size constraints. This issue can be resolved by changing the nifi.content.repository.archive.max.usage.percentage property in the nifi.properties file or setting it as an environment variable in the NiFi container.

Problem

After deploying Version 9.1.14 updates, many NiFi interface warnings are generated when building a full index. The warnings appear similar to the following:
Unable to write flowfile contents to content repository container default due to 
archive file size constraints, waiting for archive cleanup.

Solution

A change to archive memory management was made in HCL Commerce Search Version 9.1.14.0, in order to improve performance. Some memory is swapped out to disk, and the warnings about the content repository container are generated when a predefined percentage of disk space becomes unavailable during the swapping process. The confguration file nifi.properties contains a property, nifi.content.repository.archive.max.usage.percentage that is set to 80% by default. This is the maximum disk space permitted for the operation. To change the percentage value, you can set the property as an environment variable in the NiFi container. For example, to prevent the warning message from appearing unless disk space is less than 10%, you could set the environment variable NIFI_CONTENT_REPOSITORY_ARCHIVE_MAX_USAGE_PERCENTAGE: "90%" in the container.
Note: There are certain prerequisites to using this method. When you are working a Docker Compose environment, free disk space must be greater than 20%. In the K8S Docker orchestration system, NiFi PV free disk space must be higher than 20%.

Allocate at least 200 GB of disk space to virtual memory, and preferably 400GB in production environments.

The optimal percentage for performance purposes is highly deployment-specific. You may be able to operate efficiently with the environment variable set as low as 10% to 30%.