Performance Tuning
Content Integration Framework provides a way to tune the performance of content event processing for the events received from Kafka topic(s). To do so, configuration properties can be managed in following file -
ASSET_PICKER_HOME/conf/events/tuning.properties
This file is laid down by Unica Platform installer with some default settings. Given below is the list of available properties & their significance. Do not set any property to blank in favor of its default value, instead put a comment sign (#) before it.
- Kafka consumer tuning – These properties allow tuning
message consumption from Kafka topics.
Property Description kafka.consumer.threads.min Minimum Kafka consumer threads that will always be kept active. Default is 1, if not specified. Note: An additional property{service-name}.kafka.max-consumers
must be set up in Platform configuration for each individual service (event source) to determine the concurrency limits for each event source.{service-name}.kafka.max-consumers
should ideally be set to the total number of partitions of corresponding topics. The number specified forkafka.consumer.threads.min
must be able to cater to the needs of all individual event sources thus configured using additional property in Platform({service-name}.kafka.max-consumers)
.kafka.consumer.threads.max Maximum Kafka consumer thread count. In case application outruns the kafka.consumer.threads.min threads, a few more threads will be created. This number sets upper limit for overall thread count. Default is 32767, if not specified. kafka.consumer.threads.priority kafka.consumer.threads.priority Kafka consumer thread priority. Application can have many threads running for handling various things, such as requests coming from user interface, messages coming from Kafka topic etc. Thread priority determines the precedence of one thread over the other when both need CPU at the same time for their execution. This property determines the priority of overall event receipts. Default is 5, if not specified. - 1 - Least priority
- 10 - Highest priority
kafka.consumer.threads.max-idle-seconds Maximum idle time after which excessive threads over kafka.consumer.threads.min
are disposed if additional threads are no longer active. Default is 60 seconds.kafka.consumer.heartbeat-interval-ms Consumer thread heartbeat interval in milliseconds. This value is used for standard Kafka consumer configuration - heartbeat.interval.ms. Default is 3000ms, if not specified.
This setting can be overridden for individual event consumer service using
{service-name}.kafka.heartbeat.interval.ms
additional parameter in Platform configuration.Note: Refer https://kafka.apache.org/documentation/#consumerconfigs for standard Kafka consumer configurations.kafka.consumer.session-timeout-ms Consumer session time out interval in milliseconds. This value is used for standard Kafka consumer configuration - session.timeout.ms. Default is 10000ms, if not specified.
This setting can be overridden for individual event consumer service using
{service-name}.kafka.session.timeout.ms
additional parameter in Platform configuration.Note: Refer https://kafka.apache.org/documentation/#consumerconfigs for standard Kafka consumer configurations. - Event processor tuning – These properties allow tuning
processing of events received via Webhook as well as from Kafka topics.
Property Description processor.threads.min Minimum event processor threads that will always be kept active. Default is 1, if not specified. processor.threads.max Maximum event processor thread count. In case application outruns the processor.threads.max
threads, a few more threads will be created. This number sets upper limit for overall thread count. Default is 32767, if not specified.processor.threads.priority Event processor thread priority. Application can have many threads running for handling various things, such as requests coming from user interface, messages coming from Kafka topic, event processing etc. Thread priority determines the precedence of one thread over the other when both need CPU at the same time for their execution. This property determines the priority of overall event processing. Default is 5, if not specified. - 1 - Least priority
- 10 - Highest priority
processor.threads.max-idle-seconds Maximum idle time after which excessive threads over processor.threads.min
are disposed if additional threads are no longer active. Default is 60 seconds, if not specified.processor.threads.max-caller-blocking-time-ms Maximum time in milliseconds to wait before rejecting new events when all processor threads are busy processing already accepted events. If any processor thread becomes free before this duration, new event is given to it for processing. Default is 2000ms, if not specified. processor.threads.max-shutdown-await-seconds Maximum seconds to be allowed to processor threads to complete already accepted events before application shutdown. This setting is effective only in case of graceful application shutdown. If application is terminated abruptly, event processing will be stalled immediately without waiting for their completion. Default is 30 seconds if not specified. error.kafka.topic Name of Kafka topic where event processing errors can be sent in addition to logging into application logs. This is an optional property & there is no default topic name considered if it is not specified.
This value can be overridden in Platform configuration for the services listening to Kafka topic for incoming events, using
{service-name}.error.kafka.topic
additional parameter. - Kafka producer tuning – These properties help for tuning message publishing to Kafka topics.
Property | Description |
kafka.producer.batch-size |
Message producer batch size in bytes. Default is 16384 (16KB), if
not specified. This setting can be overridden for individual
message producer service using
{service-name}.kafka.batch.size additional
parameter in Platform configuration. Value of this property is
used for standard Kafka producer configuration –
batch.size.Note: Refer https://kafka.apache.org/documentation/#producerconfigs for standard Kafka
producer configurations.
|
kafka.producer.linger-ms |
Number of milliseconds to wait for gathering
kafka.producer.batch-size bytes of data
before sending the complete batch. Default is 0, wherein
producer does not wait for complete batch of messages and sends
the message immediately. This setting can be overridden for
individual message producer service using
{service-name}.kafka.linger.ms additional
parameter in Platform configuration. Value of this property is
used for standard Kafka producer configuration – linger.ms.Note: Refer https://kafka.apache.org/documentation/#producerconfigs for standard Kafka
producer configurations. |