Performance Tuning

Content Integration Framework provides a way to tune the performance of content event processing for the events received from Kafka topic(s). To do so, configuration properties can be managed in following file -

ASSET_PICKER_HOME/conf/events/tuning.properties

This file is laid down by Unica Platform installer with some default settings. Given below is the list of available properties & their significance. Do not set any property to blank in favor of its default value, instead put a comment sign (#) before it.

  • Kafka consumer tuning – These properties allow tuning message consumption from Kafka topics.
    Property Description
    kafka.consumer.threads.min Minimum Kafka consumer threads that will always be kept active. Default is 1, if not specified.
    Note: An additional property {service-name}.kafka.max-consumers must be set up in Platform configuration for each individual service (event source) to determine the concurrency limits for each event source. {service-name}.kafka.max-consumers should ideally be set to the total number of partitions of corresponding topics. The number specified for kafka.consumer.threads.min must be able to cater to the needs of all individual event sources thus configured using additional property in Platform ({service-name}.kafka.max-consumers).
    kafka.consumer.threads.max Maximum Kafka consumer thread count. In case application outruns the kafka.consumer.threads.min threads, a few more threads will be created. This number sets upper limit for overall thread count. Default is 32767, if not specified.
    kafka.consumer.threads.priority kafka.consumer.threads.priority Kafka consumer thread priority. Application can have many threads running for handling various things, such as requests coming from user interface, messages coming from Kafka topic etc. Thread priority determines the precedence of one thread over the other when both need CPU at the same time for their execution. This property determines the priority of overall event receipts. Default is 5, if not specified.
    • 1 - Least priority
    • 10 - Highest priority
    kafka.consumer.threads.max-idle-seconds Maximum idle time after which excessive threads over kafka.consumer.threads.min are disposed if additional threads are no longer active. Default is 60 seconds.
    kafka.consumer.max-poll-interval-ms

    Consumer poll interval in milliseconds. Default is 300000ms if not specified. This value is used for standard Kafka consumer configuration - max.poll.interval.ms.

    This setting can be overridden for individual event consumer service using {service-name}.kafka.max.poll.interval.ms additional parameter in Platform configuration.

    Value for this setting must be greater than the value of {processor.threads.max-caller-blocking-time-ms}.
    kafka.consumer.heartbeat-interval-ms

    Consumer thread heartbeat interval in milliseconds. This value is used for standard Kafka consumer configuration - heartbeat.interval.ms. Default is 3000ms, if not specified.

    This setting can be overridden for individual event consumer service using {service-name}.kafka.heartbeat.interval.ms additional parameter in Platform configuration.

    Note: Refer https://kafka.apache.org/documentation/#consumerconfigs for standard Kafka consumer configurations.
    kafka.consumer.session-timeout-ms

    Consumer session time out interval in milliseconds. This value is used for standard Kafka consumer configuration - session.timeout.ms. Default is 10000ms, if not specified.

    This setting can be overridden for individual event consumer service using {service-name}.kafka.session.timeout.ms additional parameter in Platform configuration.

    Note: Refer https://kafka.apache.org/documentation/#consumerconfigs for standard Kafka consumer configurations.
  • Event processor tuning – These properties allow tuning processing of events received via Webhook as well as from Kafka topics.
    Property Description
    processor.threads.min Minimum event processor threads that will always be kept active. Default is 1, if not specified.
    processor.threads.max Maximum event processor thread count. In case application outruns the processor.threads.max threads, a few more threads will be created. This number sets upper limit for overall thread count. Default is 32767, if not specified.
    processor.threads.priority Event processor thread priority. Application can have many threads running for handling various things, such as requests coming from user interface, messages coming from Kafka topic, event processing etc. Thread priority determines the precedence of one thread over the other when both need CPU at the same time for their execution. This property determines the priority of overall event processing. Default is 5, if not specified.
    • 1 - Least priority
    • 10 - Highest priority
    processor.threads.max-idle-seconds Maximum idle time after which excessive threads over processor.threads.min are disposed if additional threads are no longer active. Default is 60 seconds, if not specified.
    processor.threads.max-caller-blocking-time-ms Maximum time in milliseconds to wait before rejecting new events when all processor threads are busy processing already accepted events. If any processor thread becomes free before this duration, new event is given to it for processing. Default is 2000ms, if not specified.
    processor.threads.max-shutdown-await-seconds Maximum seconds to be allowed to processor threads to complete already accepted events before application shutdown. This setting is effective only in case of graceful application shutdown. If application is terminated abruptly, event processing will be stalled immediately without waiting for their completion. Default is 30 seconds if not specified.
    error.kafka.topic

    Name of Kafka topic where event processing errors can be sent in addition to logging into application logs. This is an optional property & there is no default topic name considered if it is not specified.

    This value can be overridden in Platform configuration for the services listening to Kafka topic for incoming events, using {service-name}.error.kafka.topic additional parameter.
  • Kafka producer tuning – These properties help for tuning message publishing to Kafka topics.
Property Description
kafka.producer.batch-size
Message producer batch size in bytes. Default is 16384 (16KB), if not specified. This setting can be overridden for individual message producer service using {service-name}.kafka.batch.size additional parameter in Platform configuration. Value of this property is used for standard Kafka producer configuration – batch.size.
Note: Refer https://kafka.apache.org/documentation/#producerconfigs for standard Kafka producer configurations.
kafka.producer.linger-ms
Number of milliseconds to wait for gathering kafka.producer.batch-size bytes of data before sending the complete batch. Default is 0, wherein producer does not wait for complete batch of messages and sends the message immediately. This setting can be overridden for individual message producer service using {service-name}.kafka.linger.ms additional parameter in Platform configuration. Value of this property is used for standard Kafka producer configuration – linger.ms.
Note: Refer https://kafka.apache.org/documentation/#producerconfigs for standard Kafka producer configurations.