Journey Engine Cluster Configuration
In production systems, Kafka is typically run as a cluster. This allows better load management and handling capabilities as well as providing fault tolerance.
The Journey application, and the Journey Engine specifically utilizes this ability of Kafka to be setup in a cluster for high volume production systems which also need redundancy in case of the failure of a single node. In this document we detail out the proposed layout and required configuration to host Journey Engine on a typical production cluster. The figures and number are indicative and will need to change as per actual requirements.
- There is single shared Database Server for all applications and nodes.
- Servers 01 – 03 run combinations of Journey Engine and Kafka.
- It is recommended that the setup has a minimum of 2 instances of Journey Engine and 3 Kafka brokers.
- Server 04 runs Platform, Journey Web and Zookeeper. If other applications are also required and installed on this Server, the configuration shall need to be adjusted accordingly.
- Journey Engine
- CPU – 8 cores
- RAM – 32 GB
- Disk Space – 250 GB
- Kafka
- CPU – At least 4 cores
- RAM – 32 GB
- Disk Space – 500 GB
- Journey Web + Platform
- CPU – 4 cores
- RAM – 32 GB
- Disk Space – 250 GB
- Database
- CPU – 4 cores
- RAM – 32 GB
- Disk Space – 500 GB
Required Configurations for Cluster
- Application properties
- spring.ignite.ipFinder.List=< IPs of all system to be part of
cluster>
- Example - spring.ignite.ipFinder.List =10.115.36.20,10.115.36.21,10.115.36.22
- This is to be done on all nodes where Journey Engine is running
- If new node is to be added in cluster then IP should be added to this property on all servers and nodes to be restarted.
- journey.ignite.reload=<set to all, will reload all active journeys>
- Example - journey.ignite.reload=all
- Example - journey.ignite.reload=all This property is to be enabled on Master Node – the node that is started first and shutdown last - only
- Other Node(s) will get replicated data
- journey.ignite.cluster=true – on all nodes
- Journey.engine.secondary.node -- this property will have value as false for primary node, for rest of Engine nodes this property will have value as true.
- Journey.cache.sync property to be enabled
- Only on Master Node or any one node which will be started first and closed in last
- spring.ignite.ipFinder.List=< IPs of all system to be part of
cluster>
- Before starting Zookeeper and Kafka – increase ( ulimit -n 100000, on linux )
- Zookeeper port (2181 default) should be accessible from all servers in cluster
- Test by using telnet <IP> 2181
- Broker port (default 9092) should be accessible from all servers where engine is running
- After starting Zookeeper, use the following command to check status if
application is up and port is occupied
- netstat –tulnp | grep 2181
- Each Broker should point to same zookeeper
- Under Broker property set replication to minimum 2,
- If cluster contains 3 servers set the following values
- offsets.topic.replication.factor=2
- transaction.state.log.replication.factor=2
- transaction.state.log.min.isr=2
- If cluster contains 3 servers set the following values
Starting Journey Engine
- Start Zookeeper
- Start Kafka brokers on all servers
- Start Journey Engine nodes in order
- The Engine node that is started first – the Master Node – should be the last node whilst shutting down. This is important for Ignite Cluster to avoid corruption of cached data