Sizing the Kubernetes cluster

Component Pack considerations

Currently, if you follow the full installation procedure and install all Component Pack offerings, including the non-mandatory Sanity, Kubernetes Dashboard, ElasticStack and Activities Plus components, you will have 108 new pods in your cluster. Note that this is the total count, as a large majority of the pods are running in three replicas.

The number of replicas can be decreased and increased of course, depending on your needs. That being said, it is always good to have some extra resources so you can afford to scale one or more pods.

It is also important to know that Component Pack has components like ElasticSearch and ElasticStack that are going to run exclusively on the workers labeled as infrastructure. If you disable scheduling on those workers, it means they will schedule only pods that have hard requirements to run on infrastructure workers. This can have a big impact on how to size your cluster.

Operational considerations

You will be installing and upgrading different Component Pack (and non-Component Pack) services, which means that there will be points in time when instead of three pods there will be six up and running for the same service, while Kubernetes is for example upgrading it, or reinstalling it.

It is also important to decide how different Kubernetes versions are going to be managed in the future. If you are managing your own Kubernetes cluster, you need to decide if you will ever upgrade it yourself, or you will have to spin up a new cluster with a new version of Kubernetes, move your traffic there, and tear down the old one.

The first approach (upgrading the Kubernetes cluster yourself) requires more masters and workers since you need to have enough resources once one of the nodes goes down for upgrade.

The second approach (migrating from one cluster to another and destroying the old one) is more industry standard, and requires more resources only when you are migrating to the new version of Kubernetes. This approach is also much easier for operations, and in the majority of cases much faster.

Masters versus workers

Master nodes, in general, never schedule any non-system pods (they are exclusively used for scheduling anything running in kube-system namespace). However, the number of pods scheduled on masters depends on the number of masters and number of workers. For example, each new master and worker in the cluster will be represented by multiple pods on each master. This means that the sizing of a master that will have one worker attached can be different from the sizing of a master that will be part of a cluster with 3 masters and 10 workers.

Worker nodes, in general, can be any type of node, depending on what workload you want to run on it. Workers don't have to be all the same--they can be sized differently, as Kubernetes will ensure that each worker is utilized based on own resources (which is a function of the number of CPUs and the amount of memory versus the requirements for each pod's CPU and memory).

With or without high availability

If you can afford downtime for Component Pack, you can still run everything on one big machine (single node cluster). However, this is not advised for production environments, one reason being the official recommendation not to run more then 100 pods on a single node.

Perfect high availability would be at least three master nodes (to ensure quorum) and at least three infrastructure worker nodes (three because of the default of three replicas for each pod). In case you ever use ElasticStack, in an ideal world you would have a total of six workers, where three of them would be infrastructure with scheduling disabled. In this case, those three would handle ElasticSearch and ElasticStack only and the other three would handle everything else. However, this means a bigger cluster as well as higher cost.