Configuring logging for Kubernetes with Elasticsearch, Fluentd, and Kibana

You can configure centralized logging for your Kubernetes deployment using Elasticsearch, Fluentd, and Kibana (EFK).

Using this selection of tools:

Fluentd collects logs from pods running in the cluster and forwards them to Elasticsearch.
Elasticsearch stores, indexes, and searches logs.
Kibana connects to Elasticsearch to generate queries and to visualize results.

Use this document as a starting point for your centralized logging solution. You can use other software, products, or guides to further customize your logging solution to suite your business requirements.

Before you begin

Ensure that you have a Kubernetes cluster with Helm installed.
Ensure that kubectl command line (kubectl CLI) is installed.
Obtain the sample Helm Chart deployment configuration files within the sample_values directory of your cloned HCL Commerce Helm Chart Git project.
Depending on your deployment type, enable JSON logging for your HCL Commerce services by using the common.enableJsonLogging Vault configuration or Docker container configuration variables.
For more information, see:
- Logging configuration within the Required Helm Chart configuration for HCL Commerce Version 9.1.
- Starting containers by retrieving parameters from container environment variables.

Procedure

Create a namespace for your EFK stack.
```
kubectl create namespace log
```

Add the bitnami Helm repository.

helm repo add bitnami https://charts.bitnami.com/bitnami

Deploy an instance of Elasticsearch for logging usage.
1. Deploy Elasticsearch using the sample log-elasticsearch-values.yaml configuration file from the sample_values directory of your cloned HCL Commerce Helm Chart Git project.
```
helm install elasticsearch bitnami/elasticsearch -n log -f log-elasticsearch-values.yaml --version "15.3.0"
```
2. Monitor the deployment and ensure that all pods are healthy.
Deploy Kibana.
1. Deploy Kibana using the sample log-kibana-values.yaml configuration file from the sample_values directory of your cloned HCL Commerce Helm Chart Git project.
```
helm install kibana bitnami/kibana -n log -f log-kibana-values.yaml --version "8.1.3"
```
2. Monitor the deployment and ensure that all pods are healthy.
Configure and deploy Fluentd.
1. Create a configuration map with the sample log-forwarder-configmap.yaml configuration map.
```
kubectl apply -f log-forwarder-configmap.yaml -n log
```
  Note: If you use a namespace other than log you must update the value used in this sample file.
  Change "log" in the line below to the namespace that was created in step #1.
  host fluentd-0.fluentd-headless.log.svc.cluster.local
2. Deploy Fluentd using the sample log-fluentd-values.yaml configuration file from the sample_values directory of your cloned HCL Commerce Helm Chart Git project.
```
helm install fluentd bitnami/fluentd -n log -f log-fluentd-values.yaml --version "3.7.4"
```
3. Monitor the deployment and ensure that all pods are healthy.

Results

A centralized logging solution is implemented for your HCL Commerce services running on Kubernetes.