ELK Overview

The ELK Stack is a collection of three open-source products: Elasticsearch, Logstash, and Kibana.

The features of the ELK stack are as follows:

  • Centralized logging and monitoring to identify problems with containers, or applications, hosted inside the pod
  • Visualization tools to represent data using a graph or a chart
  • Host ELK as a standalone system or a Docker container or a Kubernetes pod

Use the scripts, configured on Unica helm charts, to install any other database clients automatically and seamlessly.

Filebeat

Filebeat collects and ships log files, and is also the most commonly used beat. You can install Filebeat on almost any operating system, or also as a Docker container. It includes internal modules for platforms like Apache, MySQL, Docker, MariaDB, Kafka, and many more.

Filebeat is very efficient and it displays this in how it handles backpressure. If Logstash is busy, Filebeat slows down its read rate and picks up the beat once the slowdown is completed.

For Unica, Filebeat is deployed as a sidecar container inside Platform pod, and Platform and Filebeat, as a sidecar, runs as a container inside a pod. The Filebeat has a read-only access to persistent volume. It reads the logs from the logs folder, specified in configmap for all Unica applications, to keep sending logs to ELK.

metricbeat

Metricbeat collects and reports various system-level metrics for various systems and platforms. Metricbeat also supports internal modules for collecting statistics from specific platforms.

For Unica, metricbeat is deployed as a deamonset in kube-system namespace to collect metrics from the metric-server in OpenShift. In Kubernetes, it connects to both the Kubernetes API-server and the metric server.

Fluentd

It is a smart metrics and log shipper. For Unica, Fluentd is deployed as a deamonset in custom namespace to collect metrics from Kubernetes.

Advantages of using ELK

Logging , keeping historical logs, or monitoring the logs is a real challenge in containerized applications. If you destroy a container, everything is lost, including the logs. The advantages of using ELK are as follows:

  • Maintains and keeps all the data, and makes this activity easy, even if the cluster, pod, or node is destroyed.
  • Allows searching of all the logs in a single place.
  • Helps find issues in multiple servers, or pods, by connecting logs during a specific time frame.