Setting up Elastic Stack

What is Elastic Stack?

The Elastic Stack is a collection of tools that collects log data and helps you visualize those logs in a central location. Using queries, you can analyze those logs to help determine how well your Component Pack for Connections deployment is working. Built on an open-source foundation, the Elastic Stack lets you reliably and securely take data from any source, in any format, and search, analyze, and visualize it in real time.

The Elastic Stack includes the following components:

Beats and Filebeat

Beats is a platform that supports single-purpose data shippers, such as Filebeat, which collects log events in your deployment and forwards them to Logstash. Filebeat runs as a daemonset in Kubernetes, which means that there will be 1 Filebeat pod on every node in your Kubernetes cluster, including each master and worker node. Servers outside of the Kubernetes cluster are not included, so you will not see Filebeat pods running on storage nodes, load balancers, and so on.

Logstash

Logstash is an open-source data processor that receives data from Filebeat, transforms it, and then sends it to Elasticsearch. Logstash runs as a 3-pod, load-balanced, stateful set in Kubernetes, tainted (flagged) to be assigned to an infrastructure node (if an infrastructure node exists). Logstash filters and mutates logs before sending the logs to Elasticsearch. In the context of Component Pack, logs related to Filebeat and Logstash are filtered out by default, and annotations are added to the document for filtering to Elasticsearch. Adding annotations converts the raw data into information by adding useful data to the logs, such as the Kubernetes pod name. When visualizing the data, you can then filter the data using the annotations. For example, you can filter by pod name.

Elasticsearch

Elasticsearch is a search and analytics engine that stores the logging information that it receives from Logstash. (Elasticsearch can also be used in your Connections deployment to support the metrics, type-ahead search, and Orient Me features).

Elasticsearch Curator

Elasticsearch Curator is a tool for managing your indices by periodically removing older data.

Kibana

Kibana runs in Kubernetes and provides a dashboard to help you visualize the logging data stored in Elasticsearch.

Deploying the Elastic Stack with Component Pack for Connections

The Elastic Stack integrates with the existing Elasticsearch cluster deployed within Component Pack, optimizing resources on the system (the Elastic Stack is dependent on Elasticsearch). Logs collected by Filebeat are processed by Logstash for storage in Elasticsearch, and are discoverable by Kibana.

Review the following considerations before deploying the Elastic Stack:

• The Elastic Stack and Component Pack for Connections were validated with Kubernetes v1.11.6, and are only supported with that version.
• The Elasticsearch cluster must be deployed and running.
• Sufficient resources must be available on the worker nodes that will host the Elastic Stack pods.
  • For best results, deploy the Elastic Stack pods to nodes that are tainted (flagged) for infrastructure use. For information, see Labeling and tainting worker nodes for Elasticsearch.
  • Check the sizing requirements document for details; see the Connections system requirements.
• (Optional) Provide additional storage if you want to store logs from Docker within the Elasticsearch persistent volumes.