![]() In doing this, there is no modification needed to the default Docker Logstash image. In this case, our configuration container exposes the configuration files to our Logstash container through volume sharing. The sidekick creates an atomic scheduling unit within Rancher. To achieve maximum flexibility we leveraged the confd tool and Rancher sidekick containers. Each organization is going to have different processing needs, formatting, tagging etc. The logstash configurations are the most volatile, and unique to an organization as they provide the interfaces for the collection, indexing, and shipping of the logs. To achieve maximum flexibility with configuration, we built a confd container that consumes KV, or Key Value, data for its configuration values. The real magic is in configuration of Logstash at each of the tiers. Looking at the DockerHub registry, we found there is already an official Logstash image maintained by Docker. In general, we try to reuse as much as possible from the community. To build the Logstash stack we stared as we usually do. ![]() A good read on how to scale Logstash can be found on Elastic’s web page here: Deploying and Scaling Logstash. As a user, you can also scale and adjust each tier to suit your needs. Once it processes the data it forwards it along to the final destination, which is Elasticsearch.Įach Logstash container has a configuration sidekick that provides configuration through a shared volume.īy breaking the stack into these tiers, you can scale and adapt each part without major impact to the other parts of the stack. This tier does the heavy lifting and is often a bottleneck in a log pipeline. Processing tier – removes messages from the queue, and applies filter plugins to the logs that manipulate the data to a desired format. It acts as a buffer between the collection and processing tier. Queuing tier – provided by Redis, a very fast in memory database. Once it receives these logs it places them quickly onto a Redis Queue. Our Logstash implementation includes three tiers, one each for collection, queueing and processing.Ĭollection- responsible for providing remote endpoints for logging inputs. So, we have always had to run our own Logstash pipeline. There was no Logstash end point provided, though it was sufficiently documented how to use Found with Logstash. Found provided us Elasticsearch and Kibana. Each part of the stack has its own needs and considerations. One of the interesting things we realized while using Found for Elasticsearch is that the ELK stack really is made up of discrete parts. Previously, we have been using Elastic’s Found product and have been very impressed. Together with Kibana, a powerful visualization tool that consumes Elasticsearch data, you have amazing ability to gain insights from your logging. It can take our processed output from Logstash, analyze and provides an interface to query all of our logging data. It is a really powerful tool in the logging arsenal.Įlasticsearch is a document database that is really good at search. Logstash provides a pipeline for shipping logs from various sources and input types, combining, massaging and moving them into Elasticsearch, or several other stores. We decided to use Rancher to build and run a scalable ELK stack to manage all of these logs.įor those that are unfamiliar with the ELK stack, it is made up of Elasticsearch, Logstash and Kibana. As we conduct more and more testing on these environments we have found the need to centrally aggregate the logs from each environment. At Rancher Labs we generate a lot of logs in our internal environments.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |