3.4 KiB
Fluentd
Another data collector that I wanted to explore as part of this observability section was Fluentd. An Open-Source unified logging layer.
Fluentd treats logs as JSON
Collecting logs from files
We all have applications that write to a .log file format. Fluentd has the ability to read logs from a file, with the configuration we have set, we will cover that shortly. Below you can see me bringing up the two containers, docker-compose up -d file-myapp and then docker-compose up -d fluentd
If you watch Introduction to Fluentd: Collect logs and send almost anywherefrom the "That DevOps Guy" (Some amazing content that has been linked throughout this whole challenge) I am using his example here.
With the container file-myapp there is a script in there to add to an example log file. You can see this happening below.
The script that we have running inside of our file-myapp container looks like this:
#!/bin/sh
while true
do
echo "Writing log to a file"
echo '{"app":"file-myapp"}' >> /app/example-log.log
sleep 5
done
We are using the tail plugin within fluentd which allows us to read those events from the tail of text files, similar to the tail -F command.
We can also use the HTTP plugin which allows us to send events through HTTP requests. This is what we will see with the http-myapp container in the repository. This container also runs a similar shell script to generate logs
#!/bin/sh
while true
do
echo "Sending logs to FluentD"
curl -X POST -d 'json={"foo":"bar"}' http://fluentd:9880/http-myapp.log
sleep 5
done
Container logging
We are going to be using docker-compose to bring up a small demo environment that can demonstrate fluentd. The docker compose file is going to bring up the following containers.
container_name: fluentd container_name: http-myapp container_name: file-myapp container_name: elasticsearch container_name: kibana
Resources
- Understanding Logging: Containers & Microservices
- The Importance of Monitoring in DevOps
- Understanding Continuous Monitoring in DevOps?
- DevOps Monitoring Tools
- Top 5 - DevOps Monitoring Tools
- How Prometheus Monitoring works
- Introduction to Prometheus monitoring
- Promql cheat sheet with examples
- Log Management for DevOps | Manage application, server, and cloud logs with Site24x7
- Log Management what DevOps need to know
- What is ELK Stack?
- Fluentd simply explained
See you on Day 82