Monitor Docker Containers With cAdvisor, Prometheus And Grafana
![Monitor Docker Containers With cAdvisor, Prometheus And Grafana](https://www.techtutorials.tv/sections/management-and-monitoring/monitor-containers-with-cadvisor-prometheus-and-grafana/thumbnail.jpg)
In the video below, we show how to monitor Docker Containers using cAdvisor, Prometheus and Grafana
Docker is a very popular container platform and like many IT deployments it’s a good idea to monitor what’s going on
Now while you can monitor the host and have alerts sent before resources run out, at some point you’ll want to know more details about resources being used by the individual containers
And one way to monitor containers in Docker is to use cAdvisor as the exporter which will present metrics in a format that Prometheus and Grafana can take advantage of
In this video we go over how to install cAdvisor as a container and configure Prometheus and Grafana to monitor your Docker containers
Useful links:
https://prometheus.io/docs/guides/cadvisor/
https://github.com/google/cadvisor
https://grafana.com/grafana/dashboards/
Assumptions:
Now as this video is about setting up cAdvisor I’m going to assume you already have Prometheus and Grafana setup
If not then I do have another video which shows you how to install these in containers in Docker
Create cAdvisor Container:
Container Advisor, or cAdvisor for short, can be run as a container and in this example we’ll set this up using Docker Compose
In which case we’ll need to edit the compose file I already have
nano docker-compose.yml
And add a new container
services:
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
ports:
- '8080:8080'
restart: unless-stopped
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
Now save and exit
For a new file, you need to define the services section
Otherwise you can just append the details for this container at the end of the existing section like I did
We want the latest version of this Docker image and I’m going to give the container a name of cadvisor to make it easier to identify
The default port being used is TCP 8080, but bear in mind this is a popular port, so it can conflict with an Apache server for instance, while for me it conflicted with Podman
If there is a conflict then the container won’t run but you can always change the port that the host exposes, for example, 8081:8080 will expose cAdvisor on TCP port 8081 to the outside world
I like to have the ability to manually stop containers, but otherwise have them started automatically, so I’m using unless-stopped instead of always as the restart option
In order to access the necessary resources, we need to map various folders on the host to container folders, however, these will be set to read-only mode
NOTE: On the Docker website, /var/run is set to read-write, whereas on the developer’s site this is read-only, so I’ve set this to read-only
NOTE: On the developer site there is an additional mapping for /dev/disk/ so I’ve added this in
Initial Testing:
Because the container is gathering information about local resources, that’s all we need to do to configure it
So we’ll start up the container in the background
docker compose up -d cadvisor
And then check that it’s working and stable
docker ps -l
As a quick test we can connect to the container using a web browser, for example
http://192.168.102.30:8080
Interestingly enough there’s a lot of information provided in this web page, including gauges and graphs
And you can drill down to find out information about individual containers
As an aside, although we’re setting this up for Docker, it’s also aware of Podman
Now if you do run into issues where the container won’t start because the port is already in use, you can check what’s using the port by running a command like this
sudo ss -tunlp
You then have to decide whether to change the port on that application, removing that application, changing the port for cAdivsor, etc. because two applications can’t use the same port
Configure Prometheus:
In order to be able to record all this information and monitor the containers over time, we’ll want to configure Prometheus to regularly scrape the metrics from cAdvisor
To do that we’ll edit the existing configuration file for Prometheus
nano prometheus/prometheus.yml
And add a new job
- job_name: 'cadvisor'
static_configs:
- targets: ['192.168.102.30:8080']
Now save and exit
As this is a container, we could restart it for the changes to take affect
However, as I’ve configured Prometheus to let me reload the config file, I’ll issue a command to do that instead
curl -X POST localhost:9090/-/reload
We can then check this is working by connecting to Prometheus via a web browser, e.g.
http://192.168.102.30:9090
Then navigate to Status | Targets
After a short period of time we should see a status of up/up for cadvisor
Install Grafana Dashboard:
To provide us with a higher level view over time we can connect Grafana to Prometheus and view the results in a dashboard
And while we could create our own dashboard, fortunately there are community members out there creating and publishing dashboards for others to use
Point your web browser to Grafana
https://grafana.com/grafana/dashboards/
Then in the search field you could filter these by name, for example docker
There are quite a few to choose from, but bear in mind that some may no longer be maintained and so may not work
If you like, you can change the sort order to Last Updated to see more recent releases
At the time of recording, the one by sheriffanous is the most recent and seems to contain the sort of information I’m looking for, so we’ll use that
https://grafana.com/grafana/dashboards/21361-docker-cadvisor-compute-resources/
Whichever dashboard you choose to go with, you’ll want to click the Copy ID to clipboard button to make importing easier
Login to Grafana via a web browser, for example
http://192.168.102.30:3000
In the top left corner click the menu button and select Dashboards
Now click on the New drop-down menu on the right and select Import
Paste in the ID, in this case 21361, into the Import via grafana.com field then click Load
Change the name of the dashboard if you’d prefer but lower down you have to select the Prometheus data source before clicking Import
I’m only running one instance of Docker so filtering on Docker Host won’t make any difference for me but it would be useful if you’re using Docker Swarm for instance
Now what I like about this dashboard is that by default it’s displaying the resource usage for all containers and this would make it easy to spot which one is using the most resource if there’s a problem
However, if I then want to focus on a particular container there is a filter option so that I can focus on just that one container
Sharing is caring!