Monitor Docker Containers With cAdvisor, Prometheus And Grafana

In the video below, we show how to monitor Docker Containers using cAdvisor, Prometheus and Grafana

Docker is a very popular container platform and like many IT deployments it’s a good idea to monitor what’s going on

Now while you can monitor the host and have alerts sent before resources run out, at some point you’ll want to know more details about resources being used by the individual containers

And one way to monitor containers in Docker is to use cAdvisor as the exporter which will present metrics in a format that Prometheus and Grafana can take advantage of

In this video we go over how to install cAdvisor as a container and configure Prometheus and Grafana to monitor your Docker containers

Useful links:
https://prometheus.io/docs/guides/cadvisor/
https://github.com/google/cadvisor
https://grafana.com/grafana/dashboards/

Assumptions:
Now as this video is about setting up cAdvisor I’m going to assume you already have Prometheus and Grafana setup

If not then I do have another video which shows you how to install these in containers in Docker

Create cAdvisor Container:
Container Advisor, or cAdvisor for short, can be run as a container and in this example we’ll set this up using Docker Compose

In which case we’ll need to edit the compose file I already have

nano docker-compose.yml

And add a new container

services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - '8080:8080'
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro

Now save and exit

For a new file, you need to define the services section

Otherwise you can just append the details for this container at the end of the existing section like I did

We want the latest version of this Docker image and I’m going to give the container a name of cadvisor to make it easier to identify

The default port being used is TCP 8080, but bear in mind this is a popular port, so it can conflict with an Apache server for instance, while for me it conflicted with Podman

If there is a conflict then the container won’t run but you can always change the port that the host exposes, for example, 8081:8080 will expose cAdvisor on TCP port 8081 to the outside world

I like to have the ability to manually stop containers, but otherwise have them started automatically, so I’m using unless-stopped instead of always as the restart option

In order to access the necessary resources, we need to map various folders on the host to container folders, however, these will be set to read-only mode

NOTE: On the Docker website, /var/run is set to read-write, whereas on the developer’s site this is read-only, so I’ve set this to read-only

NOTE: On the developer site there is an additional mapping for /dev/disk/ so I’ve added this in

Initial Testing:
Because the container is gathering information about local resources, that’s all we need to do to configure it

So we’ll start up the container in the background

docker compose up -d cadvisor

And then check that it’s working and stable

docker ps -l

As a quick test we can connect to the container using a web browser, for example

http://192.168.102.30:8080

Interestingly enough there’s a lot of information provided in this web page, including gauges and graphs

And you can drill down to find out information about individual containers

As an aside, although we’re setting this up for Docker, it’s also aware of Podman

Now if you do run into issues where the container won’t start because the port is already in use, you can check what’s using the port by running a command like this

sudo ss -tunlp

You then have to decide whether to change the port on that application, removing that application, changing the port for cAdivsor, etc. because two applications can’t use the same port

Configure Prometheus:
In order to be able to record all this information and monitor the containers over time, we’ll want to configure Prometheus to regularly scrape the metrics from cAdvisor

To do that we’ll edit the existing configuration file for Prometheus

nano prometheus/prometheus.yml

And add a new job

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['192.168.102.30:8080']

Now save and exit

As this is a container, we could restart it for the changes to take affect

However, as I’ve configured Prometheus to let me reload the config file, I’ll issue a command to do that instead

curl -X POST localhost:9090/-/reload

We can then check this is working by connecting to Prometheus via a web browser, e.g.

http://192.168.102.30:9090

Then navigate to Status | Targets

After a short period of time we should see a status of up/up for cadvisor

Install Grafana Dashboard:
To provide us with a higher level view over time we can connect Grafana to Prometheus and view the results in a dashboard

And while we could create our own dashboard, fortunately there are community members out there creating and publishing dashboards for others to use

Point your web browser to Grafana
https://grafana.com/grafana/dashboards/

Then in the search field you could filter these by name, for example docker

There are quite a few to choose from, but bear in mind that some may no longer be maintained and so may not work

If you like, you can change the sort order to Last Updated to see more recent releases

At the time of recording, the one by sheriffanous is the most recent and seems to contain the sort of information I’m looking for, so we’ll use that
https://grafana.com/grafana/dashboards/21361-docker-cadvisor-compute-resources/

Whichever dashboard you choose to go with, you’ll want to click the Copy ID to clipboard button to make importing easier

Login to Grafana via a web browser, for example

http://192.168.102.30:3000

In the top left corner click the menu button and select Dashboards

Now click on the New drop-down menu on the right and select Import

Paste in the ID, in this case 21361, into the Import via grafana.com field then click Load

Change the name of the dashboard if you’d prefer but lower down you have to select the Prometheus data source before clicking Import

I’m only running one instance of Docker so filtering on Docker Host won’t make any difference for me but it would be useful if you’re using Docker Swarm for instance

Now what I like about this dashboard is that by default it’s displaying the resource usage for all containers and this would make it easy to spot which one is using the most resource if there’s a problem

However, if I then want to focus on a particular container there is a filter option so that I can focus on just that one container

Sharing is caring!