How To Install Prometheus And Grafana On Docker

In the video below, we show how to install Prometheus and Grafana on Docker

Monitoring IT devices is incredibly important because it’s in our interests to be proactive rather than reactive

For instance, it’s better know when a hard drive has reached 70% capacity than it is to find a computer has stopped working because it ran out of disk space

Now there are lots of monitoring tools out there that you can use

But an interesting open source combination that’s available for free is Prometheus and Grafana

Useful links:
https://prometheus.io/docs/introduction/overview/
https://github.com/prometheus/node_exporter
https://prometheus.io/docs/guides/node-exporter/
https://prometheus.io/docs/prometheus/latest/installation/
https://grafana.com/docs/grafana-cloud/quickstart/docker-compose-linux/
https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/
https://grafana.com/grafana/dashboards/
https://grafana.com/grafana/dashboards/?collector=nodeexporter&dataSource=prometheus&orderBy=reviewsCount&direction=desc&category=hostmetrics

Assumptions:
As the video is about installing Prometheus and Grafana I’m going to assume you already have Docker setup

If not then I do have another video which shows you how to install that in a Debian VM running on Proxmox VE

Setup Node Exporter:
Now in order to get metrics from a device we need an exporter which will gather these and make them available for Prometheus to collect on a regular basis

So in this demo we’ll install Node Exporter, which is popular for *NIX servers

As we’re running Docker on this server, we’ll use Docker Compose to run this in a container

nano docker-compose.yml

version: '3.8'

services:
  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    network_mode: host
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'

Now save and exit

The above is directly copied from the main website, but do check there in case any changes have been made since this recording was made

This is the first container I’m defining for Docker so a version has been added

Then under services: we define the container to run

We want the latest version of node-exporter and to make it easier to identify we give it a name of node_exporter

In the example provided, the goal is to monitor the host itself

This is a container we want running all the time. We want it to be restarted after a reboot, but not if we manually stop it during a maintenance window for instance. So we set restart to unless-stopped rather than always

As we want mertrics from the host, the example maps /host to the root drive of the host itself i.e. /, but we want to limit access and so set this to read only and rslave

To start up the container we run the command

docker compose up -d

This will start all containers defined, if they aren’t already running, but they will run in the background so we don’t have to leave a terminal session open

Then we can check this is working by pointing a web browser to port 9100 of the host and we should see a page for Node Exporter and a link which provides details about the metrics gathered

http://192.168.101.30:9100

Configure Prometheus:
For Prometheus to work, we need to create a configuration file that defines what we want it to do

Where you want to keep this is up to you but I’m going to give each container its own folder

mkdir prometheus

nano prometheus/prometheus.yml

global:
  scrape_interval: 1m

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node'
    static_configs:
      - targets: ['192.168.101.30:9100']

Now save and exit

Usually you would define default or in this case global settings and override those if necessary within a job that Prometheus will run

You can set the gathering or scrape interval to what you like, but 1 minute seems fine as I don’t want to overload devices

We then create a target job which is for Prometheus itself, and we’ve labelled this as prometheus

Then we create another which is to connect to Node Exporter, and we’ve labelled this as node

Bear in mind, although this is all running on the same computer, Prometheus and Node Exporter are separate Docker containers

From Prometheus’s perspectve, localhost means itself and so we can use that for the Prometheus target

But, for Node Exporter we need something different

Now I don’t want to start overcomplicating things so I’m just using the IP address of the computer that Docker runs on

One thing to bear in mind though, is that if you use UFW you can run into problems

Because this Docker instance is a VM in Proxmox VE, it was easier to restrict external access by using the firewall options in Proxmox VE itself

Setup Prometheus:
The next thing to do is to set up a container for Prometheus

So we’ll edit our Docker Compose file

nano docker-compose.yml

volumes:
  prometheus-data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - '9090:9090'
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--web.enable-lifecycle'
      - '--config.file=/etc/prometheus/prometheus.yml'

Now save and exit

This is the Docker Compose equivalent of what you’ll find on the main website with some adjustments

I haven’t changed the ports that it will use for connectivity but I’ve given this a name and set it to restart unless stopped because there will be times when it needs manually shutting down

By default, Prometheus only retains data it collects for 15 days, which is fine for me, but we still want to this data to survive a reboot for instance

To do that I’ve configured the file to create a volume. Otherwise data will be stored within the container itself and will be lost if it’s ever stopped or destroyed

And then mapped the /prometheus folder within the container to this volume

In this example we’ve told Docker to create a volume referenced as prometheus-data, if it doesn’t already exist

Without any further information provided Docker will create a volume made up of the user’s name and the volume name we’ve provided the first time this image runs

These volumes can be found in the /var/lib/docker/volumes folder

TIP: To see a list of volumes run the following command

docker volume ls

For further details about a volume run the following command

docker volume inspect <volumename>

But replace with the name of the volume

We need to tell Prometheus where the configuration file is, so we map the YAML file in the container to the one we created earlier

In addition, the --web.enable-lifecycle command is added so you can reload the configuration file for Promethuse while it’s running instead of having to restart the container

NOTE: Because of the extra command section I’d added in, an additional command was needed to define the config file. Without a command section, I found the container boots up fine, but after adding this, it would break and warn it cannot find the config file

In any case, do check the developer website in case any changes have been made since this recording

Now it’s time to start this container

docker compose up -d

Wait for this to complete, then if you point a web browser to port 9090 of your computer running docker, you’ll be connected to Prometheus

http://192.168.101.30:9090

Click on the Status menu option and select Service Discovery and you should see two entries because of the jobs we defined

Assuming everything is working, they’ll each show 1/1 active targets

Next, click on the Status menu option and select Targets

You should see the state for both of these to be up

If not you’ll need to investigate and resolve the cause

TIP: To check a container’s log file run the command

docker container logs <containerName>

docker container logs <containerID>

But replace <containerName> or <containerID> with the actual container’s name or ID respectively

If you make any changes to Prometheus' config file, it can be reloaded by using a command like the following

curl -X POST localhost:9090/-/reload

The -X option is used because we want to send a command to POST i.e. send data rather than the default GET command, and in this case we want to send an instruction to reload

You can check if and when the last reload was done by pointing the web browser to the /status page, for example
http://192.168.101.30:9090/status

Install Grafana:
While Prometheus is storing all of the metrics being collected, we want to be able to visualise these on a Dashboard and that’s where Grafana comes in

So we’ll update Docker Compose to run Grafana

nano docker-compose.yml

volumes:
  grafana-data: {}

services:
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - '3000:3000'
    restart: unless-stopped
    volumes:
      - grafana-data:/var/lib/grafana

Now save and exit

Again, this is a Docker Compose equivalent of what you’ll find on the main website but I’ve added a volume for this container as well to retain any data and it’s configuration settings

As usual, we want the latest version of the image, to give it a name and define the ports for connectivity

We want this to run all of the time, unless we’ve manually stopped it, then mapped the container’s folder to the volume we’ve defined

Then we can tell Docker Compose to run this

docker compose up -d

If you point a web browser to port 3000 of your computer running docker, you should be able to then login to Grafana with the credentials admin/admin

http://192.168.101.30:3000

Once you have logged in, you’ll then be asked to change the password

You can skip this if you like, but it’s always best to change the default credentials

Configure Grafana:
We now need to point Grafana to Prometheus and to do that we add it as a data source so click the button to add your first data source

The first option you’ll see is actually Prometheus so select that

For the server URL, use the same one that you used to check Prometheus from your web browser e.g. http://192.168.101.30:9090

Scroll to the bottom of the page and click Save & Test

Like some other monitoring tools, Grafana requires you to set up your own dashboard, but you can use ones that others have been created and shared and you can look for them in a search engine or directly on the website
https://grafana.com/grafana/dashboards/

Usually I would opt for one that’s being kept up to date but is also in popular use

Once you’ve found what you like, click the button Copy ID to Clipboard

In this case, the one I’ll be using has an ID of 1860

Back in Grafana, in the top left corner click the menu button and select Dashboards

Now click on the New drop-down menu on the right and select Import

Paste in the ID you copied into the Import via grafana.com field then click Load

Change the name of the dashboard if you’d prefer but lower down select the Prometheus data source we created then click Import

Now it will take time to collect more data but you should now have a visual monitoring tool setup

Summary:
Hopefully as you can see it’s not too difficult to setup Prometheus and Grafana once you know how

Granted we’ve only got Prometheus monitoring itself at this stage, so you’ll want to add other devices

For other Linux computers for instance, you can install Node Exporter on them, but as a binary rather than a Docker container

And there are a lot of other exporters available and usually someone has created a useful dashboard to go with it

Now although I haven’t touched on alerts, Prometheus does support these so it can alert you when thresholds have been reached for instance

So this monitoring solution offers a lot of scope

Sharing is caring!