Monitor Proxmox VE With Prometheus And Grafana

In the video below, we show how to monitor Proxmox VE with Prometheus and Grafana

Monitoring computers is very important and even more so for hypervisors like Proxmox VE

Because if something goes wrong it will likely affect all of the virtual computers that are being run on that physical computer

Now an interesting open source combination of monitoring tools that’s available for free is Prometheus and Grafana

And in this video we go over how to configure Prometheus and Grafana to monitor Proxmox VE, when you’re using Docker

Useful links:
https://github.com/prometheus-pve/prometheus-pve-exporter
https://grafana.com/grafana/dashboards/10347-proxmox-via-prometheus

Assumptions:
Because this video is specifically about monitoring Proxmox VE, I’m going to assume that you already have Prometheus and Grafana installed or you know how to set these up

If not then I do have another video which shows you how to install and configure these in Docker

Create Monitoring Account:
Prometheus relies on exporters or agents if you will to make metrics available to it

But as it’s best to avoid installing additional software on platforms like Proxmox VE, we’ll install an exporter in Docker that connects to the nodes using Proxmox’s API

In which case, the first thing to do is to create a user account in Proxmox VE

Navigate to Datacenter | Permissions | Users

Click Add and provide a name e.g. prometheus then click Add

NOTE: There is no option for a password as the assumption is you would create a corresponding account in Linux. As we won’t be using this to login to Linux we’ll ignore that

Navigate to Permissions and from the Add drop-down menu select User Permission

For Path select /

For User select the account just created

For Role select PVEAuditor

Now click Add

Navigate to API Tokens and click Add

For User select the account just created

For Token ID enter something useful e.g. exporter

De-select Privilege Separation otherwise we’ll have to manage permissions for both the user and the token

NOTE: We’re only using the API token for this, and as nobody will login with a username and password there’s no gain that I see in giving one method a different set of rights

For Expire set a practical expiry date, typically 1 month away

Just bear in mind you’ll need to regularly update this, which you can do by extended the expiry date or replacing the token if a breach is suspected

Click Add

Note down the Token ID and Secret

NOTE: While the Token ID we’re shown can easily be re-created, this is the only time that the secret will be shown and it needs to be stored somewhere safe

Once done, close the dialogue box

Install PVE Exporter:
Next we’ll install an exporter as a container in Docker which is also running Prometheus and Grafana

One way to configure Docker is to use Docker Compose so we’ll edit the YAML file

nano docker-compose.yml

services:
  pve-exporter:
    image: prompve/prometheus-pve-exporter
    container_name: pve-exporter
    ports:
      - '9221:9221'
    restart: unless-stopped
    volumes:
      - ./pve/pve.yml:/etc/prometheus/pve.yml

Now save and exit

NOTE: When the video was created, the configuration file for that image version was
/etc/pve.yml
This has since been changed to
/etc/prometheus/pve.yml
While this blog has been updated, the video still shows the original path

Configure PVE Exporter:
I want to keep my container files separated so the first thing we’ll do is to create a folder

mkdir pve

And then we’ll create the configuration file

nano pve/pve.yml

default:
    user: prometheus@pam
    token_name: "exporter"
    token_value: "f25b06c6-871f-4396-913b-2753e8726b4e"
    verify_ssl: false

Now save and exit

This defines the user account and token details to use to connect to the PVE nodes so remember to replace prometheus with whatever username you created

You’ll also need to replace the token name and value with what’s appropriate for you

In addition, pve exporter is being instructed to not verify the SSL certificate. This is necessary unless your PVE nodes have been configured with trusted signed certificates

Now we can start the container

docker compose up -d

Configure Prometheus:
The next thing to do is to configure Prometheus to scrape metrics from the PVE exporter

So we’ll edit the configuration file, which in my case is stored in a folder called prometheus

nano prometheus/prometheus.yml

scrape_configs:
  - job_name: 'pve'
    static_configs:
      - targets:
        - 192.168.102.10  # Proxmox VE node 1
        - 192.168.102.11  # Proxmox VE node 2
        - 192.168.102.12  # Proxmox VE node 3
    metrics_path: /pve
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.102.30:9221

Now save and exit

This configuration has been copied from the developer’s own example and is only slightly altered

Basically, we’re using an exporter to remotely gather metrics from our PVE nodes and the above configuration defines those as targets

As an exporter isn’t on the nodes we have to do some re-labelling

The IP address I’m using for the exporter incidentally is the one for the computer running Docker. This is to keep things simple as we’re using different containers for Prometheus and the exporter

As we’ve changed the configuration file we either need to reload the container or as I’ve configured Prometheus to allow me to reload its configuration I’ll run this command

curl -X POST localhost:9090/-/reload

We now have to wait some time to gather metrics, particularly because Prometheus doesn’t run a scrape immediately, it waits for the scrape time to count down

Once it’s ready, we can check the exporter for metrics from a node, for instance

http://192.168.102.30:9221/pve?target=192.168.102.10

TIP: If you just point your browser at the IP address and port it will tell you to use a URL which includes the target, hence the above

We can also check Prometheus itself

http://192.168.102.30:9090/targets?search=

In this case, I’d expect to see three entries under Endpoints with a state of UP

Install Grafana Dashboard:
While you can create your own Dashboard in Grafana you can also import ones others have created

The developer of pve exporter suggests the following https://grafana.com/grafana/dashboards/10347-proxmox-via-prometheus/

You can pick another if you prefer but in this example click Copy ID to clipboard from the web page

Login to Grafana

In the top left corner click the menu button and select Dashboards

Now click on the New drop-down menu on the right and select Import

Paste the ID you copied into the Import via grafana.com field then click Load

Change the name of the dashboard if you’d prefer but lower down select the Prometheus data source then click Import

NOTE: Although we have more than once instance to choose from in this example dashboard, if you have a cluster the results will be the same no matter which node you select as the exporter is providing cluster wide information

Now it will take time to collect more data but you now have a visual monitoring tool for PVE

Summary:
Hopefully as you’ll see, setting up this monitoring for Proxmox VE is fairly straightforward

At the time of recording, the exporter is still being maintained which is very good to know

But it provides us with a great way to monitor a Proxmox VE cluster

Because aside from centralising all your devices under one monitoring solution, with some additional work you’ll be able to setup alerts within Prometheus to notify you if problems occur

Sharing is caring!