Motivation

I have started adding more pieces into my pet project, and I would like to have some visibility about what is happening and in where.

We can classify monitoring frameworks in two big groups:

Those that actively reports their stats to a central server (like statsd).
Those that expose their metrics, to allow services to collect that data(like prometheus).

I’ve just selected Prometheus, because is widely used and the project is under the umbrella of the Cloud Native Computing Foundation

And also, I like the Grafana dashboards aesthetics :)

Minimal setup

My setup will have two different VMs:

One that will host Prometheus and Grafana, that will store the collected data.
Another VM with a RabbitMQ server (that will be used as a jobs queue).

We have three ways to install Prometheus

Docker image
Official binary
Build from source

I choose the official binary.

Installing Prometheus

There is a fine Digital Ocean blog post to setup prometheus.

I am following the steps from the tutorial except the part of not creating a home dir for it. I prefer prometheus to have its own home to store the binary, and the database.

adduser prometheus --shell /bin/false

I did not gave him a shell, so if we have to perform any action as the prometheus user:

sudo -u prometheus bash

In order to have prometheus running at startup, we can create a starup script, or run it under supervisord or similar.

Running prometheus as a service

(This is copy / paste from the DO article).

Systemd script: /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/home/prometheus/prometheus/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /home/prometheus/tsdb/  \
    --web.console.templates=/home/prometheus/prometheus/consoles \
    --web.console.libraries=/home/prometheus/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl status prometheus

Installing RabbitMQ

The version that comes in the default Debian 9 / Ubuntu Xenial repos are a little bit outdated, so it is better to download the package from the official RabbitMQ downloads page, or directly add the RabbitMQ Debian repository.

Currently, Debian 9 default RabbitMQ package is 3.5.x, while the prometheus exporter client requires at least 3.6.x.

Also, we will need an up to date erlang version (19.x and above), so again, we need to add the APT repository, following the instructions found in the official erlang page.

wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
sudo dpkg -i erlang-solutions_1.0_all.deb

deb https://packages.erlang-solutions.com/ubuntu precise contrib

Install the Erlang APT repo key

wget https://packages.erlang-solutions.com/ubuntu/erlang_solutions.asc
sudo apt-key add erlang_solutions.asc

Install the rabbitmq prometheus exporter

RabbitMQ allows to add plugins, and among them there is the Prometheus RabbitMQ plugin

The plugins must be downloaded (ez files are compiled erlang modules), and saved into the RabbitMQ plugins directory.

In my Debian installation : /usr/lib/rabbitmq/lib/rabbitmq-server-[version]/plugins (or /usr/lib/rabbitmq/plugins, I’m not sure if updating the server and not having the plugins updated would work :/ ).

wget https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/v3.7.0.1/accept-0.3.3.ez
wget https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/v3.7.0.1/prometheus-3.4.4.ez
wget https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/v3.7.0.1/prometheus_cowboy-0.1.4.ez
wget https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/v3.7.0.1/prometheus_httpd-2.1.8.ez
wget https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/v3.7.0.1/prometheus_process_collector-1.3.0.ez
wget https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/v3.7.0.1/prometheus_rabbitmq_exporter-v3.7.0.1.ez

You should be careful when upgrading rabbimq, since those files should need to be updated.

Then we must enable the plugins:

rabbitmq-plugins enable prometheus
rabbitmq-plugins enable prometheus_cowboy
rabbitmq-plugins enable prometheus_httpd
rabbitmq-plugins enable prometheus_rabbitmq_exporter

So now the metrics can be accessed through the Management API, so it makes sense to tell rabbitmq to use https:

/etc/rabbitmq/rabbitmq.config

{rabbitmq_management, [
   {listener, [
     {port, 15672},
     {ssl, true}
   ]}
]}

In rabbitmq we have the guest user, that by default is an admin one, but that is only allowed to make requests from localhost. So we must add a new user:

rabbitmqctl add_user watcher watcherpass
rabbitmqctl set_user_tag watcher monitoring

https://www.rabbitmq.com/management.html

We can test that the data is beeing exported correctly with a command:

curl -k -i -u watcher:watcherpass https://rabbitmq.lh:15672/api/metrics

(We must use the -k option, because we are using a custom signed certificate).

We need a user that can extract the prometheus stats data:

Configure prometheus to get the RabbitMQ exported data

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'rabbitmq'
    static_configs:
      - targets: ['rabbitmq.lh:15672']
    metrics_path: /api/metrics
    scheme: https
    basic_auth:
      username: gachapin
      password: gachapin
    tls_config:
      server_name: 'rabbitmq.lh'
      insecure_skip_verify: true

Run from the commandline:

sudo -u prometheus /home/prometheus/prometheus/prometheus \
     --config.file /etc/prometheus/prometheus.yml \
     --storage.tsdb.path /home/prometheus/tsdb/  \
     --web.console.templates=/home/prometheus/prometheus/consoles \
     --web.console.libraries=/home/prometheus/prometheus/console_libraries

Let’s open the browser, and go to http://watcher.lh:9090/ to see the prometheus web console. We can check that we are collecting the rabbitmq data by entering the rabbitmq_queues.

Add grafana to plot it beautifully

We go to the Grafana downloads page, and again we opt for the option of using their APT repository, so we can easily get the updates.

It tells us to use this, no matter if Ubuntu or Debian. Add the following line to your /etc/apt/sources.list file

deb https://packagecloud.io/grafana/stable/debian/ jessie main

and curl https://packagecloud.io/gpg.key | sudo apt-key add - to add packagecloud key.

We should change the admin user / pass in the /etc/grafana/grafana.ini file.

After installation we want to enable it on startup:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server

After adding the Prometheus data source, we can look for an existing RabbitMQ Grafana Dashboard, and select the one that uses rabbitmq_exporter for prometheus

Some of the graph requires minor changes, like the rabbitmq_connectionsTotal to rabbitmq_connections

Access prometheus only locally

Since we aren’t going to access the prometheus console from outside, we can make it serve the data only to localhost, so we can add the :

--web.listen-addres='127.0.0.1:9090'

So the local grafana interface can still access it. And in case we want to use it, we can still use ssh port forwarding.

Minimal setup done

With this we have the minimal setup to monitor how the rabbitmq queues / server is doing. Next steps would be set up alarms if the queue is growing too much, if we are running out of memory, or if the CPU is having too much work.

Motivation#

Minimal setup#

Installing Prometheus#

Running prometheus as a service#

Installing RabbitMQ#

Install the rabbitmq prometheus exporter#

Configure prometheus to get the RabbitMQ exported data#

Add grafana to plot it beautifully#

Access prometheus only locally#

Minimal setup done#