Adventures of a Small Time OpenStack Sysadmin Chapter 048 - Prometheus and Grafana Doubleheader

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 048 - Prometheus and Grafana Doubleheader

Intro

Ironically, I am no longer using Prometheus and Grafana.  Although I got it working, it did not do anything Zabbix did not do, and its certainly not easier to use, and its OpenStack cluster monitoring only whereas Zabbix can monitor anything.  Technically I probably could have converted literally everything on my entire network to use Prometheus/Grafana, um, no I don't have that kind of spare time.  None the less, I got it working, its pretty cool, so I write it up in this blog post.

Installation

There aren't really any OpenStack docs for Prometheus / Grafana, at least that I could find, so this is a mess of google searches and experimentation and software archeology concatenated into something resembling a checklist:

/etc/kolla/globals.d/prometheus.yml needs this line:

enable_prometheus: "yes"

/etc/kolla/globals.d/grafana.yml needs this line:

enable_grafana: "yes"

/etc/kolla/passwords.yml needs the following lines edited:

grafana_admin_password: Make this something sane, the default web UI login will be username admin password "this password"

/root/multinode (or where-ever you keep your kolla-ansible inventory file):

This is where you enable or disable what prometheus will monitor.  The defaults seem fine or at least they worked for me.

Well, lets deploy it and try it out:

kolla-ansible -i ./multinode  reconfigure -t prometheus,grafana

The deployment took about ten minutes to fail.

One multi-page error message was something about telegraf username and password failing.

One single-page long error message was about monasca data source having an invalid username or password.

Well, I tried using it anyway to see what happens with IPMI and similar monitoring, and to my happy surprise it kind of works anyway regardless of the partially failed deployment.

First steps walkthrough with Grafana

This really isn't the time for Grafana training, but here's what I did on my hardware the first time I logged in, which will probably not -exactly- match your hardware but should provide some inspiration.

Web UI for Grafana will be at the URL, your controller ip addrs, port 3000, uname is admin, pword is whatever was in /etc/kolla/passwords.yml

Left side click on dashboards.  "+ New Dashboard"

"Add a new panel"

On the right side, change "Title" from "Panel Title" to "CPU" or similar

Data source is the default dead telegraf.  Change to Prometheus.  Everything underneath the data source selection changes...

Metric: Select "node_hwmon_temp_celsius"

Labels: "chip = platform_coretemp_0" and click the "+" to add "sensor = temp1"

(your hardware is probably different...)

Click "Run Queries" and you get what seems to be the CPU temp of my cluster's hosts.

Click "Apply" in the upper right.

You probably want a better dashboard name than "New dashboard" so when looking at the pretty graph, click the "Dashboard Settings" in the upper right

"General" "Name" change from "New dashboard" to "Temperatures"

Click "Save Dashboard" and "Save"

Now you could spend endless hours making fancier and fancier dashboards, but this is a five minute intro to Grafana and it seems to work well enough.

Conclusion

The problem with me personally using Prometheus and Grafana, is I have a working Zabbix across my entire system (not just this cluster) so I have zero motivation to use Prometheus and Grafana.  I am happy to report the version installed with Kolla-Ansible does mostly work and I provided a walkthrough above of how to start using it to create a dashboard showing the cluster host CPU temperatures.

Tomorrow, the sequel to the old post "Central Logging", it's the all new long awaited "Central Logging Redux"

Stay tuned for the next chapter!