Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.
Adventures of a Small Time OpenStack Sysadmin Chapter 047 - Zabbix
Intro
Technically Zabbix is not an OpenStack project member or service. However, I find it VERY useful to monitor the hardware and bare metal OS. Also, for consistencies sake, I use Zabbix across my entire system; if it has a Zabbix software agent OR a SNMP port OR an IPMI port, then I use Zabbix to monitor it. Very versatile software!
I run Zabbix inside my OpenStack clusters inside my "infrastructure" project. A Zun container named zabbixserver holds the backend server. There are Zun containers for the zabbixproxy (which is technically no longer necessary...) and zabbixweb frontend. The only persistent state in the monitoring system is in the MySQL server, I have an Ubuntu instance running that holds Docker containers backed by my NAS NFS server. The NAS is backed up via a process that is far off topic for this series. I also have an Adminer Zun container connected to the zabbixmysql database so I can, if I want, use a web UI to poke around, although thats more for fun than productivity.
How to enable IPMI pollers in Zabbix on a Zun OpenStack container
https://www.zabbix.com/documentation/current/en/manual/config/items/itemtypes/ipmi
This is done in a Zun container using a Heat orchestration template by adding the following environment variable to the OS::Zun::Container configuration:
ZBX_IPMIPOLLERS: "1"
See the Gitlab repo:
How to configure an individual IPMI host in Zabbix
Its possible in Zabbix to configure multiple interfaces. Normal hosts only have an "Agent" but all the OpenStack hosts have an IPMI interface, so I added it. Note you have to add the IPMI interface, update, then click on the IPMI tab, what works for me is:
Authentication algorithm: Default
Privilege Level: Admin (change)
Username: LOL NOT SAYIN (but everyone in real life uses ADMIN, of course)
Password: LOL NOT SAYIN
It will take quite awhile to eventually get around to polling.
Add the "Template Server Chassis by IPMI" named template.
Click the Host "Macros" tab and add the following two macros:
{$IPMI.USER}
{$IPMI.PASSWORD}
with the obvious text values and click "Update".
Zabbix Agent2
Agent2 replaces the old agent and adds Docker monitoring and could optionally do some interesting MySQL monitoring. Each OS has a somewhat different way to install Agent2, ranging from FreeBSD where its simply a new package to Ubuntu where you have to download a package manually then manually install, all via a little script I wrote and have Ansible place in the /root directory of every Ubuntu install.
https://www.zabbix.com/documentation/6.2/en/manual/config/templates_out_of_the_box/zabbix_agent2
Zabbix Community Template Collection
https://github.com/zabbix/community-templates
Active Checks Requirements
For active checks to work, MUST configure a host as:
"Host name" the FQDN (example os4.cedar.mulhollon.com
"Visible name" the short name (example os4)
My suggested Zabbix host configuration for monitoring an OpenStack host
Template App NTP Service
Template App SSH Service
Template Module ICMP Ping
Template OS Linux by Zabbix agent
Template Server Chassis by IPMI
SMART by Zabbix agent 2
Host groups: Hypervisors ALSO all are members of OpenStack Cluster 1 or 2
Autodiscovery "Fun" with OpenStack
Autodiscovery is OK before Kolla-Ansible is deployed. After deployment, zabbix will autodiscover all the new LVM drives and network interfaces and ethernet bridges, then perseverate upon them forever as a complete waste of time. I've seen over 1400 items monitored on one host, very slowly, LOL. The plan is to disable autodiscovery AFTER bare metal host prep, but BEFORE kolla-ansible deployment.
"Configuration" "Hosts" click the items for your host. For example for me it was "os4 Items 1351" or whatever it was. Click over on the "Discovery Rules 3". In the "Status" column, all should be "Enabled" click on "Block Devices discovery" and "Network interface discovery" to change them to disabled. I've not run into problems with "mounted filesystem discovery"... so far. Then click back on "Items". One cool feature of a relatively modern Zabbix web UI is you can select multiple tags by clicking on them. If you have "junk" openstack interfaces from instances that no longer exist or it just doesn't matter, click several of the "TAGS" so they are listed below, click the checkbox in the upper left to select all, make sure, then click the "Delete" button. Repeat for all hosts and your Zabbix will run dramatically faster with lower system load.
SMART drive monitoring using Zabbix
https://www.zabbix.com/integrations/smart
Need to be using agent2, which I am doing for Docker support anyway.
Need to "apt-get install smartmontools" for Zabbix SMART monitoring to work.
"chmod u+s /usr/sbin/smartctl" is usually handled via Ansible
Test by logging in to an OpenStack host as root, then "sudo -u zabbix smartctl -a /dev/sda" or similar.
Note there is no default template for SMART although one can be downloaded and installed:
Download the SMART template from the integrations link above
In the Zabbix web UI, "Configuration", "Templates", "Import"
Select the file saved in the step above
Mark the required obvious options in import rules
"Import"
Your list of templates now includes "SMART by Zabbix agent 2"
Add the above template to hosts.
Note this adds an autodiscovery rule.
I've tried setting "Plugins.Smart.Path=/usr/sbin/smartctl" in /etc/zabbix/zabbix_agent2.conf
The only place its documented ANYWHERE that a sudo entry is required for SMART is:
See:
zabbix ALL=(ALL) NOPASSWD:/usr/sbin/smartctl
In:
https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/roles/sudo/files/sudoers
At this point, Zabbix SMART monitoring will work.
Tomorrow, a Prometheus / Grafana doubleheader! Two for the price of one!
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.