Spring City Solutions LLC: November 2023

Wednesday, November 29, 2023

Proxmox VE Cluster - Chapter 017 - Install Proxmox VE on the old OS2 cluster hardware

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Some notes on installing Proxmox VE on the old OS2 cluster hardware. The main difference between installation on OS1 and OS2 is adding NTP serving to OS2, more or less as per Chapter 014 NTP notes. The main reference document:

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html

The plan to work around the networking challenges is to get everything working on a single plain temporary 1G ethernet connection, then use that as a management web interface to get the dual 10G LAG with VLANs up and running, then use the new "20G" ethernet connected web management interface to connect and reconfigure the dual 1G LAG / VLAN ethernet ports, at which point everything will be working.

First Steps

Install using IPMI KVM and USB key, so find the usb key and plug in the IPMI ethernet.

On boot, DEL for setup, alter the boot options to include the USB drive, reboot, hit F11 for boot menu, then boot off the USB.

Proxmox "OS" install process

I have a habit of using the console install environment.
Installer wants to default to install on the M2 drive although I am using the SATA.
Country: United States
Timezone: The "timezone" field will not let me enter a timezone, only city names, none of which are nearby. Super annoying I can't just enter a timezone like a real operating system. I ended up selecting a city a thousand miles away. This sucks. Its a "timezone" setting not "name a far away city that coincidentally is in the same timezone". I expect better from Proxmox.
Keyboard Layout: U.S. English
Password: (mind your caps-lock)
Administrator email: vince.mulhollon@springcitysolutions.com
Management Interface: the first 1G ethernet (eno1, aka the "bottom left corner")
Hostname FQDN: as appropriate, as per the sticker on the device
IP address (CIDR): as appropriate, as per the sticker on the device / 016
Gateway address: 10.10.1.1
DNS server address: 10.10.8.221
Note you can't set up VLANs in the installer, AFAIK.
Hit enter to reboot, yank the USB flash install drive, yank the USB keyboard, watch the monitor... seems to boot properly...
Web interface is on port 8006. Log in as root. Note I installed 8.0-2 and on the first boot, the web gui reports version 8.0.3, it must have auto-updated as part of the install process?

Upgrade the new Proxmox VE node

Double check there's no production workload on the server; its a new install there shouldn't be anything, but its a good habit.
Select the "Server View" then node name, then on the right side, "Updates", "Repositories", disable both enterprise license repos. Add the community repos as explained at https://pve.proxmox.com/wiki/Package_Repositories
Or in summary, click "add", select "No-subscription", "add", then repeat for the "Ceph Quincy No-Subscription" repo.
In right pane, select "Updates" then "Refresh" and watch the update. Click "Upgrade" and watch the upgrade.
Optimistically get a nice message on the console of "Your system is up-to-date" and a request to reboot.
Reboot and verify operation.

Install hardware in permanent location with temporary ethernet cables

Perform some basic operation testing
In the web UI "Shutdown" then wait for power down.
Reinstall in permanent location.
Connect eno1 to any untagged "Prod" VLAN 10 access-only ethernet port, temporarily, for remote management via the web interface.
Connect the 10G ethernets eno3 and eno4 to the LAG'd and VLAN'd 10G ethernet switch ports.

Move the Linux Bridge from single 1 gig eno1 to dual 10 gig LAG on eno3 and eno4

You are going to need this:

https://pve.proxmox.com/wiki/Network_Configuration

Modify eno3 and eno4, checkmark "Advanced", change MTU to 9000.
Create a Linux Bond named bond1, Checkmark "Advanced", change MTU to 9000, Mode "balance-xor", slaves "eno3 eno4" (note space in between, not comma etc). Note bond0 will eventually be the 1G LAG, and the old OpenStack used "balance-xor" so I will start with that on the Proxmox.
Create Linux VLAN named bond1.10 with MTU 9000, can create the other VLANs now if you want.
Edit vmbr0 Linux bridge to have a MTU of 9000 and Bridge Ports of bond1.10
Double check everything then "Apply Configuration", and after about twelve to thirteen heart stopping seconds it should be up and working.

At some later date I will try some LAG bond modes more interesting than "balance-xor".

Note the network interfaces do not have "VLAN aware" checked. Everything works. I will research this later in a dedicated advanced networking post.

Convert the single 1 gig eno1 to dual 1 gig LAG on eno1 and eno2

Edit eno1 and eno2 and set MTU to 9000
Create a Linux Bond named bond0, Checkmark "Advanced", change MTU to 9000, Mode "balance-xor", slaves "eno1 eno2" (space in between).
Create VLAN interfaces now on bond0, or create them later.

Configure NTP

Create (or copy) the files for sources into "/etc/chrony/sources.d" I put exactly one clock in each file. Files in sources.d can be re-read without restarting the entire service by running "chronyc reload sources". If successful you should see the other clocks are now accessible when running "chronyc sources".
Remove the default clocks shipped by Proxmox and enable NTP serving. Edit /etc/chrony/chrony.conf and comment out the "pool" directive and add a line underneath "allow 10.0.0.0/8" This will require a service restart not a mere reload, so "service chrony restart" and verify Chrony operation after a few minutes using "chronyc sources"
Edit DNS for ntp4 (or as appropriate) to point to the new proxmox node IP address.
Edit NTP on ALL THE OTHER NODES to reflect the presence of this new NTP server.
Test NTP from various nodes, VMs, and hardware to verify NTP is working.

Final Installation Tasks

Join the new node(s) to the existing cluster. In "Datacenter" on any cluster member, "Cluster" "Join Information" "Copy Information" cut and paste into "Datacenter" on the new node, "Join Cluster", enter the peer's root password, "Join 'Proxmox'". Will have to log back into the web UI after the SSL certs update...
Verify information in Netbox to include MAC, serial number, ethernet cabling, platform should be Proxmox VE, remove old Netbox device information.
Add new hosts to Zabbix.

The next post will be about adding the remaining "paused" workload to the now "full sized" Proxmox VE cluster.

Monday, November 27, 2023

Proxmox VE Cluster - Chapter 016 - Hardware Prep Work on the OS2 Cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

This will be similar to Chapter 012 although different hardware.

These microservers are three old SuperMicro SYS-E200-8D that were used for Homelab workloads. They will become Proxmox cluster nodes proxmox004, proxmox005, and proxmox006.

This server hardware was stereotypical for a late 2010's "VMware ESXi Eval Experience"-licensed cluster, and later worked very well under OpenStack. 1.90 GHz Xeon D-1528 with six cores and 96 GB of ram, 1 TB SATA SSD for boot and local storage, new 1 TB M2 NVME SSD for eventual CEPH cluster storage.

Hardware reliability history

Proxmox004 had its AC power brick replaced 2022-07-10

Proxmox005 had a NVME failure 2020-05-24, took advantage of that outage to also upgrade its SSD to a new 1TB (on suspicion, the old one was working fine although wearout measurement was getting to a high percentage per SMART reports) on 2020-05-27.

Proxmox006 had a NVME failure 2021-02-20, and had its AC power brick replaced 2022-06-10

Previously in Chapter012 I claimed that 5/6 of the power supplies had failed on my E800 microservers, but I made a mistake, and it seems "only" TWO THIRDS of the power supplies have failed as of late 2023, currently proxmox001 and proxmox005 are still running on the original mid 2010s power supplies. I will keep a close eye on the output voltages (monitorable via IMPI using Observium and probably Zabbix and maybe somehow via Elasticsearch)

FIVE Ethernet ports

Even the official manufacturer's operating manual fails to explain the layout of the five ethernet ports on this server. Looking at the back of the server, the lone port on the left side is the IPMI, then:

eno1 1G ethernet bottom left corner, 9000 byte MTU

eno2 1G ethernet top left corner, 9000 byte MTU

eno3 10G ethernet bottom right corner, 9000 byte MTU

eno4 10G ethernet top right corner, 9000 byte MTU

eno1 and eno2 are combined into bond12, which uses balance-xor mode to provide 2 GB of bandwidth.

eno3 and eno4 are combined into bond34, which uses balance-xor mode to provide 20 GB of bandwidth. 20 GB ethernet is pretty fast!

I run the VLANs as subinterfaces of the bond interfaces. So, "Production" VLAN 10, has an interface name of "bond34.10"

Hardware Preparation task list

Clean and wipe old servers, both installed software and physical dusting.
Relabel ethernet cables and servers.
Update port names in the managed Netgear ethernet switch. VLAN and LAG configs remain the same, making installation "exciting" and "interesting".
Remove monitoring of old server in Zabbix.
Verify IPAM information in Netbox.
Test and verify new server DNS entries.
Install new 1TB M.2/NVME SSDs.
Replace old CMOS CR2032 battery as it's probably 5 to 7 years old. This is child's-play compared to replacing the battery on a hyper-compact Intel-NUC.
Reconfigure the BIOS in each server. For a variety of reasons, PXE netboot requires UEFI and BIOS initialization of the network, so I used that in the OpenStack era which was installed on top of Ubuntu. However, I could not force the UEFI bios to boot the SATA SSD it insisted on booting the M.2 only, which is odd because it worked fine under older, USB-stick installed Ubuntu. Another problem with the BIOS config was "something" about pre-initializing the ethernet system for PXEBoot messes up the bridge configuration on Proxmox's Debian OS, resulting in traffic not flowing; I experimented with manually adding other interfaces to the bridge; no go; symptoms were no packets flowing in (brctl showmac is essentially the bridge's ARP table) also no packets out, although link light up and everything looks OK. Anyway, in summary, disable PXEboot entirely and convert entirely from UEFI to Legacy BIOS booting. This was typical of the UEFI experience in the late 2010s, it doesn't really work most of the time, but Legacy BIOS booting always works. Things are better now.

Next post will be about installing Proxmox VE on the old OS2 cluster hardware.

Friday, November 24, 2023

Proxmox VE Cluster - Chapter 015 - Migrate workload of the old OS2 cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

I plan to reuse the OpenStack cluster OS2 hardware to increase Proxmox cluster capacity. Before I can reuse the hardware, I need to move all remaining workload off the OpenStack hardware. The OS2 workload is either immediately migrated to Proxmox if its important enough, or "paused" for a few days and restored on Proxmox after the cluster work is done. Here is a list of the OS2 cluster workload, and a description of what I did to each application:

storage

An Ubuntu Samba and NFS fileserver.

Converted from FreeBSD to Ubuntu. While doing that, I added NFS file serving for Ubuntu on Ansible, so that is scripted and automated now.

netbox

https://netbox.dev/

A complete FOSS IPAM system.

This one was tricky, its a docker-compose with local volumes so need to backup the DB on the old system then restore onto the new system, more or less. I need to completely redesign this so all data is stored over NFS instead of on local host volumes. Its the only container I have that stores data in local volumes which makes management a hassle.

I recommend against directly following the database restore instructions on:

https://github.com/netbox-community/netbox-docker/wiki/Troubleshooting#database-operations

I unfortunately have extensive experience with restoring an older schema on top of a freshly installed empty new schema, resulting in considerable data loss. At least I keep good backups LOL, so I was able to recover from that. A better strategy is NOT to start everything then shut off the client processes, then (try to) restore an old schema backup over an empty new schema as per existing online docs, but instead to start ONLY the postgres container, then while it's empty, restore the old schema backup into postgres, then and only then start up the client containers (everything else) and let the automatic upgrade process upgrade the schema of the freshly restored old schema database. That strategy worked perfectly with no data loss.

I filed a docs improvement bug regarding the above adventure at GitHub:

https://github.com/netbox-community/netbox-docker/issues/1113

unifi

The "unifi" controller software in a docker container for a cloudy-ish WIFI network based on Ubiquiti hardware.

The Unifi Controller is just another Docker container. However, a new controller IP address means re-homing all Unifi devices off the old server and old IP address and onto the new server with it's new IP address. I copied the NFS mounted volume for unifi controller over to unificontroller-old, because then I could start controllers on both the new and old servers. Obviously, every Ubiquiti hardware device on the LAN reconnected to the unifi-old server, although the new unifi server looked fine (other than acting like all devices were suddenly disconnected from it, which is accurate). Then on the old controller, in "Settings" "System" "Advanced" there's an "inform host" setting which had the old server's IP address, so I put in the new address and hit apply. There's also a way to manually SSH into each device individually, which can be a bit of a pain, so I used the web UI "Inform" method.

The above resulted in a minor problem, the "inform host" on the new controller was pointing to the old controller IP address because the new controller was a clone of the old controller, and the "inform host" on the old controller was pointing to the new host, so the devices ping-pong-ed back and forth between the old and new controllers for awhile. I fixed the "inform host" setting on the new controller and Wi-Fi devices started coming online. Cool. The ethernet switches were mad at me for at least several minutes, I think I crashed some firmware or something, although they did eventually ALL come online. Mildly interesting that the Wi-Fi devices connect much faster than the ethernet switches. In summary, that was exciting for awhile, but in the end it all worked pretty well.

redmine

https://www.redmine.org/

A FOSS project management suite

Shut down the docker-compose on the old server, start it on the new server, painless. All the data volumes reside on the same NAS NFS server, this was just moving the docker "compute host" from the old OpenStack cluster number two, to the new Proxmox docker "compute host".

es02, es03, kibana

https://www.elastic.co/

Elasticsearch infrastructure for syslog storage and analysis

This turned into a larger adventure that initially planned. Ended up being an upgrade of Elasticsearch from 8.6 to 8.10 and a new cluster being formed on ES01, ES02, and ES03. Later I will add ES04, ES05 and ES06. This seems like a lot of work to store syslog messages, but Elasticsearch as a database technology is fun to play with and Kibana can make really cool graphical dashboards, so its worth the effort.

portainer

https://www.portainer.io/

A FOSS centralized web based Docker management tool.

Shut down the docker container on the old server, start it on the new server, painless.

guacamole

https://guacamole.apache.org/

The Apache Guacamole project provides a website that turns any web browser into a SSH client or RDesktop client.

Shut down the old docker container, start on the new server, painless.

dc21, dc22 to dc02, dc03

https://www.samba.org/

Samba Active Directory Domain Controller Cluster.

Need to remove dc21 and dc22 as the second to last VMs removed from OS2, some VMs on OS2 will point to dc21 and dc22 for DNS resolution.

Probably the only "interesting" thing to remember to do was move the FSMO roles to dc01 off of dc21.

I took this opportunity to clean up the old DNS entries. I use the RSAT tools on a windows 11 desktop, works pretty well to control Samba Active Directory.

dns21, dns22 to dns02, dns03

Ubuntu servers doing DNS resolving requests that are forwarded from the Domain Controller cluster.

dc01 forwards DNS resolution to dns01 and dns02, dc02 forwards to dns02 and dns03, and so forth, so everything has multiple backups. This works pretty well.

Need to remove dns21 dns22 after removing dc21 dc22. These will be the last VMs removed from OS2.

emby

https://emby.media/

A media server and DVR for Roku and other household set top boxes.

Just an Ubuntu server with emby package installed.

Need to convert when the recordings list is empty

Note there is, of course, a new IP addrs.

Emby has a very elaborate and detailed manual provisioning process, documented in it's Redmine runbook "issue".

ubuntu

General end user use.

Move old ubuntu to ubuntu-old in DNS

Set up a new Ubuntu for enduser use.

Make sure Ansible runs on the new ubuntu before shutting down the old ubuntu.

backup

An Ubuntu NFS and Samba fileserver holding backup data.

This was a test fileserver before converting "storage2" to "storage". Only problem I ran into was minor, FreeBSD prefers the use of the group "wheel" whereas Ubuntu prefers the use of the group "root".

Next blog post, there's nothing left on OpenStack cluster 2, shut it down and prepare the OS2 cluster hardware for reuse as additional Proxmox capacity.

Wednesday, November 22, 2023

Proxmox VE Cluster - Chapter 014 - Configure NTP on the Proxmox Cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

The NTP architecture here uses the E800 nodes as a central NTP source for the entire network. Those six nodes are the only NTP servers that get time from the internet or from the local GPS refclock, and the rest of the network syncs time off those six. Helpfully, a long time ago, I set up DNS aliases ntp1 thru ntp6 for these clocks so I don't need to change any configurations after I alter the DNS ... probably.

Basically, today I am converting from classic NTP on the OpenStack servers to Chrony on the Proxmox servers.

References

https://pve.proxmox.com/wiki/Time_Synchronization

https://ubuntu.com/server/docs/how-to-serve-the-network-time-protocol-with-chrony

The Big Picture Plan

Configure proxmox001-003 to get time from the local GPS clock, from the other proxmox servers, and one internet time pool source.
Configure proxmox001-003 to serve time.
Modify dns such that ntp1-ntp3 will now point to proxmox001-003. Note some devices will require manual configuration such as the Ethernet switches, maybe the Ubiquity wifi, maybe the TrueNAS, who knows?
After proxmox004-006 are set up, the DNS hosts ntp4-ntp6 will need to be updated.

I will set this up manually because its simple and I have not integrated proxmox with Ansible yet. But eventually Proxmox will be configured via Ansible.

Manually configuring chrony on Proxmox VE

Create (or copy) the files for sources into "/etc/chrony/sources.d" I put exactly one clock in each file. Files in sources.d can be re-read without restarting the entire service by running "chronyc reload sources". If successful you should see the other clocks are now accessible when running "chronyc sources".
Remove the default clocks shipped by Proxmox and enable NTP serving. Edit /etc/chrony/chrony.conf and comment out the "pool" directive and add a line underneath "allow 10.0.0.0/8" This will require a service restart not a mere reload, so "service chrony restart" and verify Chrony operation after a few minutes using "chronyc sources"
Edit DNS for ntp1 (or as appropriate) to point to the new proxmox node IP address.
Test NTP from various VMs and hardware to verify NTP is working.

List of clocks in /etc/chrony/sources.d:

gpsclock.sources = the local, on LAN, "stratum 1-ish" GPS clock
proxmox001.sources = should be five files pointing to the other five E800 nodes
pool.sources = "server 0.pool.ntp.org" as an external reference.

Cool, the new Proxmox nodes are now providing NTP time service to the network. Next blog post will be about moving all the workload off the old OpenStack OS2 cluster so as to repurpose that hardware as yet more Proxmox capacity.

Monday, November 20, 2023

Proxmox VE Cluster - Chapter 013 - Install Proxmox VE on the old OS1 cluster hardware

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Some notes on installing Proxmox VE on the old OS1 cluster hardware. The main reference document:

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html

First Steps

Install using IPMI KVM and USB key, so find the usb key and plug in the IPMI ethernet.

On boot, DEL for setup, alter the boot options to include the USB drive, reboot, hit F11 for boot menu, then boot off the USB.

Proxmox "OS" install process

I have a habit of using the console install environment.
Installer wants to default to install on the M2 drive although I am using the SATA.
Country: United States
Timezone: The "timezone" field will not let me enter a timezone, only city names, none of which are nearby. Super annoying I can't just enter a timezone like a real operating system. I ended up selecting a city a thousand miles away. This sucks. Its a "timezone" setting not "name a far away city that coincidentally is in the same timezone". I expect better from Proxmox.
Keyboard Layout: U.S. English
Password: (mind your caps-lock)
Administrator email: vince.mulhollon@springcitysolutions.com
Management Interface: the first 1G ethernet (eno1, aka the "bottom left corner")
Hostname FQDN: as appropriate, as per the sticker on the device
IP address (CIDR): as appropriate, as per the sticker on the device / 016
Gateway address: 10.10.1.1
DNS server address: 10.10.8.221
Note you can't set up VLANs in the installer, AFAIK.
Hit enter to reboot, yank the USB flash install drive, yank the USB keyboard, watch the monitor... seems to boot properly...
Web interface is on port 8006. Log in as root. Note I installed 8.0-2 and on the first boot, the web gui reports version 8.0.3, it must have auto-updated as part of the install process?

Upgrade the new Proxmox VE node

Double check there's no production workload on the server; its a new install there shouldn't be anything, but its a good habit.
Select the "Server View" then node name, then on the right side, "Updates", "Repositories", disable both enterprise license repos. Add the community repos as explained at https://pve.proxmox.com/wiki/Package_Repositories
Or in summary, click "add", select "No-subscription", "add", then repeat for the "Ceph Quincy No-Subscription" repo.
In right pane, select "Updates" then "Refresh" and watch the update. Click "Upgrade" and watch the upgrade.
Optimistically get a nice message on the console of "Your system is up-to-date" and a request to reboot.
Reboot and verify operation.

Install hardware in permanent location with temporary ethernet cables

Perform some basic operation testing
In the web UI "Shutdown" then wait for power down.
Reinstall in permanent location.
Connect eno1 to any untagged "Prod" VLAN 10 access-only ethernet port, temporarily, for remote management via the web interface.
Connect the 10G ethernets eno3 and eno4 to the LAG'd and VLAN'd 10G ethernet switch ports.

Move the Linux Bridge from single 1 gig eno1 to dual 10 gig LAG on eno3 and eno4

You are going to need this:

https://pve.proxmox.com/wiki/Network_Configuration

Modify eno3 and eno4, checkmark "Advanced", change MTU to 9000.
Create a Linux Bond named bond1, Checkmark "Advanced", change MTU to 9000, Mode "balance-xor", slaves "eno3 eno4" (note space in between, not comma etc). Note bond0 will eventually be the 1G LAG, and the old OpenStack used "balance-xor" so I will start with that on the Proxmox.
Create Linux VLAN named bond1.10 with MTU 9000, can create the other VLANs now if you want.
Edit vmbr0 Linux bridge to have a MTU of 9000 and Bridge Ports of bond1.10
Double check everything then "Apply Configuration", and after about twelve to thirteen heart stopping seconds it should be up and working.

At some later date I will try some LAG bond modes more interesting than "balance-xor".

Note the network interfaces do not have "VLAN aware" checked. Everything works. I will research this later in a dedicated advanced networking post.

Convert the single 1 gig eno1 to dual 1 gig LAG on eno1 and eno2

Edit eno1 and eno2 and set MTU to 9000
Create a Linux Bond named bond0, Checkmark "Advanced", change MTU to 9000, Mode "balance-xor", slaves "eno1 eno2" (space in between).
Create VLAN interfaces now on bond0, or create them later.

Final Installation Tasks

Join the new node(s) to the existing cluster.
Verify information in Netbox to include MAC, serial number, ethernet cabling, platform should be Proxmox VE, remove old Netbox device information.
Add new hosts to Zabbix.

The next post will be about setting up NTP on the new Proxmox VE cluster.

Friday, November 17, 2023

Proxmox VE Cluster - Chapter 012 - Hardware Prep Work on OS1 cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

These microservers are three old SuperMicro SYS-E200-8D that were used for Homelab workloads. They will become Proxmox cluster nodes proxmox001, proxmox002, and proxmox003.

Hardware reliability history

Proxmox001 is the only server out of six that is still running the original AC power supply. The other five required replacement. Voltage would sag lower and lower until there were random reboots under heavy load, and eventually the supplies would fail completely. Thankfully its just a dead power supply and the rest of the hardware has been extremely reliable. I can't recommend SuperMicro hardware enough, its really good stuff... other than the power supplies from the late 2010s.

Proxmox002 had its AC power brick replaced 2020-11-19 and AGAIN on 2023-07-01

Proxmox003 had a NVME failure 2021-02-20, AC power brick replaced 2022-05-31

FIVE Ethernet ports

eno1 1G ethernet bottom left corner, 9000 byte MTU

eno2 1G ethernet top left corner, 9000 byte MTU

eno3 10G ethernet bottom right corner, 9000 byte MTU

eno4 10G ethernet top right corner, 9000 byte MTU

eno1 and eno2 are combined into bond12, which uses balance-xor mode to provide 2 GB of bandwidth.

eno3 and eno4 are combined into bond34, which uses balance-xor mode to provide 20 GB of bandwidth. 20 GB ethernet is pretty fast!

I run the VLANs as subinterfaces of the bond interfaces. So, "Production" VLAN 10, has an interface name of "bond34.10"

Hardware Preparation task list

Clean and wipe old servers, both installed software and physical dusting.
Relabel ethernet cables and servers.
Update port names in the managed Netgear ethernet switch. VLAN and LAG configs remain the same, making installation "exciting" and "interesting".
Remove monitoring of old server in Zabbix.
Verify IPAM information in Netbox.
Test and verify new server DNS entries.
Install new 1TB M.2/NVME SSDs.
Replace old CMOS CR2032 battery as it's probably 5 to 7 years old. This is child's-play compared to replacing the battery on a hyper-compact Intel-NUC.
Reconfigure the BIOS in each server. For a variety of reasons, PXE netboot requires UEFI and BIOS initialization of the network, so I used that in the OpenStack era which was installed on top of Ubuntu. However, I could not force the UEFI bios to boot the SATA SSD it insisted on booting the M.2 only, which is odd because it worked fine under older, USB-stick installed Ubuntu. Another problem with the BIOS config was "something" about pre-initializing the ethernet system for PXEBoot messes up the bridge configuration on Proxmox's Debian OS, resulting in traffic not flowing; I experimented with manually adding other interfaces to the bridge; no go; symptoms were no packets flowing in (brctl showmac is essentially the bridge's ARP table) also no packets out, although link light up and everything looks OK. Anyway, in summary, disable PXEboot entirely and convert entirely from UEFI to Legacy BIOS booting. This was typical of the UEFI experience in the late 2010s, it doesn't really work most of the time, but Legacy BIOS booting always works. Things are better now.

In the next post, we install Proxmox VE on the old OS1 cluster hardware. It'll be interesting with all those VLANs and LAGs.

Wednesday, November 15, 2023

Proxmox VE Cluster - Chapter 011 - Move OpenStack cluster 1 workload to Proxmox

Proxmox VE Cluster - Chapter 011 - Move OpenStack cluster 1 workload

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Before the hardware used for OpenStack cluster OS1 can be repurposed for the Proxmox cluster, I need to move all the virtual machines and containers off OS1. There are several options: temporarily delete them until there is more capacity, permanently delete them if no longer needed, or move to the Proxmox cluster.

The old "warm backup" availability strategy for OpenStack was some workload was installed on both clusters, but only operating on one cluster at a time, for example, one of the minor file servers. It was expensive to keep two copies around of "everything" and only running one copy on the Proxmox cluster should save quite a bit of capacity, overall.

Here is a list of workload I moved to Proxmox:

netbootxyz

Netboot.xyz provides network booting infrastructure. Network booting starts with a DHCP server like ISC-DHCP (Or KEA...) pointing a booting PC to a TFTP address, the address of the netboot.xyz server. This is where Netboot.xyz comes into the picture, it serves a really nice CLI menu of dozens of operating system install ISO files. A very convenient way to install an OS. There are also plenty of testing and troubleshooting images available.

https://netboot.xyz/

The VM is a simple Ubuntu 20.04 install that runs Docker. I NFS mount all my Docker volumes, this has worked well for several years. The move was uneventful. Shut down the old VM on the OS1 cluster, start the new VM (on a new address) on the Proxmox cluster, run a script I keep in the NFS mounted docker directory to pull and start a netboot.xyz container, repoint the DHCP servers to the new netboot.xyz ip address, and it just works.

wiki

This is a Docker container of DokuWiki. I use it as a "home page" or "phone book" for the LAN. If its a web-accessible server, it has a link to it on the wiki.

https://www.dokuwiki.org/dokuwiki

This is another simple Ubuntu VM holding a Docker container, much like the netbootxyz VM above. One of many advantages of storing my Docker volumes over NFS is a move like this is so simple; shut down the Docker container on the old server, start the Docker container on the new server, done. The move was uneventful.

dhcp11, dhcp12, dhcp21, dhcp22 all replaced by dhcp01, dhcp02

This is a classic dual server ISC-DHCPD cluster.

https://www.isc.org/dhcp/

On the OS1 cluster this was running FreeBSD and converted this to Ubuntu 20.04. This conversion was uneventful. I am aware ISC DHCP is discontinued as of 2022, and KEA is the next generation of ISC supported DHCP servers. Will convert to KEA later, stay tuned for a Spring City Solutions Youtube channel video about that conversion process.

dc11, dc12 replaced by dc01

This LAN uses Samba servers as Active Directory Domain Controllers. Really nice to have network access from any machine to my home directory, and SSO is also pretty cool.

https://www.samba.org/

This was also a conversion from FreeBSD Samba (which is pretty easy to use) to Ubuntu Samba (which is definitely not as easy to use). After the initial OS install and Ansible configuration, Do not do a "net ads join -U administrator" during the Ansible process. The Ubuntu samba-tool utility has to create a fresh smb.conf file from scratch during the joining process, so just move the Ansible provided file out of the way temporarily (the Ansible file should be identical other than configuring DNS forwarder servers). After initial configuration you will have to manually edit (or use ansible) to fix the DNS forwarder entry in /etc/samba/smb.conf. You can join a Ubuntu DC to a domain with no error messages while running the "user" set of samba services SMBD, NMBD, Winbond and there will be zero error messages aside from "samba-tool drs showrepl" command failing to connect to port 135, and of course the DC not working in general. It seems that on Ubuntu, you need to shut down the user class daemons SMBD, NMBD, and WINBOND using systemctl, then look up how systemd permanently shuts down samba-ad-dc in order to figure out how to "unmask" that service. The next Ubuntu Samba related problem is systemd-resolved is autoconfigured to start on port 53 before trying to start Samba while refusing external connections, and samba-ad-dc will successfully start and not output any error messages while failing to bind to port 53, so in summary by default domain controller authoritative DNS will fail to work. Systemd is always so annoying to use and just makes everything harder. The solution is to "systemctl stop" and systemctl mask" the "systemd-resolved.service". A final minor problem, or surprise, is I couldn't get replication working without a reboot. Yes I know its on a fifteen minute (or so) timer but it just wouldn't start without a reboot. Believe it or not, on FreeBSD, none of this drama is necessary. Regardless of extensive effort required to work around systemd, Samba worked eventually.

dns11, dns12 replaced by dns01

These are typical ISC Bind DNS servers. Samba Active Directory Domain Controllers do not have a very advanced resolver, so I forward DNS queries for everything they're not authoritative for to a resolver cluster. I've always preferred a DNS architecture that keeps authoritative DNS and DNS resolving separate.

https://www.isc.org/bind/

The VM is a simple Ubuntu server acting as a DNS resolver (not doing authoritative DNS). The conversion was uneventful. Active directory domain controllers DC11 and DC12 use DNS11 and DNS12 so I could not shut down and remove DNS11 and DNS12 until after removing DC11 and DC12.

observium

Observium is a SNMP-based monitoring system. Mostly monitors my Netgear ethernet switches, but can monitor other devices.

https://www.observium.org/

The VM is another simple Ubuntu Docker server. I had to expand the memory to 8 GB and expand the hard disk in Proxmox (then expand the LVM PV, then the LVM LV, finally expand the FS) to 32 GB and expand the CPU allocation to dual core. Starts up slow, but works.

zabbix

Zabbix is an agent-based detailed monitoring system. All VMs and some physical hardware run the zabbix agent for advanced monitoring and trend analysis.

https://www.zabbix.com/

The Zabbix system has two VMs. In the past I've run into weird problems with the Zabbix trying to monitor itself. One way around that is Zabbix has a proxy/concentration server, so I install one of those, and it can connect to the main Zabbix server. Probably no longer necessary, but it is pretty cool.

The zabbix VM is yet another simple Ubuntu Docker server. This was a bit of a headache. I ran into some kind of incompatibility between old and new MySQL versions. Would have impacted me upon next container upgrade had I not moved to a new cluster. Ended up doing a complete Zabbix reinstall, there are too many new features to make the backup useful, etc. In the end after some labor, this works pretty well.

The zabbixproxy VM, again, yet another simple Ubuntu Docker server, holds the Docker container for zabbix-proxy. The move from the OS1 cluster was uneventful.

zerotier

ZeroTier is a complete VPN solution. Pretty cool!

https://www.zerotier.com/

Uneventful reinstallation. Ended up creating a new connection to Zerotier then reroute traffic for the LAN to the new connection (because for awhile I was running both in parallel for testing purposes, which also necessitated some LAN IP address and static route juggling). Also updated DNS to point to dc01 (and later on, added dc02 and dc03, although they don't exist yet).

Next blog post will be about hardware prep work to turn the old OpenStack OS1 cluster into more Proxmox nodes.

Monday, November 13, 2023

Proxmox VE Cluster - Chapter 010 - OS Images on Proxmox and manual VM operations

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Today is about manually testing and experimenting with the Proxmox VE cluster.

References for this post:

https://pve.proxmox.com/wiki/Qemu/KVM_Virtual_Machines

Load some ISO images into the shared Proxmox Cluster:

Download some ISO files, such as Ubuntu 20.04.6 LTS from

https://ubuntu.com/download/server

Add the ISO install image to Proxmox:

Log into the Proxmox web UI
Select any cluster node
Expand in the left pane
Select proxmox-isoimages as a storage location
Click "ISO Images" on the right pane
"Upload"
Wait for the pop up window to report "TASK OK" then close the window out.

Verify the ISO image uploaded using the CLI on any Linux box.

Previously when I set up the NFS shares I added symlinks from my NFS home directory to the proper automounter location. So ~/proxmox-isoimages is symlinked to /net/freenas/mnt/freenas-pool/proxmox-isoimages. This means I can log into any Linux system on the LAN (Via SSO using Samba and Active Directory) and "cd ~/proxmox-isoimages"
The "~/proxmox-isoimages" directory contains "template"
The "~/proxmox-isoimages/template" directory contains "iso"
The "~/proxmox-isoimages/template/iso" directory contains the Ubuntu Server install ISO uploaded previously.

Verify the ISO image uploaded using the Web UI.

Log into any Proxmox cluster node web UI.
In the left pane select "Folder View" then "Storage" then any of the node's "proxmox-isoimages".
In the right pane select "ISO Images" on the left side, and the right side of the right pane should populate with a list of ISO images identical to the directory contents seen in the previous verification task.

Create a test VM and experiment with it.

Select any node. Click "Create VM".
In general pane, Name: Test, probably want to check "Start at boot" for most VMs.
In OS pane, Storage: "proxmox-isoimages", ISO image: select the Ubuntu dropdown.
In System pane, default for everything EXCEPT checkmark "Qemu Agent".
In Disks pane, Storage: proxmox-diskimages, Disk size: 16 GB.
In CPU pane, default single core seems OK?
In Memory, default 2GB seems OK?
In Network, default bridge seems OK, should get a DHCP address.
Select the test VM in the left pane, click "Start" button.
Click on the console and install the operating system.

Note that for Ubuntu 20.04, the default LVM config will only use about half the PV, have to modify the LV to use most of the disk rather than leaving most of it unused.

After the install I disconnect the media from the virtual cdrom although I leave the virtual cdrom drive in place for possible later use.

Verify disk image in CLI

Previously when I set up the NFS shares I added symlinks from my NFS home directory to the proper automounter location. So ~/proxmox-diskimages is symlinked to /net/freenas/mnt/freenas-pool/proxmox-diskimages. This means I can log into any Linux system on the LAN (Via SSO using Samba and Active Directory) and "cd ~/proxmox-diskimages
The "~/proxmox-isoimages" directory contains "images"
The "~/proxmox-isoimages/images" directory contains a directory with the VM ID number, probably 100 for your first VM.
The "~/proxmox-isoimages/images/100" directory contains the qcow2 disk image, probably named something like "vm-100-disk-0.qcow2" As near as I can tell it's thin provisioned, I allocated sixteen gigs at the time of VM creation, but I'm using maybe six or so gigs per VM image.

Verify disk image in Web UI

Log into the Proxmox web UI, any cluster member is fine.
Left pane, "Folder View", "Datacenter", "Storage", select any host's copy of "proxmox-diskimages"
Right pane, "VM Disks", see the list of disks.

Two interesting things to note about VM disk storage in Proxmox, at least in Proxmox version 8:

Configure a VM's disk image as 16 gigs, it thin-provisions on the NFS server using perhaps six or so gigs, it displays in the Web UI in "Storage" as exactly 17.18 GB.
You CAN NOT migrate storage, or what VMware would call "Storage vMotion" in the web UI from "Folder View" "Datacenter" "Storage" "Any shared NFS mount". However, you CAN migrate storage from "Folder View" "Datacenter" "Virtual Machine" "Any VM" right pane "Hardware", "Hard Disk (scsi0)" (which isn't the scsi controller its the virtual drive attached to the virtual scsi controller), then in the menu bar "Disk Action" dropdown, then "Move Storage". Just a UI peculiarity.

Enable the QEMU guest agent

Have to enable the guest agent in Proxmox VE first, then enable in the VM, then power cycle the VM for Proxmox to make the internal changes necessary to connect to the agent. Apparently it connect to the agent over a Virt-IO port which can not hot-add, or not hot-add reliably.

If not enabled when the VM was created, in the config for the Proxmox VM, "Options" "QEMU Guest Agent", "Edit", checkmark "Use QEMU Guest Agent".

In the console for the VM assuming Ubuntu 20.04 as a test VM:

"sudo apt-get install qemu-guest-agent"

Shutdown the VM "sudo shutdown now", then start the image in the web UI to let Proxmox connect it's guest agent tentacles into the VM. Just rebooting usually will not let Proxmox initially connect to the VM.

Verify QEMU guest agent operation by looking at the "Summary" tab for the VM. If the guest agent is connected the "IP" section will list the current VM IP addresses. Which is handy, because if you installed SSHD on the VM when you initially installed the OS, you can now SSH to the VM at that IP address. On the other hand if the guest agent is not working the "IP" section will contain a complaint to that effect.

Some basic cluster operations testing ideas:

Run a test Ubuntu server doing "something" in a tmux/screen session (more than just idling, perhaps running "top") for a couple days.
Use live migration to send a working VM to visit every host in the cluster. Total downtime to visit every host on the cluster one time added up to around half a second and no individual host ever exceeded a tenth of a second downtime.

Next post will be about migrating production workload off OpenStack cluster number 1 into the new small Proxmox VE cluster. Later on, the hosts formerly in cluster OS1 will be migrated into the Proxmox VE cluster, providing a lot more capacity. The other advantage of slowly rolling workload is this is a good test strategy for the Proxmox cluster system.

Friday, November 10, 2023

Proxmox VE Cluster - Chapter 009 - Creating the cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

This post is about turning half a dozen separate Proxmox VE hosts into a combined cluster.

Reference docs for today:

https://pve.proxmox.com/wiki/Cluster_Manager

https://pve.proxmox.com/pve-docs/chapter-pvecm.html

Cluster creation was done "by the book" and it was uneventful.

Two interesting points to note:

You'll need a peer root password.
After a node is added to the cluster you'll have to manually refresh the web UI because the https SSL certs are changing.

In retrospect, I wish I had manually assigned node IDs to match the hostname rather than just letting the system assign incrementing numbers. My node "proxmox021" would be better identified by manually assigned id 21, rather than autoassigned id 1.

NFS Cluster-wide Storage References

https://pve.proxmox.com/wiki/Storage:_NFS

https://pve.proxmox.com/pve-docs/chapter-pvesm.html#storage_nfs

The NFS shares were created on the TrueNAS server back in blog post Chapter 004

Add NFS shares to the cluster:

Log into proxmox, "Datacenter", "Storage", "Add", "NFS".
ID: use the name from the list of shares such as proxmox-diskimages
Server: 10.10.20.4 (always use IP Address in case DNS is down)
Export: Should autopopulate with dropdowns
Content: Exit selections to match
"Add"

Verify all six NFS shares are "Shared Yes" and "Enabled Yes"

We now have a working Proxmox VE cluster. The next step will be loading some software ISO images into the cluster and creating some test VMs to experiment on and gain Proxmox experience before applying a real workload to the cluster.

Wednesday, November 8, 2023

Proxmox VE Cluster - Chapter 008 - Install Proxmox VE on the old Harvester cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Today, installing Proxmox VE on the old Harvester cluster hardware. The main reference document:

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html

Before starting, note a peculiarity of older 2010s NUC hardware. HDMI will refuse to output if connected to "too high" of a resolution monitor, although the NUC will otherwise boot and work. Meanwhile, a problem with my cheap no name tiny 1080 HDMI monitor is it "often" locks up reporting no signal input on resolution changes, so booting and selecting console install means power cycling the monitor.

Notes from the install process:

I have a habit of using the console install environment.
Country: United States
Timezone: The "timezone" field will not let me enter a timezone, only city names, none of which are nearby. Super annoying I can't just enter a timezone like a real operating system. I ended up selecting a city a thousand miles away. This sucks. Its a "timezone" setting not "name a far away city that coincidentally is in the same timezone". I expect better from Proxmox.
Keyboard Layout: U.S. English
Password: (mind your caps-lock)
Administrator email: vince.mulhollon@springcitysolutions.com
Management Interface: enp2s0 (not the wifi)
Hostname FQDN: as appropriate, as per the sticker on the device
IP address (CIDR): as appropriate, as per the sticker on the device / 016
Gateway address: 10.10.1.1
DNS server address: 10.10.7.21 (my "old" dns22, which will probably get re addressed soon)
Note you can't set up VLANs in the installer, AFAIK. I intend to use VLANs in the distant future.
Hit enter to reboot, yank the USB flash install drive, yank the USB keyboard, watch the monitor... seems to boot properly...
Web interface is on port 8006. Log in as root. Note I installed 8.0-2 and on the first boot, the web gui reports version 8.0.3, it must have auto-updated as part of the install process?

Upgrade the new Proxmox VE node

Double check there's no production workload on the server; its a new install there shouldn't be anything, but its a good habit.
Select the "Server View" then node name, then on the right side, "Updates", "Repositories", disable both enterprise license repos. Add the community repos as explained at https://pve.proxmox.com/wiki/Package_Repositories
Or in summary, click "add", select "No-subscription", "add", then repeat for the "Ceph Quincy No-Subscription" repo.
In right pane, select "Updates" then "Refresh" and watch the update. Click "Upgrade" and watch the upgrade.
Optimistically get a nice message on the console of "Your system is up-to-date" and a request to reboot.
Reboot and verify operation.

Increase the Ethernet MTU

The plan is to change the MTU of the ethernet physical port and the Proxmox internal bridge to 9000. The Netgear ethernet switch was set to 9200+ a long time ago.

Select the node, "System", "Network", select the ethernet port, edit, change the MTU from 1500 to 9000, "OK", "Apply Configuration".
Repeat process selecting the bridge instead of the ethernet port.
On the right pane in "Updates" along the top there is a "Reboot" button, hit it.

Final Installation Checklist after the install and post-installation tasks above:

Perform some basic operation testing
In the web UI "Shutdown" then wait for power down.
Reinstall in permanent location, power up.
Verify information in Netbox to include MAC, serial number, ethernet cabling, platform should be Proxmox VE, remove old Netbox device information.
Add new hosts to Zabbix.
Verify operation one final time.

The next post will be about creating the Proxmox VE cluster out of our currently separate half dozen hosts.

Monday, November 6, 2023

Proxmox VE Cluster - Chapter 007 - Hardware Prep Work on the old Harvester cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

This post documents preparing the old Harvester cluster hardware for it's new Proxmox cluster use. These microservers are three roughly decade-old Intel NUC6i3SYH devices marketed as tiny desktops. These will be nodes proxmox11, 12, and 13, of the eventual thirteen node cluster. These NUC microservers were the stereotypical mid-2010s homelab for VMware ESXi experimentation. They still work great as cluster nodes for other technologies.

The specs for each microserver are quad-core i3-6100U CPU running at 2.30 GHz with upgraded 32 GB of ram and two SSD, a quarter gig SATA and a new-ish 1 TB M2/NVME SSD. The 1 TB drive will eventually be used as part of the Ceph cluster filesystem.

Physically, these microservers are "short cubes" about the size of a CDrom and stack nicely. They only draw a couple watts of power. For $10 to $20 per year of electricity, a microcluster of these microservers in a home lab or experimentation lab can run somewhat less than a dozen VMs for somewhat more than a dollar per year of electricity; a price Amazon AWS simply cannot compete with. With realistic hardware depreciation, a small VM running on this hardware costs perhaps $3 per year.

If the CMOS battery dies, it will force the NUC to only PXEboot off the LAN. So, that's how you know if a NUC server's battery died, it no longer reboots. If power never fails, in theory, you can disable the network boot feature so it will try to boot off the SATA SSD. That, of course, also does not work because the BIOS defaults disable UEFI booting and Proxmox only installs (at least by default) on UEFI. The "F10" boot menu will permit UEFI booting even if UEFI is disabled, leading to hours of confusion and delay. In the end, replace the CMOS battery for preventative reasons, then set the BIOS to defaults, enable UEFI, and it'll probably work.

A preparation checklist:

Clean and wipe old servers, both installed software and physical dusting.
Replace the CMOS batteries which are mostly dead. I will include a link to a Youtube video of the replacement process.
Relabel ethernet cables and servers.
Update port names in the managed Netgear ethernet switch. VLAN config remains the same, single VLAN, untagged.
Remove monitoring of old server in Zabbix.
Verify new devices were added in Netbox.

In the next post, install Proxmox VE on the old Harvester Cluster hardware.

Friday, November 3, 2023

Proxmox VE Cluster - Chapter 006 - Install Proxmox VE on the old Rancher cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Today, installing Proxmox VE on the old Rancher cluster hardware. The main reference document:

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html

On the Beelink N5095, my BIOS boot order defaults to the hard drive (and its old RKE2 K8S installation), but this version of the BIOS has a handy feature where you boot, hit ESC to enter the bios, left arrow will wrap-around to select the rightmost BIOS menu, then select a boot option of single time only UEFI boot the USB key containing the Proxmox VE installation media. Why doesn't PXEboot work over the LAN like everything else? Well that's a long story, booting USB for right now.

Notes from the install process:

Note the USB keyboard I have did not work in the Proxmox graphical install environment, so I used the console install environment.
Country: United States
Timezone: The "timezone" field will not let me enter a timezone, only city names, none of which are nearby. Super annoying I can't just enter a timezone like a real operating system. I ended up selecting a city a thousand miles away. This sucks. Its a "timezone" setting not "name a far away city that coincidentally is in the same timezone". I expect better from Proxmox.
Keyboard Layout: U.S. English
Password: (mind your caps-lock)
Administrator email: vince.mulhollon@springcitysolutions.com
Management Interface: enp2s0 (not the wifi)
Hostname FQDN: as appropriate, as per the sticker on the device
IP address (CIDR): as appropriate, as per the sticker on the device / 016
Gateway address: 10.10.1.1
DNS server address: 10.10.7.21 (my "old" dns22, which will probably get re addressed soon)
Note you can't set up VLANs in the installer, AFAIK. I intend to use VLANs in the distant future.
Hit enter to reboot, yank the USB flash install drive, yank the USB keyboard, watch the monitor... seems to boot properly...
Web interface is on port 8006. Log in as root. Note I installed 8.0-2 and on the first boot, the web gui reports version 8.0.3, it must have auto-updated as part of the install process?

Upgrade the new Proxmox VE node

Double check there's no production workload on the server; its a new install there shouldn't be anything, but its a good habit.
Select the "Server View" then node name, then on the right side, "Updates", "Repositories", disable both enterprise license repos. Add the community repos as explained at https://pve.proxmox.com/wiki/Package_Repositories
Or in summary, click "add", select "No-subscription", "add", then repeat for the "Ceph Quincy No-Subscription" repo.
In right pane, select "Updates" then "Refresh" and watch the update. Click "Upgrade" and watch the upgrade.
Optimistically get a nice message on the console of "Your system is up-to-date" and a request to reboot.
Reboot and verify operation.

Increase the Ethernet MTU

The plan is to change the MTU of the ethernet physical port and the Proxmox internal bridge to 9000. The Netgear ethernet switch was set to 9200+ a long time ago.

Select the node, "System", "Network", select the ethernet port, edit, change the MTU from 1500 to 9000, "OK", "Apply Configuration".
Repeat process selecting the bridge instead of the ethernet port.
On the right pane in "Updates" along the top there is a "Reboot" button, hit it.

R8168 Ethernet Driver Problem in Debian Linux

There is a driver problem with the Debian Linux / Proxmox version 8, R8168 ethernet driver. Every couple seconds, depending on network load, the syslog will report some variation upon:

kernel: pcieport 0000:00:1c.5: AER: Multiple Corrected error received: 0000:00:1c.5

kernel: pcieport 0000:00:1c.5: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)

kernel: pcieport 0000:00:1c.5: device [8086:4dbd] error status/mask=00000041/00002000

kernel: pcieport 0000:00:1c.5: [ 0] RxErr (First)

kernel: pcieport 0000:00:1c.5: [ 6] BadTLP

And it will crash the ethernet connectivity entirely approximately once per day, depending on network use level. The system keeps running and spamming the syslog with the error messages above, however the ethernet driver stops processing packets. Link light will be on and flashing. A power cycle will fix it for "about a day", depending on network load.

This problem is documented in the Proxmox Forums at:

https://forum.proxmox.com/threads/pve8-netdev-watchdog-enp1s0-r8169-transmit-queue-0-timed-out-fix-to-some-extent.133752/

Which references a Medium article at:

https://medium.com/@pattapongj/how-to-fix-network-issues-after-upgrading-proxmox-from-7-to-8-and-encountering-the-r8169-error-d2e322cc26ed

Which references a RealTechTalk article at:

https://realtechtalk.com/Ubuntu_Debian_Linux_Mint_r8169_r8168_Network_Driver_Problem_and_Solution-2253-articles

To paraphrase and summarize the above articles, and implement the work-around:

Verify the Beelink has a R8168 network card with an installed R8169 driver which "almost" works most of the time, by using "lspci -nnk":

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:0123]

Kernel driver in use: r8169

Kernel modules: r8168

It has a r8168 on board and a r8169 driver auto-installed. Well, that's not going to work.

Access the console, ssh or the web UI both work.

Then use vi to edit /etc/apt/sources.list to add non-free and non-free-firmware to BOTH the bookworm and bookworm-updates lines.

"apt update"

"apt install pve-headers"

"apt install r8168-dkms"

"dkms status" should show the r8168 dkms module is installed. Not just ready or something, it needs to report as "installed".

vi /etc/modprobe.d/r8168-dkms.conf and uncomment the blacklist line, such that the r8169 driver will not load.

Reboot and it should be fixed.

After rebooting, in the console, run "lspci -nnk", it should now report:

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:0123]

Kernel driver in use: r8168

Kernel modules: r8168

So, now we have an r8168 on board ethernet hardware device, using the newly installed r8168 driver. Cool.

Is it working better? Immediately I noticed the syslog is not being spammed anymore. I ran it under load for a couple days and it continues to work. I'd consider this fixed. With the wrong driver installed, none of the nodes would never run more than, perhaps, 24 hours. I'm sure this mandatory DKMS driver will make future Proxmox upgrades "more exciting".

Final Installation Checklist after the install and post-installation tasks above:

Perform some basic operation testing
In the web UI "Shutdown" then wait for power down.
Reinstall in permanent location, power up.
Verify information in Netbox to include MAC, serial number, ethernet cabling, platform should be Proxmox VE, remove old Netbox device information.
Add new hosts to Zabbix.
Verify operation one final time.

Next post will be about preparing the old Harvester cluster hardware, before installation of Proxmox VE.

Wednesday, November 1, 2023

Proxmox VE Cluster - Chapter 005 - Hardware Prep Work on the old Rancher cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

This post is about preparing the old Rancher cluster hardware for it's new Proxmox cluster use. These microservers are three relatively new Beelink N5095 devices marketed as tiny desktops. Not suited for heavy loads but more that capable for testing, and running a couple very small VMs in production. These will be nodes proxmox21, 22, and 23, of the eventual thirteen node cluster.

The specs for each microserver are quad-core 2.00 GHz Celeron processors with 8 GB of ram and a single storage device, a quarter TB SSD. The specs are inferior to an Intel NUC from a decade ago, then again the price is about an eighth of what Intel NUCs cost in the mid 2010s, and during the recent "supply chain" problem these devices were always available unlike Raspberry Pi. Compared to a Pi, they both outperform and are substantially lower cost. Great little microservers. The only problem I've found, aside from an ethernet driver issue in Debian/Proxmox discussed later in the series, is they're limited to a single SSD, and the memory as shipped is too small for microserver / homelab applications.

In my experience, as a cluster node, they are heavily memory limited. CPU use rarely exceeds 25% when the 8 GB of ram is near 100% full. At least that means it runs nice and cool...

A preparation checklist:

Clean and wipe old servers, both installed software and physical dusting.
Relabel ethernet cables and servers.
Update port names in the managed Netgear ethernet switch. VLAN config remains the same, single VLAN, untagged.
Remove monitoring of old server in Zabbix.
Verify new devices were added in Netbox.

In the next post, install Proxmox VE on the old Rancher Cluster hardware.