Proxmox VE Cluster - Chapter 020 - Architecture 1.0 Review and Future Directions

Proxmox VE Cluster - Chapter 020 - Architecture 1.0 Review and Future Directions

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

What worked

So far, everything.  And it all works better than I expected, and was generally less of a headache than I anticipated.  Performance is vastly better than OpenStack for various design and overall architectural reasons.

How long it took

I am writing these blog posts in a non-linear fashion so the final editing of this post is being done on a CEPH cluster with HA and the new software defined networking system and quite a few other interesting items, most of which already have rough draft blog posts.

However, if you believe Clockify, over the course of half a year of hobby-scale effort, I have logged 125 hours, 57 minutes, and 52 seconds getting to this point.  So, around "three weeks full time labor" to convert a small OpenStack cluster to a medium size half-way-configured Proxmox VE cluster.  I believe this is about half the time it took to convert from VMware to OpenStack a couple years back.

Future Adventures

This is just a list of topics you can expect to see in blog posts and Spring City Solutions Youtube videos probably after the holidays or in late winter / early spring:  Setting up and upgrading CEPH.  Optimizing memory, CPU, storage.  Adding the new SDN feature.  Open vSwitch and Netgear hardware QoS.  Connecting Ansible and probably Terraform to Proxmox.  Monitoring using Observium, Zabbix, and Elasticsearch.  Setting up Rancher and RKE2 production clusters on top of Proxmox.  Backups using the Proxmox Backup Server product.  Cloud-init, will I ever it it working the way I want it to work?  HA High Availability, unfortunately I can verify this software feature works excellently during hardware failures.  USB pass thru.  

Most of the stuff listed above is done or in process, and already partially documented in rough draft blog posts.  CEPH integration, for example, has been unimaginably cool.

Anyway, thanks for reading and have a great day!

Proxmox VE Cluster - Chapter 019 - Proxmox Operations

Proxmox VE Cluster - Chapter 019 - Proxmox Operations

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Proxmox Operations is a broad and complicated topic.

Day to day operations are performed ALMOST entirely in the web GUI, with very few visits to the CLI.  I have years of experience with VMware and OpenStack, and weeks, maybe even months of experience with Proxmox, so let's compare the experience:

• VMware:  vSphere is installed on the cluster as an image, and as an incredibly expensive piece of licensed software, you get one (maybe two, depending on HA success) installation of vSphere and you get to hope it works.  Backup, restore, upgrades, and installation work about as well as you expect for "enterprise" grade software.
• OpenStack: Horizon is installed on the controller and the controller is NOT part of the cluster.  It's free, feel free to install multiple controllers although I never operated that way.  Its expensive in terms of hardware as the core assumptions of the design assume you're throwing a rather large cloud, not a couple hosts in a rack.  Upgrades are terrifying and moderately painful and long process.  The kolla-ansible solution of running it all in containers is interesting although it replaces the un-troubleshoot-able complication of bare metal installation with an equal level of un-troubleshoot-able complication of Docker containers.
• Proxmox VE: Every VE node has a web front end to do CRUD operations against the shared cluster configuration database.  The VE system magically synchronizes the hardware to match the configuration database.  Very cool design and 100% reliable so far.  Scalability is excellent; whereas OpenStack assumes you're rolling in with a minimum of a dozen or so nodes, Proxmox works from as low as one isolated node.

An interesting operational note is the UI on Proxmox is more "polished" and "professional" and "complete" than either alternative.  Usually FOSS has a reputation for inadequate UI but Proxmox has the best UI of the three.

Upgrades

Lets consider one operational task.  Upgrades.  Proxmox is essentially a Debian Linux installation with a bunch of Proxmox specific packages installed on top of it.  Not all that different from installing Docker or ElasticSearch from upstream.  I try to upgrade every node in the cluster at least monthly, the less stuff that changes per upgrade the less "exciting" the upgrade.  The level of excitement and drama and stress scales exponentially with the number of upgraded software packages with Debian-based operating systems in general.

The official Proxmox process for upgrades is just hit it, maybe have to reboot, all good.

https://pve.proxmox.com/wiki/System_Software_Updates

As you'd expect, there are complications, IRL.

First I make a plan, upgrading all the hosts in one sitting because I don't want cross-version compatibility cluster issues, and I start with the least sensitive cluster host.  Note that if you log into proxmox001 and upgrade/reboot proxmox002, you stay logged into the cluster.  However if you log into proxmox001 and upgrade and reboot proxmox001, you lose web access to the rest of the cluster during the reboot (as a work around, simply log into the proxmox002 webui while rebooting proxmox001).

Next I verify the backups of the VMs on a node, and generally poke thru the logs.  If I'm getting hardware errors or something I want to know before I start changing software.  Yes this blog post series is non-linear and I haven't mentioned backups or the Proxmox Backup Server product but those posts are coming soon.

I generally shutdown clustered VMs and unimportant VMs and migrate "important" VMs to other hosts. 

There are special notes about Beelink DKMS process for the custom ethernet driver using non-free firmware.  Basically Proxmox 8.0 shipped with a Linux kernel that could be modified to use the DKMS driver for the broken Realtek ethernet driver, however, the DKMS driver does NOT seem compatible with the kernel shipped with Proxmox 8.1, so after some completely fruitless hours of effort, I simply removed my three Beelink microservers from the cluster.  "Life's too short to use Realtek".  You'd think Linux compatibility would be better in 2023 than 1993 when I got started, but really there isn't much difference between 2023 and 1993 and plenty of stuff just doesn't work.  So, here's a URL to remove nodes from a cluster, which is a bit more involved than adding nodes LOL:

https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node

Other than fully completing and verifying operation of exactly one node at a time, I have no serious advice.  Upgrades on Proxmox generally just work, somehow even less drama than VMware upgrades.  Lightyears less stress than an OpenStack upgrade.  Don't forget to update the Runbook docs and due date in Redmine after each node upgrade.

Note that upgrading the Proxmox VE software is only half the job, once that's done entirely across the cluster its time to look at CEPH.  Again I mention these blog posts are being written long after the action, and I haven't mentioned CEPH in a blog post.  Those posts are on the way.

https://pve.proxmox.com/wiki/Ceph_Quincy_to_Reef

Shortly after I rough drafted these blog posts, Proxmox 8.1 dropped along with an upgrade from CEPH Quincy to CEPH Reef.  AFAIK any CEPH upgrade even a minor version number is basically the same as a major upgrade, just much less exciting and stressful.  I do everything for a minor upgrade in the same order and process, more or less, as a major CEPH version upgrade, and that may even be correct.  It does work, at least so far.

Next post, a summary and evaluation of "Architecture Level 1.0" where we've been and where we're going.

Proxmox VE Cluster - Chapter 018 - Moving the remainder of workload to the full size cluster

Proxmox VE Cluster - Chapter 018 - Moving the remainder of workload to the full size cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Some notes on moving the remainder of the old OpenStack workload to the full size Proxmox cluster.  These VMs were "paused" for a couple days and recreated on Proxmox.

Elasticsearch cluster members es04, es05, es06

This is the other half of the six host Elasticsearch cluster.  Rather than storing the disk images over CEPH (foreshadowing of future posts...) or enabling HA high availability (more foreshadowing of adventures to come...) I use local 100 LVM disks because the Proxmox VE system only uses a couple gigs of my 1 TB SSD OS install drives.

Adding more cluster members to an existing Elasticsearch cluster is no big deal.  Create a temporary cluster enrollment token on any existing cluster member, install a blank unused Elasticsearch binary on the VM, run the cluster-mode reconfiguration script with the previously mentioned token, wait until it's done.  The main effort is adjusting the config for kibana, filebeat, and metricbeat on Ansible so I can push out config changes to all hosts to use the additional three cluster members.  It 'just works'.  Currently, I have index lifecycle management to store only a couple days of logs and metrics because it seems 600 gigs of logs fills up faster than it did back in the 'old days'.

jupyter, mattermost, navidrome, pocketmine, tasmoadmin, ttrss, others..

These are just docker hosts that run docker containers.  The scripts to set up the docker containers, and the docker volumes, are stored on the main NFS server, so re-deployment amounts to install an Ubuntu server, let Ansible set it up to join the AD domain, install Docker for me, set up autofs, etc, then simply run my NFS mounted scripts to run Docker containers accessing NFS mounted Docker volumes.

booksonic, others...

Another Docker host like the above paragraph.  I had set up Active Directory authentication for a couple applications running in Docker containers and I had some "fun" reconfiguring them to use the new domain controller IP addresses.  No big deal, however, AD auth reconfiguration was an unexpected additional step.  If "everything" is configured automatically in Ansible, but its not REALLY "everything", then its easy to forget some application-level configuration remains necessary.  Every system that's big enough, has a couple loose ends somewhere.

kapua, kura, hawkbit, mqttrouter (containing Eclipse Mosquitto)

This is my local install of the Eclipse project Java IoT suite that I use for microcontroller experimentation and applications.

Kapua is a web based server for IoT that does everything except firmware updates.  The software is run via a complicated shell script running version 1 docker-compose that works fine with version 2 docker compose, after exporting some shell environment variables to force the correct Kapua version and editing the start up script to run v2 "docker compose" instead of v1 "docker-compose".  Kapua overall is a bit too complicated to explain in this blog post.

Kura is an example Java IoT device framework running locally in Docker instead of on real hardware, for testing Kapua and generally messing around.

Hawkbit is a firmware updater and it works great, anything with wifi/ethernet and MCUboot can upgrade itself very reliably, or recover from being bricked.  Works great with STM32 boards.

Finally, as for mqttrouter, simply start the NFS config and Eclipse Mosquitto works.

The Eclipse project Java-based IoT suite is REALLY cool and once upon a time I planned a multi-video Youtube series using it and Zephyr but I ran out of RAM on my STM32 boards before implementing more than 50% of the Kapua/Kura protocol and now-a-days I'd just install Kura on a Raspberry Pi, if not Node-RED or on the smaller end install one of the microcontroller Python implementations and call it good; maybe some day I'll get back into Eclipse Java IoT.

win11

This was a gigantic struggle.  The Proxmox side works perfectly, with emulated TPM and the install went perfectly smoothly.  The problem was I have a valid windows license on microsoft.com for this VM but the image refused to 'activate'.  I paid list price for this license that I can't even use; I can see why people have a bad attitude about Microsoft...  None the less, via various technical means I now have a remotely accessible domain-joined windows 11 image that I can access via Apache Guacamole's rdesktop feature from any modern web browser (including my Chromebook) to run windows "stuff" remotely.  Works pretty well, aside from the previously mentioned license activation problem.  Everything 'Microsoft' is a struggle all the time.

ibm7090, pdp8, rdos, rsx11, tops10, mvs, a couple others

Runs the latest OpenSIMH retrocomputing emulator in a tmux window.  The MVS host has the "famous" MVS/370 Turnkey 5 installed along with a console 3270 emulator.  The disk images are normally stored over NFS along with all configs.  All data is stored in projects on Redmine.  I have login entries on Apache Guacamole so I have full access to my retrocomputing environment via any web browser.

Next blog post:  Various operations issues.  Upgrading Proxmox VE software, daily stuff like that.

Proxmox VE Cluster - Chapter 017 - Install Proxmox VE on the old OS2 cluster hardware

Proxmox VE Cluster - Chapter 017 - Install Proxmox VE on the old OS2 cluster hardware

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Some notes on installing Proxmox VE on the old OS2 cluster hardware.  The main difference between installation on OS1 and OS2 is adding NTP serving to OS2, more or less as per Chapter 014 NTP notes.  The main reference document:

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html

The plan to work around the networking challenges is to get everything working on a single plain temporary 1G ethernet connection, then use that as a management web interface to get the dual 10G LAG with VLANs up and running, then use the new "20G" ethernet connected web management interface to connect and reconfigure the dual 1G LAG / VLAN ethernet ports, at which point everything will be working.

First Steps

Install using IPMI KVM and USB key, so find the usb key and plug in the IPMI ethernet.

On boot, DEL for setup, alter the boot options to include the USB drive, reboot, hit F11 for boot menu, then boot off the USB.

Proxmox "OS" install process

• I have a habit of using the console install environment.
• Installer wants to default to install on the M2 drive although I am using the SATA.
• Country: United States
• Timezone: The "timezone" field will not let me enter a timezone, only city names, none of which are nearby.  Super annoying I can't just enter a timezone like a real operating system.  I ended up selecting a city a thousand miles away.  This sucks.  Its a "timezone" setting not "name a far away city that coincidentally is in the same timezone".  I expect better from Proxmox.
• Keyboard Layout: U.S. English
• Password: (mind your caps-lock)
• Administrator email: vince.mulhollon@springcitysolutions.com
• Management Interface: the first 1G ethernet (eno1, aka the "bottom left corner")
• Hostname FQDN: as appropriate, as per the sticker on the device
• IP address (CIDR): as appropriate, as per the sticker on the device / 016
• Gateway address: 10.10.1.1
• DNS server address: 10.10.8.221
• Note you can't set up VLANs in the installer, AFAIK.
• Hit enter to reboot, yank the USB flash install drive, yank the USB keyboard, watch the monitor... seems to boot properly...
• Web interface is on port 8006.  Log in as root.  Note I installed 8.0-2 and on the first boot, the web gui reports version 8.0.3, it must have auto-updated as part of the install process?

Upgrade the new Proxmox VE node

1. Double check there's no production workload on the server; its a new install there shouldn't be anything, but its a good habit.
2. Select the "Server View" then node name, then on the right side, "Updates", "Repositories", disable both enterprise license repos.  Add the community repos as explained at https://pve.proxmox.com/wiki/Package_Repositories
3. Or in summary, click "add", select "No-subscription", "add", then repeat for the "Ceph Quincy No-Subscription" repo.
4. In right pane, select "Updates" then "Refresh" and watch the update.  Click "Upgrade" and watch the upgrade.
5. Optimistically get a nice message on the console of "Your system is up-to-date" and a request to reboot.
6. Reboot and verify operation.

Install hardware in permanent location with temporary ethernet cables

1. Perform some basic operation testing
2. In the web UI "Shutdown" then wait for power down.
3. Reinstall in permanent location.
4. Connect eno1 to any untagged "Prod" VLAN 10 access-only ethernet port, temporarily, for remote management via the web interface.
5. Connect the 10G ethernets eno3 and eno4 to the LAG'd and VLAN'd 10G ethernet switch ports.

Move the Linux Bridge from single 1 gig eno1 to dual 10 gig LAG on eno3 and eno4

You are going to need this:

https://pve.proxmox.com/wiki/Network_Configuration

1. Modify eno3 and eno4, checkmark "Advanced", change MTU to 9000.
2. Create a Linux Bond named bond1, Checkmark "Advanced", change MTU to 9000, Mode "balance-xor", slaves "eno3 eno4" (note space in between, not comma etc).  Note bond0 will eventually be the 1G LAG, and the old OpenStack used "balance-xor" so I will start with that on the Proxmox.
3. Create Linux VLAN named bond1.10 with MTU 9000, can create the other VLANs now if you want.
4. Edit vmbr0 Linux bridge to have a MTU of 9000 and Bridge Ports of bond1.10
5. Double check everything then "Apply Configuration", and after about twelve to thirteen heart stopping seconds it should be up and working.

At some later date I will try some LAG bond modes more interesting than "balance-xor".

Note the network interfaces do not have "VLAN aware" checked.  Everything works.  I will research this later in a dedicated advanced networking post.

Convert the single 1 gig eno1 to dual 1 gig LAG on eno1 and eno2

1. Edit eno1 and eno2 and set MTU to 9000
2. Create a Linux Bond named bond0, Checkmark "Advanced", change MTU to 9000, Mode "balance-xor", slaves "eno1 eno2" (space in between).
3. Create VLAN interfaces now on bond0, or create them later.

Configure NTP

1. Create (or copy) the files for sources into "/etc/chrony/sources.d" I put exactly one clock in each file.  Files in sources.d can be re-read without restarting the entire service by running "chronyc reload sources".  If successful you should see the other clocks are now accessible when running "chronyc sources".
2. Remove the default clocks shipped by Proxmox and enable NTP serving.  Edit /etc/chrony/chrony.conf and comment out the "pool" directive and add a line underneath "allow 10.0.0.0/8"  This will require a service restart not a mere reload, so "service chrony restart" and verify Chrony operation after a few minutes using "chronyc sources"
3. Edit DNS for ntp4 (or as appropriate) to point to the new proxmox node IP address.
4. Edit NTP on ALL THE OTHER NODES to reflect the presence of this new NTP server.
5. Test NTP from various nodes, VMs, and hardware to verify NTP is working.

Final Installation Tasks

1. Join the new node(s) to the existing cluster.  In "Datacenter" on any cluster member, "Cluster" "Join Information" "Copy Information" cut and paste into "Datacenter" on the new node, "Join Cluster", enter the peer's root password, "Join 'Proxmox'".  Will have to log back into the web UI after the SSL certs update...
2. Verify information in Netbox to include MAC, serial number, ethernet cabling, platform should be Proxmox VE, remove old Netbox device information.
3. Add new hosts to Zabbix.

The next post will be about adding the remaining "paused" workload to the now "full sized" Proxmox VE cluster.

Proxmox VE Cluster - Chapter 016 - Hardware Prep Work on the OS2 Cluster

Proxmox VE Cluster - Chapter 016 - Hardware Prep Work on the OS2 Cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

This will be similar to Chapter 012 although different hardware.

These microservers are three old SuperMicro SYS-E200-8D that were used for Homelab workloads.  They will become Proxmox cluster nodes proxmox004, proxmox005, and proxmox006.  

This server hardware was stereotypical for a late 2010's "VMware ESXi Eval Experience"-licensed cluster, and later worked very well under OpenStack.  1.90 GHz Xeon D-1528 with six cores and 96 GB of ram, 1 TB SATA SSD for boot and local storage, new 1 TB M2 NVME SSD for eventual CEPH cluster storage.

Hardware reliability history

Proxmox004 had its AC power brick replaced 2022-07-10

Proxmox005 had a NVME failure 2020-05-24, took advantage of that outage to also upgrade its SSD to a new 1TB (on suspicion, the old one was working fine although wearout measurement was getting to a high percentage per SMART reports) on 2020-05-27.

Proxmox006 had a NVME failure 2021-02-20, and had its AC power brick replaced 2022-06-10

Previously in Chapter012 I claimed that 5/6 of the power supplies had failed on my E800 microservers, but I made a mistake, and it seems "only" TWO THIRDS of the power supplies have failed as of late 2023, currently proxmox001 and proxmox005 are still running on the original mid 2010s power supplies.  I will keep a close eye on the output voltages (monitorable via IMPI using Observium and probably Zabbix and maybe somehow via Elasticsearch)

FIVE Ethernet ports

Even the official manufacturer's operating manual fails to explain the layout of the five ethernet ports on this server.  Looking at the back of the server, the lone port on the left side is the IPMI, then:

eno1 1G ethernet bottom left corner, 9000 byte MTU

eno2 1G ethernet top left corner, 9000 byte MTU

eno3 10G ethernet bottom right corner, 9000 byte MTU

eno4 10G ethernet top right corner, 9000 byte MTU

eno1 and eno2 are combined into bond12, which uses balance-xor mode to provide 2 GB of bandwidth.

eno3 and eno4 are combined into bond34, which uses balance-xor mode to provide 20 GB of bandwidth.  20 GB ethernet is pretty fast!

I run the VLANs as subinterfaces of the bond interfaces.  So, "Production" VLAN 10, has an interface name of "bond34.10"

Hardware Preparation task list

1. Clean and wipe old servers, both installed software and physical dusting.
2. Relabel ethernet cables and servers.
3. Update port names in the managed Netgear ethernet switch.  VLAN and LAG configs remain the same, making installation "exciting" and "interesting".
4. Remove monitoring of old server in Zabbix.
5. Verify IPAM information in Netbox.
6. Test and verify new server DNS entries.
7. Install new 1TB M.2/NVME SSDs.
8. Replace old CMOS CR2032 battery as it's probably 5 to 7 years old.  This is child's-play compared to replacing the battery on a hyper-compact Intel-NUC.
9. Reconfigure the BIOS in each server.  For a variety of reasons, PXE netboot requires UEFI and BIOS initialization of the network, so I used that in the OpenStack era which was installed on top of Ubuntu.  However, I could not force the UEFI bios to boot the SATA SSD it insisted on booting the M.2 only, which is odd because it worked fine under older, USB-stick installed Ubuntu.  Another problem with the BIOS config was "something" about pre-initializing the ethernet system for PXEBoot messes up the bridge configuration on Proxmox's Debian OS, resulting in traffic not flowing; I experimented with manually adding other interfaces to the bridge; no go; symptoms were no packets flowing in (brctl showmac is essentially the bridge's ARP table) also no packets out, although link light up and everything looks OK.  Anyway, in summary, disable PXEboot entirely and convert entirely from UEFI to Legacy BIOS booting.  This was typical of the UEFI experience in the late 2010s, it doesn't really work most of the time, but Legacy BIOS booting always works.  Things are better now.

Next post will be about installing Proxmox VE on the old OS2 cluster hardware.

Proxmox VE Cluster - Chapter 015 - Migrate workload of the old OS2 cluster

Proxmox VE Cluster - Chapter 015 - Migrate workload of the old OS2 cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

I plan to reuse the OpenStack cluster OS2 hardware to increase Proxmox cluster capacity.  Before I can reuse the hardware, I need to move all remaining workload off the OpenStack hardware.  The OS2 workload is either immediately migrated to Proxmox if its important enough, or "paused" for a few days and restored on Proxmox after the cluster work is done.  Here is a list of the OS2 cluster workload, and a description of what I did to each application:

storage

An Ubuntu Samba and NFS fileserver.

Converted from FreeBSD to Ubuntu.  While doing that, I added NFS file serving for Ubuntu on Ansible, so that is scripted and automated now.

netbox

https://netbox.dev/

A complete FOSS IPAM system.

This one was tricky, its a docker-compose with local volumes so need to backup the DB on the old system then restore onto the new system, more or less.  I need to completely redesign this so all data is stored over NFS instead of on local host volumes.  Its the only container I have that stores data in local volumes which makes management a hassle.

I recommend against directly following the database restore instructions on:

https://github.com/netbox-community/netbox-docker/wiki/Troubleshooting#database-operations

I unfortunately have extensive experience with restoring an older schema on top of a freshly installed empty new schema, resulting in considerable data loss.  At least I keep good backups LOL, so I was able to recover from that.  A better strategy is NOT to start everything then shut off the client processes, then (try to) restore an old schema backup over an empty new schema as per existing online docs, but instead to start ONLY the postgres container, then while it's empty, restore the old schema backup into postgres, then and only then start up the client containers (everything else) and let the automatic upgrade process upgrade the schema of the freshly restored old schema database.  That strategy worked perfectly with no data loss.

I filed a docs improvement bug regarding the above adventure at GitHub:

https://github.com/netbox-community/netbox-docker/issues/1113

unifi

The "unifi" controller software in a docker container for a cloudy-ish WIFI network based on Ubiquiti hardware.

The Unifi Controller is just another Docker container.  However, a new controller IP address means re-homing all Unifi devices off the old server and old IP address and onto the new server with it's new IP address.  I copied the NFS mounted volume for unifi controller over to unificontroller-old, because then I could start controllers on both the new and old servers.  Obviously, every Ubiquiti hardware device on the LAN reconnected to the unifi-old server, although the new unifi server looked fine (other than acting like all devices were suddenly disconnected from it, which is accurate).  Then on the old controller, in "Settings" "System" "Advanced" there's an "inform host" setting which had the old server's IP address, so I put in the new address and hit apply.  There's also a way to manually SSH into each device individually, which can be a bit of a pain, so I used the web UI "Inform" method.

The above resulted in a minor problem, the "inform host" on the new controller was  pointing to the old controller IP address because the new controller was a clone of the old controller, and the "inform host" on the old controller was pointing to the new host, so the devices ping-pong-ed back and forth between the old and new controllers for awhile.  I fixed the "inform host" setting on the new controller and Wi-Fi devices started coming online.  Cool.  The ethernet switches were mad at me for at least several minutes, I think I crashed some firmware or something, although they did eventually ALL come online.  Mildly interesting that the Wi-Fi devices connect much faster than the ethernet switches.  In summary, that was exciting for awhile, but in the end it all worked pretty well.

redmine

https://www.redmine.org/

A FOSS project management suite

Shut down the docker-compose on the old server, start it on the new server, painless.  All the data volumes reside on the same NAS NFS server, this was just moving the docker "compute host" from the old OpenStack cluster number two, to the new Proxmox docker "compute host".

es02, es03, kibana

https://www.elastic.co/

Elasticsearch infrastructure for syslog storage and analysis

This turned into a larger adventure that initially planned.  Ended up being an upgrade of Elasticsearch from 8.6 to 8.10 and a new cluster being formed on ES01, ES02, and ES03.  Later I will add ES04, ES05 and ES06.  This seems like a lot of work to store syslog messages, but Elasticsearch as a database technology is fun to play with and Kibana can make really cool graphical dashboards, so its worth the effort.

portainer

https://www.portainer.io/

A FOSS centralized web based Docker management tool.

Shut down the docker container on the old server, start it on the new server, painless.

guacamole

https://guacamole.apache.org/

The Apache Guacamole project provides a website that turns any web browser into a SSH client or RDesktop client.

Shut down the old docker container, start on the new server, painless.

dc21, dc22 to dc02, dc03

https://www.samba.org/

Samba Active Directory Domain Controller Cluster.

Need to remove dc21 and dc22 as the second to last VMs removed from OS2, some VMs on OS2 will point to dc21 and dc22 for DNS resolution.

Probably the only "interesting" thing to remember to do was move the FSMO roles to dc01 off of dc21.

I took this opportunity to clean up the old DNS entries.  I use the RSAT tools on a windows 11 desktop, works pretty well to control Samba Active Directory.

dns21, dns22 to dns02, dns03

Ubuntu servers doing DNS resolving requests that are forwarded from the Domain Controller cluster.

dc01 forwards DNS resolution to dns01 and dns02, dc02 forwards to dns02 and dns03, and so forth, so everything has multiple backups.  This works pretty well.

Need to remove dns21 dns22 after removing dc21 dc22.  These will be the last VMs removed from OS2.

emby

https://emby.media/

A media server and DVR for Roku and other household set top boxes.

Just an Ubuntu server with emby package installed.

Need to convert when the recordings list is empty

Note there is, of course, a new IP addrs.

Emby has a very elaborate and detailed manual provisioning process, documented in it's Redmine runbook "issue".

ubuntu

General end user use.

Move old ubuntu to ubuntu-old in DNS

Set up a new Ubuntu for enduser use.

Make sure Ansible runs on the new ubuntu before shutting down the old ubuntu.

backup

An Ubuntu NFS and Samba fileserver holding backup data.

This was a test fileserver before converting "storage2" to "storage".  Only problem I ran into was minor, FreeBSD prefers the use of the group "wheel" whereas Ubuntu prefers the use of the group "root".

Next blog post, there's nothing left on OpenStack cluster 2, shut it down and prepare the OS2 cluster hardware for reuse as additional Proxmox capacity.

Proxmox VE Cluster - Chapter 014 - Configure NTP on the Proxmox Cluster

Proxmox VE Cluster - Chapter 014 - Configure NTP on the Proxmox Cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

The NTP architecture here uses the E800 nodes as a central NTP source for the entire network.  Those six nodes are the only NTP servers that get time from the internet or from the local GPS refclock, and the rest of the network syncs time off those six.  Helpfully, a long time ago, I set up DNS aliases ntp1 thru ntp6 for these clocks so I don't need to change any configurations after I alter the DNS ... probably.

Basically, today I am converting from classic NTP on the OpenStack servers to Chrony on the Proxmox servers.

References

https://pve.proxmox.com/wiki/Time_Synchronization

https://ubuntu.com/server/docs/how-to-serve-the-network-time-protocol-with-chrony

https://ubuntu.com/server/docs/how-to-serve-the-network-time-protocol-with-chrony

The Big Picture Plan

1. Configure proxmox001-003 to get time from the local GPS clock, from the other proxmox servers, and one internet time pool source.
2. Configure proxmox001-003 to serve time.
3. Modify dns such that ntp1-ntp3 will now point to proxmox001-003.  Note some devices will require manual configuration such as the Ethernet switches, maybe the Ubiquity wifi, maybe the TrueNAS, who knows?
4. After proxmox004-006 are set up, the DNS hosts ntp4-ntp6 will need to be updated.

I will set this up manually because its simple and I have not integrated proxmox with Ansible yet.  But eventually Proxmox will be configured via Ansible.

Manually configuring chrony on Proxmox VE

1. Create (or copy) the files for sources into "/etc/chrony/sources.d" I put exactly one clock in each file.  Files in sources.d can be re-read without restarting the entire service by running "chronyc reload sources".  If successful you should see the other clocks are now accessible when running "chronyc sources".
2. Remove the default clocks shipped by Proxmox and enable NTP serving.  Edit /etc/chrony/chrony.conf and comment out the "pool" directive and add a line underneath "allow 10.0.0.0/8"  This will require a service restart not a mere reload, so "service chrony restart" and verify Chrony operation after a few minutes using "chronyc sources"
3. Edit DNS for ntp1 (or as appropriate) to point to the new proxmox node IP address.
4. Test NTP from various VMs and hardware to verify NTP is working.

List of clocks in /etc/chrony/sources.d:

• gpsclock.sources = the local, on LAN, "stratum 1-ish" GPS clock
• proxmox001.sources = should be five files pointing to the other five E800 nodes
• pool.sources = "server 0.pool.ntp.org" as an external reference.

Cool, the new Proxmox nodes are now providing NTP time service to the network.  Next blog post will be about moving all the workload off the old OpenStack OS2 cluster so as to repurpose that hardware as yet more Proxmox capacity.