Proxmox VE Cluster - Chapter 014 - Configure NTP on the Proxmox Cluster

Proxmox VE Cluster - Chapter 014 - Configure NTP on the Proxmox Cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

The NTP architecture here uses the E800 nodes as a central NTP source for the entire network.  Those six nodes are the only NTP servers that get time from the internet or from the local GPS refclock, and the rest of the network syncs time off those six.  Helpfully, a long time ago, I set up DNS aliases ntp1 thru ntp6 for these clocks so I don't need to change any configurations after I alter the DNS ... probably.

Basically, today I am converting from classic NTP on the OpenStack servers to Chrony on the Proxmox servers.

References

https://pve.proxmox.com/wiki/Time_Synchronization

https://ubuntu.com/server/docs/how-to-serve-the-network-time-protocol-with-chrony

https://ubuntu.com/server/docs/how-to-serve-the-network-time-protocol-with-chrony

The Big Picture Plan

1. Configure proxmox001-003 to get time from the local GPS clock, from the other proxmox servers, and one internet time pool source.
2. Configure proxmox001-003 to serve time.
3. Modify dns such that ntp1-ntp3 will now point to proxmox001-003.  Note some devices will require manual configuration such as the Ethernet switches, maybe the Ubiquity wifi, maybe the TrueNAS, who knows?
4. After proxmox004-006 are set up, the DNS hosts ntp4-ntp6 will need to be updated.

I will set this up manually because its simple and I have not integrated proxmox with Ansible yet.  But eventually Proxmox will be configured via Ansible.

Manually configuring chrony on Proxmox VE

1. Create (or copy) the files for sources into "/etc/chrony/sources.d" I put exactly one clock in each file.  Files in sources.d can be re-read without restarting the entire service by running "chronyc reload sources".  If successful you should see the other clocks are now accessible when running "chronyc sources".
2. Remove the default clocks shipped by Proxmox and enable NTP serving.  Edit /etc/chrony/chrony.conf and comment out the "pool" directive and add a line underneath "allow 10.0.0.0/8"  This will require a service restart not a mere reload, so "service chrony restart" and verify Chrony operation after a few minutes using "chronyc sources"
3. Edit DNS for ntp1 (or as appropriate) to point to the new proxmox node IP address.
4. Test NTP from various VMs and hardware to verify NTP is working.

List of clocks in /etc/chrony/sources.d:

• gpsclock.sources = the local, on LAN, "stratum 1-ish" GPS clock
• proxmox001.sources = should be five files pointing to the other five E800 nodes
• pool.sources = "server 0.pool.ntp.org" as an external reference.

Cool, the new Proxmox nodes are now providing NTP time service to the network.  Next blog post will be about moving all the workload off the old OpenStack OS2 cluster so as to repurpose that hardware as yet more Proxmox capacity.

Proxmox VE Cluster - Chapter 013 - Install Proxmox VE on the old OS1 cluster hardware

Proxmox VE Cluster - Chapter 013 - Install Proxmox VE on the old OS1 cluster hardware

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Some notes on installing Proxmox VE on the old OS1 cluster hardware.  The main reference document:

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html

The plan to work around the networking challenges is to get everything working on a single plain temporary 1G ethernet connection, then use that as a management web interface to get the dual 10G LAG with VLANs up and running, then use the new "20G" ethernet connected web management interface to connect and reconfigure the dual 1G LAG / VLAN ethernet ports, at which point everything will be working.

First Steps

Install using IPMI KVM and USB key, so find the usb key and plug in the IPMI ethernet.

On boot, DEL for setup, alter the boot options to include the USB drive, reboot, hit F11 for boot menu, then boot off the USB.

Proxmox "OS" install process

• I have a habit of using the console install environment.
• Installer wants to default to install on the M2 drive although I am using the SATA.
• Country: United States
• Timezone: The "timezone" field will not let me enter a timezone, only city names, none of which are nearby.  Super annoying I can't just enter a timezone like a real operating system.  I ended up selecting a city a thousand miles away.  This sucks.  Its a "timezone" setting not "name a far away city that coincidentally is in the same timezone".  I expect better from Proxmox.
• Keyboard Layout: U.S. English
• Password: (mind your caps-lock)
• Administrator email: vince.mulhollon@springcitysolutions.com
• Management Interface: the first 1G ethernet (eno1, aka the "bottom left corner")
• Hostname FQDN: as appropriate, as per the sticker on the device
• IP address (CIDR): as appropriate, as per the sticker on the device / 016
• Gateway address: 10.10.1.1
• DNS server address: 10.10.8.221
• Note you can't set up VLANs in the installer, AFAIK.
• Hit enter to reboot, yank the USB flash install drive, yank the USB keyboard, watch the monitor... seems to boot properly...
• Web interface is on port 8006.  Log in as root.  Note I installed 8.0-2 and on the first boot, the web gui reports version 8.0.3, it must have auto-updated as part of the install process?

Upgrade the new Proxmox VE node

1. Double check there's no production workload on the server; its a new install there shouldn't be anything, but its a good habit.
2. Select the "Server View" then node name, then on the right side, "Updates", "Repositories", disable both enterprise license repos.  Add the community repos as explained at https://pve.proxmox.com/wiki/Package_Repositories
3. Or in summary, click "add", select "No-subscription", "add", then repeat for the "Ceph Quincy No-Subscription" repo.
4. In right pane, select "Updates" then "Refresh" and watch the update.  Click "Upgrade" and watch the upgrade.
5. Optimistically get a nice message on the console of "Your system is up-to-date" and a request to reboot.
6. Reboot and verify operation.

Install hardware in permanent location with temporary ethernet cables

1. Perform some basic operation testing
2. In the web UI "Shutdown" then wait for power down.
3. Reinstall in permanent location.
4. Connect eno1 to any untagged "Prod" VLAN 10 access-only ethernet port, temporarily, for remote management via the web interface.
5. Connect the 10G ethernets eno3 and eno4 to the LAG'd and VLAN'd 10G ethernet switch ports.

Move the Linux Bridge from single 1 gig eno1 to dual 10 gig LAG on eno3 and eno4

You are going to need this:

https://pve.proxmox.com/wiki/Network_Configuration

1. Modify eno3 and eno4, checkmark "Advanced", change MTU to 9000.
2. Create a Linux Bond named bond1, Checkmark "Advanced", change MTU to 9000, Mode "balance-xor", slaves "eno3 eno4" (note space in between, not comma etc).  Note bond0 will eventually be the 1G LAG, and the old OpenStack used "balance-xor" so I will start with that on the Proxmox.
3. Create Linux VLAN named bond1.10 with MTU 9000, can create the other VLANs now if you want.
4. Edit vmbr0 Linux bridge to have a MTU of 9000 and Bridge Ports of bond1.10
5. Double check everything then "Apply Configuration", and after about twelve to thirteen heart stopping seconds it should be up and working.

At some later date I will try some LAG bond modes more interesting than "balance-xor".

Note the network interfaces do not have "VLAN aware" checked.  Everything works.  I will research this later in a dedicated advanced networking post.

Convert the single 1 gig eno1 to dual 1 gig LAG on eno1 and eno2

1. Edit eno1 and eno2 and set MTU to 9000
2. Create a Linux Bond named bond0, Checkmark "Advanced", change MTU to 9000, Mode "balance-xor", slaves "eno1 eno2" (space in between).
3. Create VLAN interfaces now on bond0, or create them later.

Final Installation Tasks

1. Join the new node(s) to the existing cluster.
2. Verify information in Netbox to include MAC, serial number, ethernet cabling, platform should be Proxmox VE, remove old Netbox device information.
3. Add new hosts to Zabbix.

The next post will be about setting up NTP on the new Proxmox VE cluster.

Proxmox VE Cluster - Chapter 012 - Hardware Prep Work on OS1 cluster

Proxmox VE Cluster - Chapter 012 - Hardware Prep Work on OS1 cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

These microservers are three old SuperMicro SYS-E200-8D that were used for Homelab workloads.  They will become Proxmox cluster nodes proxmox001, proxmox002, and proxmox003.  

This server hardware was stereotypical for a late 2010's "VMware ESXi Eval Experience"-licensed cluster, and later worked very well under OpenStack.  1.90 GHz Xeon D-1528 with six cores and 96 GB of ram, 1 TB SATA SSD for boot and local storage, new 1 TB M2 NVME SSD for eventual CEPH cluster storage.

Hardware reliability history

Proxmox001 is the only server out of six that is still running the original AC power supply.  The other five required replacement.  Voltage would sag lower and lower until there were random reboots under heavy load, and eventually the supplies would fail completely.  Thankfully its just a dead power supply and the rest of the hardware has been extremely reliable.  I can't recommend SuperMicro hardware enough, its really good stuff... other than the power supplies from the late 2010s.

Proxmox002 had its AC power brick replaced 2020-11-19 and AGAIN on 2023-07-01

Proxmox003 had a NVME failure 2021-02-20, AC power brick replaced 2022-05-31

FIVE Ethernet ports

Even the official manufacturer's operating manual fails to explain the layout of the five ethernet ports on this server.  Looking at the back of the server, the lone port on the left side is the IPMI, then:

eno1 1G ethernet bottom left corner, 9000 byte MTU

eno2 1G ethernet top left corner, 9000 byte MTU

eno3 10G ethernet bottom right corner, 9000 byte MTU

eno4 10G ethernet top right corner, 9000 byte MTU

eno1 and eno2 are combined into bond12, which uses balance-xor mode to provide 2 GB of bandwidth.

eno3 and eno4 are combined into bond34, which uses balance-xor mode to provide 20 GB of bandwidth.  20 GB ethernet is pretty fast!

I run the VLANs as subinterfaces of the bond interfaces.  So, "Production" VLAN 10, has an interface name of "bond34.10"

Hardware Preparation task list

1. Clean and wipe old servers, both installed software and physical dusting.
2. Relabel ethernet cables and servers.
3. Update port names in the managed Netgear ethernet switch.  VLAN and LAG configs remain the same, making installation "exciting" and "interesting".
4. Remove monitoring of old server in Zabbix.
5. Verify IPAM information in Netbox.
6. Test and verify new server DNS entries.
7. Install new 1TB M.2/NVME SSDs.
8. Replace old CMOS CR2032 battery as it's probably 5 to 7 years old.  This is child's-play compared to replacing the battery on a hyper-compact Intel-NUC.
9. Reconfigure the BIOS in each server.  For a variety of reasons, PXE netboot requires UEFI and BIOS initialization of the network, so I used that in the OpenStack era which was installed on top of Ubuntu.  However, I could not force the UEFI bios to boot the SATA SSD it insisted on booting the M.2 only, which is odd because it worked fine under older, USB-stick installed Ubuntu.  Another problem with the BIOS config was "something" about pre-initializing the ethernet system for PXEBoot messes up the bridge configuration on Proxmox's Debian OS, resulting in traffic not flowing; I experimented with manually adding other interfaces to the bridge; no go; symptoms were no packets flowing in (brctl showmac is essentially the bridge's ARP table) also no packets out, although link light up and everything looks OK.  Anyway, in summary, disable PXEboot entirely and convert entirely from UEFI to Legacy BIOS booting.  This was typical of the UEFI experience in the late 2010s, it doesn't really work most of the time, but Legacy BIOS booting always works.  Things are better now.

In the next post, we install Proxmox VE on the old OS1 cluster hardware.  It'll be interesting with all those VLANs and LAGs.

Proxmox VE Cluster - Chapter 011 - Move OpenStack cluster 1 workload to Proxmox

Proxmox VE Cluster - Chapter 011 - Move OpenStack cluster 1 workload

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Before the hardware used for OpenStack cluster OS1 can be repurposed for the Proxmox cluster, I need to move all the virtual machines and containers off OS1.  There are several options: temporarily delete them until there is more capacity, permanently delete them if no longer needed, or move to the Proxmox cluster.

The old "warm backup" availability strategy for OpenStack was some workload was installed on both clusters, but only operating on one cluster at a time, for example, one of the minor file servers.  It was expensive to keep two copies around of "everything" and only running one copy on the Proxmox cluster should save quite a bit of capacity, overall.

Here is a list of workload I moved to Proxmox:

netbootxyz

Netboot.xyz provides network booting infrastructure.  Network booting starts with a DHCP server like ISC-DHCP (Or KEA...) pointing a booting PC to a TFTP address, the address of the netboot.xyz server.  This is where Netboot.xyz comes into the picture, it serves a really nice CLI menu of dozens of operating system install ISO files.  A very convenient way to install an OS.  There are also plenty of testing and troubleshooting images available.

https://netboot.xyz/

The VM is a simple Ubuntu 20.04 install that runs Docker.  I NFS mount all my Docker volumes, this has worked well for several years.  The move was uneventful.  Shut down the old VM on the OS1 cluster, start the new VM (on a new address) on the Proxmox cluster, run a script I keep in the NFS mounted docker directory to pull and start a netboot.xyz container, repoint the DHCP servers to the new netboot.xyz ip address, and it just works.

wiki

This is a Docker container of DokuWiki.  I use it as a "home page" or "phone book" for the LAN.  If its a web-accessible server, it has a link to it on the wiki.

https://www.dokuwiki.org/dokuwiki

This is another simple Ubuntu VM holding a Docker container, much like the netbootxyz VM above.  One of many advantages of storing my Docker volumes over NFS is a move like this is so simple; shut down the Docker container on the old server, start the Docker container on the new server, done.  The move was uneventful.

dhcp11, dhcp12, dhcp21, dhcp22 all replaced by dhcp01, dhcp02

This is a classic dual server ISC-DHCPD cluster.

https://www.isc.org/dhcp/

On the OS1 cluster this was running FreeBSD and converted this to Ubuntu 20.04.  This conversion was uneventful.  I am aware ISC DHCP is discontinued as of 2022, and KEA is the next generation of ISC supported DHCP servers.  Will convert to KEA later, stay tuned for a Spring City Solutions Youtube channel video about that conversion process.

dc11, dc12 replaced by dc01

This LAN uses Samba servers as Active Directory Domain Controllers.  Really nice to have network access from any machine to my home directory, and SSO is also pretty cool.

https://www.samba.org/

This was also a conversion from FreeBSD Samba (which is pretty easy to use) to Ubuntu Samba (which is definitely not as easy to use).  After the initial OS install and Ansible configuration, Do not do a "net ads join -U administrator" during the Ansible process.  The Ubuntu samba-tool utility has to create a fresh smb.conf file from scratch during the joining process, so just move the Ansible provided file out of the way temporarily (the Ansible file should be identical other than configuring DNS forwarder servers).  After initial configuration you will have to manually edit (or use ansible) to fix the DNS forwarder entry in /etc/samba/smb.conf.  You can join a Ubuntu DC to a domain with no error messages while running the "user" set of samba services SMBD, NMBD, Winbond and there will be zero error messages aside from "samba-tool drs showrepl" command failing to connect to port 135, and of course the DC not working in general.  It seems that on Ubuntu, you need to shut down the user class daemons SMBD, NMBD, and WINBOND using systemctl, then look up how systemd permanently shuts down samba-ad-dc in order to figure out how to "unmask" that service.  The next Ubuntu Samba related problem is systemd-resolved is autoconfigured to start on port 53 before trying to start Samba while refusing external connections, and samba-ad-dc will successfully start and not output any error messages while failing to bind to port 53, so in summary by default domain controller authoritative DNS will fail to work.  Systemd is always so annoying to use and just makes everything harder.  The solution is to  "systemctl stop" and systemctl mask" the "systemd-resolved.service".  A final minor problem, or surprise, is I couldn't get replication working without a reboot.  Yes I know its on a fifteen minute (or so) timer but it just wouldn't start without a reboot.  Believe it or not, on FreeBSD, none of this drama is necessary.  Regardless of extensive effort required to work around systemd, Samba worked eventually.

dns11, dns12 replaced by dns01

These are typical ISC Bind DNS servers.  Samba Active Directory Domain Controllers do not have a very advanced resolver, so I forward DNS queries for everything they're not authoritative for to a resolver cluster.  I've always preferred a DNS architecture that keeps authoritative DNS and DNS resolving separate.

https://www.isc.org/bind/

The VM is a simple Ubuntu server acting as a DNS resolver (not doing authoritative DNS).  The conversion was uneventful.  Active directory domain controllers DC11 and DC12 use DNS11 and DNS12 so I could not shut down and remove DNS11 and DNS12 until after removing DC11 and DC12.

observium

Observium is a SNMP-based monitoring system.  Mostly monitors my Netgear ethernet switches, but can monitor other devices.

https://www.observium.org/

The VM is another simple Ubuntu Docker server.  I had to expand the memory to 8 GB and expand the hard disk in Proxmox (then expand the LVM PV, then the LVM LV, finally expand the FS) to 32 GB and expand the CPU allocation to dual core.  Starts up slow, but works.

zabbix

Zabbix is an agent-based detailed monitoring system.  All VMs and some physical hardware run the zabbix agent for advanced monitoring and trend analysis.

https://www.zabbix.com/

The Zabbix system has two VMs.  In the past I've run into weird problems with the Zabbix trying to monitor itself.  One way around that is Zabbix has a proxy/concentration server, so I install one of those, and it can connect to the main Zabbix server.  Probably no longer necessary, but it is pretty cool.

The zabbix VM is yet another simple Ubuntu Docker server.  This was a bit of a headache.  I ran into some kind of incompatibility between old and new MySQL versions.  Would have impacted me upon next container upgrade had I not moved to a new cluster.  Ended up doing a complete Zabbix reinstall, there are too many new features to make the backup useful, etc.  In the end after some labor, this works pretty well.

The zabbixproxy VM, again, yet another simple Ubuntu Docker server, holds the Docker container for zabbix-proxy.  The move from the OS1 cluster was uneventful.

zerotier

ZeroTier is a complete VPN solution.  Pretty cool!

https://www.zerotier.com/

Uneventful reinstallation.  Ended up creating a new connection to Zerotier then reroute traffic for the LAN to the new connection (because for awhile I was running both in parallel for testing purposes, which also necessitated some LAN IP address and static route juggling).  Also updated DNS to point to dc01 (and later on, added dc02 and dc03, although they don't exist yet).

Next blog post will be about hardware prep work to turn the old OpenStack OS1 cluster into more Proxmox nodes.

Proxmox VE Cluster - Chapter 010 - OS Images on Proxmox and manual VM operations

Proxmox VE Cluster - Chapter 010 - OS Images on Proxmox and manual VM operations

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Today is about manually testing and experimenting with the Proxmox VE cluster.

References for this post:

https://pve.proxmox.com/wiki/Qemu/KVM_Virtual_Machines

Load some ISO images into the shared Proxmox Cluster:

Download some ISO files, such as Ubuntu 20.04.6 LTS from 

https://ubuntu.com/download/server

Add the ISO install image to Proxmox:

1. Log into the Proxmox web UI
2. Select any cluster node
3. Expand in the left pane
4. Select proxmox-isoimages as a storage location
5. Click "ISO Images" on the right pane
6. "Upload"
7. Wait for the pop up window to report "TASK OK" then close the window out.

Verify the ISO image uploaded using the CLI on any Linux box.

1. Previously when I set up the NFS shares I added symlinks from my NFS home directory to the proper automounter location.  So ~/proxmox-isoimages is symlinked to /net/freenas/mnt/freenas-pool/proxmox-isoimages.  This means I can log into any Linux system on the LAN (Via SSO using Samba and Active Directory) and "cd ~/proxmox-isoimages"
2. The "~/proxmox-isoimages" directory contains "template"
3. The "~/proxmox-isoimages/template" directory contains "iso"
4. The "~/proxmox-isoimages/template/iso" directory contains the Ubuntu Server install ISO uploaded previously.

Verify the ISO image uploaded using the Web UI.

1. Log into any Proxmox cluster node web UI.
2. In the left pane select "Folder View" then "Storage" then any of the node's "proxmox-isoimages".
3. In the right pane select "ISO Images" on the left side, and the right side of the right pane should populate with a list of ISO images identical to the directory contents seen in the previous verification task.

Create a test VM and experiment with it.

1. Select any node. Click "Create VM". 
2. In general pane, Name: Test, probably want to check "Start at boot" for most VMs.
3. In OS pane, Storage: "proxmox-isoimages", ISO image: select the Ubuntu dropdown.
4. In System pane, default for everything EXCEPT checkmark "Qemu Agent".
5. In Disks pane, Storage: proxmox-diskimages, Disk size: 16 GB.
6. In CPU pane, default single core seems OK?
7. In Memory, default 2GB seems OK?
8. In Network, default bridge seems OK, should get a DHCP address.
9. Select the test VM in the left pane, click "Start" button.
10. Click on the console and install the operating system.

Note that for Ubuntu 20.04, the default LVM config will only use about half the PV, have to modify the LV to use most of the disk rather than leaving most of it unused.

After the install I disconnect the media from the virtual cdrom although I leave the virtual cdrom drive in place for possible later use.

Verify disk image in CLI

1. Previously when I set up the NFS shares I added symlinks from my NFS home directory to the proper automounter location.  So ~/proxmox-diskimages is symlinked to /net/freenas/mnt/freenas-pool/proxmox-diskimages.  This means I can log into any Linux system on the LAN (Via SSO using Samba and Active Directory) and "cd ~/proxmox-diskimages
2. The "~/proxmox-isoimages" directory contains "images"
3. The "~/proxmox-isoimages/images" directory contains a directory with the VM ID number, probably 100 for your first VM.
4. The "~/proxmox-isoimages/images/100" directory contains the qcow2 disk image, probably named something like "vm-100-disk-0.qcow2"  As near as I can tell it's thin provisioned, I allocated sixteen gigs at the time of VM creation, but I'm using maybe six or so gigs per VM image.

Verify disk image in Web UI

1. Log into the Proxmox web UI, any cluster member is fine.
2. Left pane, "Folder View", "Datacenter", "Storage", select any host's copy of "proxmox-diskimages"
3. Right pane, "VM Disks", see the list of disks.

Two interesting things to note about VM disk storage in Proxmox, at least in Proxmox version 8:

1. Configure a VM's disk image as 16 gigs, it thin-provisions on the NFS server using perhaps six or so gigs, it displays in the Web UI in "Storage" as exactly 17.18 GB.
2. You CAN NOT migrate storage, or what VMware would call "Storage vMotion" in the web UI from "Folder View" "Datacenter" "Storage" "Any shared NFS mount".  However, you CAN migrate storage from "Folder View" "Datacenter" "Virtual Machine" "Any VM" right pane "Hardware", "Hard Disk (scsi0)" (which isn't the scsi controller its the virtual drive attached to the virtual scsi controller), then in the menu bar "Disk Action" dropdown, then "Move Storage".  Just a UI peculiarity.

Enable the QEMU guest agent

Have to enable the guest agent in Proxmox VE first, then enable in the VM, then power cycle the VM for Proxmox to make the internal changes necessary to connect to the agent.  Apparently it connect to the agent over a Virt-IO port which can not hot-add, or not hot-add reliably.

If not enabled when the VM was created, in the config for the Proxmox VM, "Options" "QEMU Guest Agent", "Edit", checkmark "Use QEMU Guest Agent".

In the console for the VM assuming Ubuntu 20.04 as a test VM:

"sudo apt-get install qemu-guest-agent"

Shutdown the VM "sudo shutdown now", then start the image in the web UI to let Proxmox connect it's guest agent tentacles into the VM.  Just rebooting usually will not let Proxmox initially connect to the VM.

Verify QEMU guest agent operation by looking at the "Summary" tab for the VM.  If the guest agent is connected the "IP" section will list the current VM IP addresses.  Which is handy, because if you installed SSHD on the VM when you initially installed the OS, you can now SSH to the VM at that IP address.  On the other hand if the guest agent is not working the "IP" section will contain a complaint to that effect.

Some basic cluster operations testing ideas:

• Run a test Ubuntu server doing "something" in a tmux/screen session (more than just idling, perhaps running "top") for a couple days.
• Use live migration to send a working VM to visit every host in the cluster. Total downtime to visit every host on the cluster one time added up to around half a second and no individual host ever exceeded a tenth of a second downtime.

Next post will be about migrating production workload off OpenStack cluster number 1 into the new small Proxmox VE cluster.  Later on, the hosts formerly in cluster OS1 will be migrated into the Proxmox VE cluster, providing a lot more capacity.  The other advantage of slowly rolling workload is this is a good test strategy for the Proxmox cluster system.

Proxmox VE Cluster - Chapter 009 - Creating the cluster

Proxmox VE Cluster - Chapter 009 - Creating the cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

This post is about turning half a dozen separate Proxmox VE hosts into a combined cluster.

Reference docs for today:

https://pve.proxmox.com/wiki/Cluster_Manager

https://pve.proxmox.com/pve-docs/chapter-pvecm.html

Cluster creation was done "by the book" and it was uneventful.

Two interesting points to note:

1. You'll need a peer root password.
2. After a node is added to the cluster you'll have to manually refresh the web UI because the https SSL certs are changing.

In retrospect, I wish I had manually assigned node IDs to match the hostname rather than just letting the system assign incrementing numbers.  My node "proxmox021" would be better identified by manually assigned id 21, rather than autoassigned id 1.

NFS Cluster-wide Storage References

https://pve.proxmox.com/wiki/Storage:_NFS

https://pve.proxmox.com/pve-docs/chapter-pvesm.html#storage_nfs

The NFS shares were created on the TrueNAS server back in blog post Chapter 004

Add NFS shares to the cluster:

1. Log into proxmox, "Datacenter", "Storage", "Add", "NFS".
2. ID: use the name from the list of shares such as proxmox-diskimages
3. Server: 10.10.20.4 (always use IP Address in case DNS is down)
4. Export: Should autopopulate with dropdowns
5. Content: Exit selections to match
6. "Add"

Verify all six NFS shares are "Shared Yes" and "Enabled Yes"

We now have a working Proxmox VE cluster.  The next step will be loading some software ISO images into the cluster and creating some test VMs to experiment on and gain Proxmox experience before applying a real workload to the cluster.

Proxmox VE Cluster - Chapter 008 - Install Proxmox VE on the old Harvester cluster

Proxmox VE Cluster - Chapter 008 - Install Proxmox VE on the old Harvester cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Today, installing Proxmox VE on the old Harvester cluster hardware.  The main reference document:

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html

Before starting, note a peculiarity of older 2010s NUC hardware.  HDMI will refuse to output if connected to "too high" of a resolution monitor, although the NUC will otherwise boot and work.  Meanwhile, a problem with my cheap no name tiny 1080 HDMI monitor is it "often" locks up reporting no signal input on resolution changes, so booting and selecting console install means power cycling the monitor.

Notes from the install process:

• I have a habit of using the console install environment.
• Country: United States
• Timezone: The "timezone" field will not let me enter a timezone, only city names, none of which are nearby.  Super annoying I can't just enter a timezone like a real operating system.  I ended up selecting a city a thousand miles away.  This sucks.  Its a "timezone" setting not "name a far away city that coincidentally is in the same timezone".  I expect better from Proxmox.
• Keyboard Layout: U.S. English
• Password: (mind your caps-lock)
• Administrator email: vince.mulhollon@springcitysolutions.com
• Management Interface: enp2s0 (not the wifi)
• Hostname FQDN: as appropriate, as per the sticker on the device
• IP address (CIDR): as appropriate, as per the sticker on the device / 016
• Gateway address: 10.10.1.1
• DNS server address: 10.10.7.21 (my "old" dns22, which will probably get re addressed soon)
• Note you can't set up VLANs in the installer, AFAIK.  I intend to use VLANs in the distant future.
• Hit enter to reboot, yank the USB flash install drive, yank the USB keyboard, watch the monitor... seems to boot properly...
• Web interface is on port 8006.  Log in as root.  Note I installed 8.0-2 and on the first boot, the web gui reports version 8.0.3, it must have auto-updated as part of the install process?

Upgrade the new Proxmox VE node

1. Double check there's no production workload on the server; its a new install there shouldn't be anything, but its a good habit.
2. Select the "Server View" then node name, then on the right side, "Updates", "Repositories", disable both enterprise license repos.  Add the community repos as explained at https://pve.proxmox.com/wiki/Package_Repositories
3. Or in summary, click "add", select "No-subscription", "add", then repeat for the "Ceph Quincy No-Subscription" repo.
4. In right pane, select "Updates" then "Refresh" and watch the update.  Click "Upgrade" and watch the upgrade.
5. Optimistically get a nice message on the console of "Your system is up-to-date" and a request to reboot.
6. Reboot and verify operation.

Increase the Ethernet MTU

The plan is to change the MTU of the ethernet physical port and the Proxmox internal bridge to 9000.  The Netgear ethernet switch was set to 9200+ a long time ago.

1. Select the node, "System", "Network", select the ethernet port, edit, change the MTU from 1500 to 9000, "OK", "Apply Configuration".
2. Repeat process selecting the bridge instead of the ethernet port. 
3. On the right pane in "Updates" along the top there is a "Reboot" button, hit it.

Final Installation Checklist after the install and post-installation tasks above:

1. Perform some basic operation testing
2. In the web UI "Shutdown" then wait for power down.
3. Reinstall in permanent location, power up.
4. Verify information in Netbox to include MAC, serial number, ethernet cabling, platform should be Proxmox VE, remove old Netbox device information.
5. Add new hosts to Zabbix.
6. Verify operation one final time.

The next post will be about creating the Proxmox VE cluster out of our currently separate half dozen hosts.