Proxmox VE Cluster - Chapter 006 - Install Proxmox VE on the old Rancher cluster

Proxmox VE Cluster - Chapter 006 - Install Proxmox VE on the old Rancher cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Today, installing Proxmox VE on the old Rancher cluster hardware.  The main reference document:

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html

On the Beelink N5095, my BIOS boot order defaults to the hard drive (and its old RKE2 K8S installation), but this version of the BIOS has a handy feature where you boot, hit ESC to enter the bios, left arrow will wrap-around to select the rightmost BIOS menu, then select a boot option of single time only UEFI boot the USB key containing the Proxmox VE installation media.  Why doesn't PXEboot work over the LAN like everything else?  Well that's a long story, booting USB for right now.

Notes from the install process:

• Note the USB keyboard I have did not work in the Proxmox graphical install environment, so I used the console install environment.
• Country: United States
• Timezone: The "timezone" field will not let me enter a timezone, only city names, none of which are nearby.  Super annoying I can't just enter a timezone like a real operating system.  I ended up selecting a city a thousand miles away.  This sucks.  Its a "timezone" setting not "name a far away city that coincidentally is in the same timezone".  I expect better from Proxmox.
• Keyboard Layout: U.S. English
• Password: (mind your caps-lock)
• Administrator email: vince.mulhollon@springcitysolutions.com
• Management Interface: enp2s0 (not the wifi)
• Hostname FQDN: as appropriate, as per the sticker on the device
• IP address (CIDR): as appropriate, as per the sticker on the device / 016
• Gateway address: 10.10.1.1
• DNS server address: 10.10.7.21 (my "old" dns22, which will probably get re addressed soon)
• Note you can't set up VLANs in the installer, AFAIK.  I intend to use VLANs in the distant future.
• Hit enter to reboot, yank the USB flash install drive, yank the USB keyboard, watch the monitor... seems to boot properly...
• Web interface is on port 8006.  Log in as root.  Note I installed 8.0-2 and on the first boot, the web gui reports version 8.0.3, it must have auto-updated as part of the install process?

Upgrade the new Proxmox VE node

1. Double check there's no production workload on the server; its a new install there shouldn't be anything, but its a good habit.
2. Select the "Server View" then node name, then on the right side, "Updates", "Repositories", disable both enterprise license repos.  Add the community repos as explained at https://pve.proxmox.com/wiki/Package_Repositories
3. Or in summary, click "add", select "No-subscription", "add", then repeat for the "Ceph Quincy No-Subscription" repo.
4. In right pane, select "Updates" then "Refresh" and watch the update.  Click "Upgrade" and watch the upgrade.
5. Optimistically get a nice message on the console of "Your system is up-to-date" and a request to reboot.
6. Reboot and verify operation.

Increase the Ethernet MTU

The plan is to change the MTU of the ethernet physical port and the Proxmox internal bridge to 9000.  The Netgear ethernet switch was set to 9200+ a long time ago.

1. Select the node, "System", "Network", select the ethernet port, edit, change the MTU from 1500 to 9000, "OK", "Apply Configuration".
2. Repeat process selecting the bridge instead of the ethernet port. 
3. On the right pane in "Updates" along the top there is a "Reboot" button, hit it.

R8168 Ethernet Driver Problem in Debian Linux

There is a driver problem with the Debian Linux / Proxmox version 8, R8168 ethernet driver.  Every couple seconds, depending on network load, the syslog will report some variation upon:

kernel: pcieport 0000:00:1c.5: AER: Multiple Corrected error received: 0000:00:1c.5

kernel: pcieport 0000:00:1c.5: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)

kernel: pcieport 0000:00:1c.5:   device [8086:4dbd] error status/mask=00000041/00002000

kernel: pcieport 0000:00:1c.5:    [ 0] RxErr                  (First)

kernel: pcieport 0000:00:1c.5:    [ 6] BadTLP     

And it will crash the ethernet connectivity entirely approximately once per day, depending on network use level.  The system keeps running and spamming the syslog with the error messages above, however the ethernet driver stops processing packets.  Link light will be on and flashing.  A power cycle will fix it for "about a day", depending on network load.

This problem is documented in the Proxmox Forums at:

https://forum.proxmox.com/threads/pve8-netdev-watchdog-enp1s0-r8169-transmit-queue-0-timed-out-fix-to-some-extent.133752/

Which references a Medium article at:

https://medium.com/@pattapongj/how-to-fix-network-issues-after-upgrading-proxmox-from-7-to-8-and-encountering-the-r8169-error-d2e322cc26ed

Which references a RealTechTalk article at:

https://realtechtalk.com/Ubuntu_Debian_Linux_Mint_r8169_r8168_Network_Driver_Problem_and_Solution-2253-articles

To paraphrase and summarize the above articles, and implement the work-around:

Verify the Beelink has a R8168 network card with an installed R8169 driver which "almost" works most of the time, by using "lspci -nnk":

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:0123]

        Kernel driver in use: r8169

        Kernel modules: r8168

It has a r8168 on board and a r8169 driver auto-installed.  Well, that's not going to work.

Access the console, ssh or the web UI both work.

Then use vi to edit /etc/apt/sources.list to add non-free and non-free-firmware to BOTH the bookworm and bookworm-updates lines.

"apt update"

"apt install pve-headers"

"apt install r8168-dkms"

"dkms status" should show the r8168 dkms module is installed.  Not just ready or something, it needs to report as "installed".

vi /etc/modprobe.d/r8168-dkms.conf and uncomment the blacklist line, such that the r8169 driver will not load.

Reboot and it should be fixed.

After rebooting, in the console, run "lspci -nnk", it should now report:

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:0123]

        Kernel driver in use: r8168

        Kernel modules: r8168

So, now we have an r8168 on board ethernet hardware device, using the newly installed r8168 driver.  Cool.

Is it working better?  Immediately I noticed the syslog is not being spammed anymore.  I ran it under load for a couple days and it continues to work.  I'd consider this fixed.  With the wrong driver installed, none of the nodes would never run more than, perhaps, 24 hours.  I'm sure this mandatory DKMS driver will make future Proxmox upgrades "more exciting".

Final Installation Checklist after the install and post-installation tasks above:

1. Perform some basic operation testing
2. In the web UI "Shutdown" then wait for power down.
3. Reinstall in permanent location, power up.
4. Verify information in Netbox to include MAC, serial number, ethernet cabling, platform should be Proxmox VE, remove old Netbox device information.
5. Add new hosts to Zabbix.
6. Verify operation one final time.

Next post will be about preparing the old Harvester cluster hardware, before installation of Proxmox VE.

Proxmox VE Cluster - Chapter 005 - Hardware Prep Work on the old Rancher cluster

Proxmox VE Cluster - Chapter 005 - Hardware Prep Work on the old Rancher cluster

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

This post is about preparing the old Rancher cluster hardware for it's new Proxmox cluster use.  These microservers are three relatively new Beelink N5095 devices marketed as tiny desktops.  Not suited for heavy loads but more that capable for testing, and running a couple very small VMs in production.  These will be nodes proxmox21, 22, and 23, of the eventual thirteen node cluster.

The specs for each microserver are quad-core 2.00 GHz Celeron processors with 8 GB of ram and a single storage device, a quarter TB SSD.  The specs are inferior to an Intel NUC from a decade ago, then again the price is about an eighth of what Intel NUCs cost in the mid 2010s, and during the recent "supply chain" problem these devices were always available unlike Raspberry Pi.  Compared to a Pi, they both outperform and are substantially lower cost.  Great little microservers.  The only problem I've found, aside from an ethernet driver issue in Debian/Proxmox discussed later in the series, is they're limited to a single SSD, and the memory as shipped is too small for microserver / homelab applications.

In my experience, as a cluster node, they are heavily memory limited.  CPU use rarely exceeds 25% when the 8 GB of ram is near 100% full.  At least that means it runs nice and cool...

Physically, these microservers are "short cubes" about the size of a CDrom and stack nicely.  They only draw a couple watts of power.  For $10 to $20 per year of electricity, a microcluster of these microservers in a home lab or experimentation lab can run somewhat less than a dozen VMs for somewhat more than a dollar per year of electricity; a price Amazon AWS simply cannot compete with.  With realistic hardware depreciation, a small VM running on this hardware costs perhaps $3 per year.

A preparation checklist:

1. Clean and wipe old servers, both installed software and physical dusting.
2. Relabel ethernet cables and servers.
3. Update port names in the managed Netgear ethernet switch.  VLAN config remains the same, single VLAN, untagged.
4. Remove monitoring of old server in Zabbix.
5. Verify new devices were added in Netbox.

In the next post, install Proxmox VE on the old Rancher Cluster hardware.

Proxmox VE Cluster - Chapter 004 - Infrastructure Prep

Proxmox VE Cluster - Chapter 004 - Infrastructure Prep

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Before I change anything, I wanted to prepare as much infrastructure as possible.  I did not want to interrupt a conversion step via the realization I forgot to allocate IP addresses or forgot to configure the DNS server.

For day-to-day documentation, such as lists of IP address or URL links to services, I use Dokuwiki which is the most basic and simple wiki software I can find.  I created a new wiki page and added the entire IP allocation for the new cluster:

• proxmox001 (old os1 node in OS1 cluster) SuperMicro SYS-E200-8D 10.10.8.1
• proxmox002 (old os2 node in OS1 cluster) SuperMicro SYS-E200-8D 10.10.8.2
• proxmox003 (old os3 node in OS1 cluster) SuperMicro SYS-E200-8D 10.10.8.3
• proxmox004 (old os4 node in OS2 cluster) SuperMicro SYS-E200-8D 10.10.8.4
• proxmox005 (old os5 node in OS2 cluster) SuperMicro SYS-E200-8D 10.10.8.5
• proxmox006 (old os6 node in OS2 cluster) SuperMicro SYS-E200-8D 10.10.8.6
• proxmox011 (old harvester-small-1) Intel NUC6i3SYH 10.10.8.11
• proxmox012 (old harvester-small-2) Intel NUC6i3SYH 10.10.8.12
• proxmox013 (old harvester-small-3) Intel NUC6i3SYH 10.10.8.13
• proxmox021 (old rancher1) Beelink N5095 10.10.8.21
• proxmox022 (old rancher2) Beelink N5095 10.10.8.22
• proxmox023 (old rancher3) Beelink N5095 10.10.8.23
• proxmox031 (old docker) Intel NUC6i3SYH 10.10.8.31
• proxmoxbackup (old server bare metal hardware) 10.10.8.254
• freenas.cedar.mulhollon.com 10.10.20.4 (use IP addresses for NFS so as to not rely on DNS)

As systems came online, I added hyperlinks to their web GUI and generally keep the wiki page up-to-date with the current configuration.  If I'm working on the cluster, I probably have this wiki page open.

Set up Redmine project for Proxmox:

I use Redmine for project level long term, large scale documentation.  I created a project in Redmine to basically be a more detailed version of the wiki page.

Set up simple insecure NFS on TrueNAS for non-production short term testing:

https://pve.proxmox.com/wiki/Storage:_NFS

https://pve.proxmox.com/pve-docs/chapter-pvesm.html#storage_nfs

A list of six content types to create shares for in TrueNAS (as documented in the Wiki and in Redmine, of course):

1. proxmox-containers
2. proxmox-containertemplates
3. proxmox-diskimages
4. proxmox-isoimages
5. proxmox-snippets
6. proxmox-vzdumps

Creating the six NFS exports in TrueNAS for Promox:

1. First create the datasets, for later NFS export.  
2. "Storage", "Pools", in freenas-pool, three dots "Add Dataset".
3. Name: the id from the list
4. Comments: "something that makes sense"
5. Compression Level: off
6. "Submit"
7. Then export the six datasets.
8. "Sharing", "Unix Shares (NFS)",  "Add", "Advanced Options"
9. Path: /mnt/freenas-pool/proxmox-diskimages (or similar)
10. Maproot User blank
11. Maproot Group blank
12. Mapall User: root
13. Mapall Group: wheel
14. Optionally add authorized networks and hosts, later.  No need to access outside 10.0.0.0/8, obviously.
15. Add symlinks in my homedir to the automounter locations "ln -s /net/freenas/mnt/freenas-pool/proxmox-diskimages ~"
16. Test by creating and deleting some files.

Add sysadmin issue tasks in Redmine in the "Systems Administration" project for all cluster nodes.

Document all the changes and allocations in Netbox

Here is the link to the Port / Services list for Proxmox VE nodes:

https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_requirements

Set up a USB flash boot drive for servers that can't PXEboot

Download Proxmox VE 8.0-2 and put it on a properly labeled USB flash drive.

https://www.proxmox.com/en/downloads/proxmox-virtual-environment/iso

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html#installation_prepare_media

At this point I think I've prepared everything possible.  In the next post, I start the conversion work.

Proxmox VE Cluster - Chapter 003 - Order of Operations for Level 1.0

Proxmox VE Cluster - Chapter 003 - Order of Operations for Level 1.0

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

The order of operations is complicated because I intend to keep most of the workload fully operational during the conversion.  Its kind of like rebuilding a car engine while driving the car down the road.

1. Convert the old Rancher "RKE2" K8S cluster into a very small Proxmox VE cluster.  This step will be successful if I have a working three node Proxmox cluster.
2. Move everything off the small Rancher Harvester test cluster, which is currently slowly running an old version of Harvester, and add those nodes into the small Proxmox VE cluster.  The measure of success will be having six clustered Proxmox nodes.
3. Get some practice with VMs on the small test cluster.  Success will be defined as some working scratch (test load only) Ubuntu servers with a couple days "burn-in" and operational experimentation.
4. Move a minimal set of very small production services on the small Proxmox VE cluster.  Maybe start with the wiki server, one of the multiple Active Directory Domain Controllers, just enough to prove out the operation of the cluster.  I consider this step a success if everything "important" on the old OS1 cluster is minimally running on the new Proxmox server reliably for a couple days.
5. Migrate all production workload off the OS1 OpenStack cluster then add the former OS1 nodes into the now medium-sized Proxmox VE cluster.  Success at this step looks like a Proxmox cluster of nine nodes running a minimal production workload for a couple uninterrupted days.
6. Roll ALL production workload off the OS2 OpenStack cluster into the now medium sized Proxmox VE cluster, likely a tight fit.  Success looks like OS2 having zero load and Proxmox carrying the entire production load, although in theory if Proxmox crashed I have the OS2 cluster has a hot-backup to Proxmox.
7. Convert the remaining OS2 OpenStack cluster into even more Proxmox cluster capacity.  This step is a success if the Proxmox cluster has twelve operating nodes holding the entire production load, and I'm no longer running RKE2 or Harvester or OpenStack or any other cluster system on bare metal.
8. Verify Operation, load balancing, run it for awhile before working on Architecture level 2.0.  Success looks like no crashing, no bugs, no issues, optimized CPU/memory settings and optimized workload across the dozen cluster nodes.

Next post will be about Infrastructure preparation efforts, get as much stuff ready as possible before starting the big conversion project.

Proxmox VE Cluster - Chapter 002 - Plan for Architecture Level 1.0

Proxmox VE Cluster - Chapter 002 - Plan for Architecture Level 1.0

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Here's my detailed plan for Architecture Level 1.0:

In summary, Level 1.0 means drop everything from multiple old clusters into a single large, simple-as-possible Proxmox VE cluster via several conversion phases, while making absolute minimum changes to design and workload.

A list of Goals for Architecture Level 1.0: 

1. All production storage on the giant TrueNAS over NFS.  Cluster-wide filesystems implemented later.
2. VMs manually provisioned, like the older VMWare era.  I like Terraform and Ansible as tools to provision IaaS, but I will implement that later.
3. Generally document and make minimal changes in Ansible with respect to the workload VMs and containers.

Some of the eternal ongoing projects such as re-IP addressing will continue as part of Arch 1.0, which is ambitious.  In a way it makes the conversion from OpenStack and Harvester to Proxmox VE simpler, if the new VM has a new IP address.  As usual I will polish and refine my Netbox information, clean up runbooks stored in Redmine, but the theme will be making minimum-possible changes rather than implementing ambitious new ideas at the same time as the cluster conversion.

The question of Docker Containers...

https://pve.proxmox.com/wiki/Linux_Container

"If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox QEMU VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers."

OpenStack Zen containers were cool, and K8S obviously runs container workloads very well, however the above Proxmox information implies I will have to go back to the VMware era of setting up "container containers" to hold my Docker containers.  No big deal, but it will take some time to roll everything.  Generally I do "container containers" by installing a simple Ubuntu server, then running Docker off a NFS mount so there is no local state stored on the Ubuntu server (making it trivial to rebuild, and also making backups very simple as all state is just files on the NFS server).

In the next post, I will discuss the complicated order of operations due to various dependencies and operations requirements.

Proxmox VE Cluster - Chapter 001 - Why switch to Proxmox VE?

Proxmox VE Cluster - Chapter 001 - Why switch to Proxmox VE?

A voyage of adventure, moving a diverse workload running on OpenStack, Harvester, and RKE2 K8S clusters over to a Proxmox VE cluster.

Why switch from OpenStack / Harvester / RKE2 K8S, VMware, and other tech to a standard data center baseline of Proxmox VE?

• The hardware load and requirement is high for Harvester.  Harvester is awesome but some of my smaller nodes spend most of their CPU cycles and memory running Harvester itself rather than running my workloads.  A full Rancher cluster to control Harvester is very cool technology, but the hardware load is expensive.
• Harvester upgrades fail because the hardware load is too high.  Related to the above, I'm having trouble upgrading the more heavily loaded nodes because they can barely run Harvester at all, much less afford the extra system load to upgrade K8S.
• I can't really go backward to VMware.  Hardware compatibility lists, etc.  Technically I could spend the money but I don't think it would be worth it.
• I've learned everything I can learn from OpenStack and looking at trends it's time to 'jump ship' from OpenStack.
• Upgrades for OpenStack kolla-ansible are non-trivial and a little more interactive than I would prefer.  I'm running two OS clusters and will push the workload over to one cluster temporarily while upgrading the other cluster.  It takes a lot of time and sysadmin effort.
• Proxmox has some cool new features to experiment with, like native CEPH distributed cluster filesystem integrated into the system, and the very cool looking Proxmox Backup Server system.
• I want a "single pane of glass" to manage my cluster hardware with respect to monitoring, control, backup, etc.  I will put everything on Proxmox and control everything via a single Proxmox cluster.  I don't want a RKE2 K8S cluster AND a Harvester cluster AND two OpenStack clusters to manage, just one big Proxmox cluster, ideally.

Very high level plan for the overall conversion project:

1. Architecture level 1.0 will be a phased conversion of all workload into a large Proxmox VE cluster.  This will be an OpenStack / Harvester type design and workload wedged into fitting in to Proxmox VE.
2. Architecture level 2.0 will be system integration to make this a Proxmox-styled cluster, integrating with monitoring and automation and generally adapting the workload to make everything feel integrated rather than a different system's workload being temporarily run on Proxmox.
3. Architecture level 3.0 will be more R+D focused, advancing into interesting extra features that are Proxmox-specific, such as a cluster wide filesystem, the Proxmox Backup Server system, some interesting advanced networking ideas, new stuff in general.

How did I get here?

Can't figure out how to get where you're going, unless you know how you got where you are right now.

• Around the turn of the century, had bare metal Linux servers running LXC and also the FreeBSD equivalent.
• Around the 2010s, had some sysadmin level experience with VMware at work, and I signed up for the "ESXi Evaluation Experience" which gives you limited non-commercial license to pretty much the entire collection of VMware software.  This was pretty awesome for several years, although the continual drift in the hardware compatibility list and hardware requirements increasing dramatically over time, and generally being tired of paying for even a discounted VMware license, meant I moved away from VMware.
• Around the late 2010s / 2020 timeframe, replaced the VMware cluster with OpenStack.  OpenStack is FOSS, but the labor required to keep it up is expensive.
• In the early 2020s I started experimenting with RKE2 K8S on bare metal, and Rancher's Harvester bare metal virtualization solution.  Nice tech and works well, but it's designed for "larger" individual nodes than I can afford.

The above leads me to being interested in Proxmox VE to underlie my entire mini-datacenter.  I will eventually put everything on Proxmox except for my NAS and a stand alone Proxmox backup server.

Something to note about this series is the blog posts appear some time after "the action".  So as you read a new blog post, this all happened some weeks / months ago.  This gives me time to document things I've missed, circle back around, etc.  On the bad side I might miss some details of something I did months ago.  On the good side, I've circled back to document bugs and workarounds and any other areas of friction, which should save you, as the reader, some time if you implement a Proxmox cluster at your site.

Next post in the series will be a more detailed description of my Architecture level 1.0.

Rancher Suite K8S Adventure - Chapter 020 - Prepare Terraform for Harvester

Rancher Suite K8S Adventure - Chapter 020 - Prepare Terraform for Harvester

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

I don't like manually configuring things.  I like IaaC with templates stored in a nice Git repo, it eliminates errors, deployments are faster, fewer human errors, it's just all around better than mousing and typing a virtual infrastructure.  So today we prepare Terraform to work with Harvester, but first, some work with multiple cluster kubeconfig files.

Multiple Cluster Kubectl

Start automating by configuring kubectl to talk to multiple clusters.

Reference:

https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/

The good news is its pretty easy to configure multiple clusters into separate kubectl contexts.  The bad news is its easy to select different contexts at runtime, in fact its so easy to select different contexts, that there have been multiple headline news stories about devops who thought they were permanently erasing their test cluster deployment, only to rapidly discovery they were actually in their production context, resulting in some amazing news headlines about outages and deleted data.  So, keep your wits about you and be careful.  I will set up multiple contexts some other time.

One of the cultural oddities of the K8S community is they like to call the kubectl config file by the generic phrase "your kubeconfig file".  What makes that odd is most installs do not have a file named kubeconfig or dot kubeconfig or kubeconfig.conf or whatever.  On my Ubuntu system, kubectl's config file, aka the "kubeconfig file" is configured by a file located at ~/.kube/config

I will usually be working with Harvester, so in my ~/.kube directory I keep yaml files named rancher.yaml and harvester.yaml and I can simply copy them over the ~/.kube/config file.

In summary, make certain that running "kubectl get nodes" displays the correct cluster... 

Terraform

https://developer.hashicorp.com/terraform

Terraform is similar in concept to CloudFormation from AWS or HEAT templates from OpenStack.  You write your infrastructure as source code, run the template, and terraform makes the cloud gradually closely resemble your template.  Not a script, so much as a specification.

Install Terraform

I should have installed Terraform back when I was installing support software like kubectl and helm.  Better late than never...

https://developer.hashicorp.com/terraform/downloads

https://www.hashicorp.com/official-packaging-guide

The exact version of the Ubuntu package I'm installing is 1.3.9 as seen at

https://releases.hashicorp.com/terraform/

And I'm doing an "apt hold" on it to make sure its not accidentally upgraded.

Here is a link to the Gitlab repo directory for the Ansible helm role:

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/tree/master/roles/terraform

If you look at the Ansible task named packages.yml, the task installs some boring required packages first, then deletes the repo key if its too old, then downloads a new copy of the repo key if its not already present, gpg dearmor the key into 'apt' format, add the local copy of the repo key to apt's list of known good keys, install the sources.list file for the repo, does an apt-get update, takes terraform out of "hold" state, installs terraform version 1.3.9, finally places terraform back on "hold" state so its not magically upgraded to the latest version (1.4 or 1.5 or something by now).  Glad I don't have to do that manually by hand on every machine on my LAN, LOL.

Simply add "- terraform" to a machine's Ansible playbook, then run "ansible-playbook --tags terraform playbooks/someHostname.yml" and it works.  Ansible is super cool!

As of the time this blog was written, "terraform --version" looks like this:
vince@ubuntu:~$ terraform --version
Terraform v1.3.9
on linux_amd64
vince@ubuntu:~$ 

References

https://www.suse.com/c/rancher_blog/managing-harvester-with-terraform/

https://docs.harvesterhci.io/v1.1/terraform/

https://github.com/harvester/terraform-provider-harvester

https://registry.terraform.io/providers/harvester/harvester/latest