Rancher Suite K8S Adventure - Chapter 005 - Ubuntu 20.04 install on a Beelink Mini S 5095

Rancher Suite K8S Adventure - Chapter 005 - Ubuntu 20.04 install on a Beelink Mini S 5095

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

I do all installs using MatterMost Playbooks, this particular example is named "Bare Metal Ubuntu 20".  Life is easier with MatterMost.  When I figure out a convenient way to share Mattermost playbooks I'll add a link here.  If you've never used this software, you're missing out... I would describe it as similar to Slack meets an outline oriented todo app.

https://mattermost.com/

Hardware

The hardware I selected for my three node Rancher RKE2 cluster is Beelink Mini S 5095.  It's considerably cheaper than a Raspberry Pi, easier to get, Intel CPU based, much faster, much more storage, sadly the Raspberry Pi platform has been eliminated from the market by heavy competition and supply chain problems.  The Raspberry Pi was cool tech, for it's day, but its unavailable and/or too expensive now.  The Beelink is simply a mini size PC.  This particular model seems very popular in the set top box media player subculture, often used as a Plex or Emby front end instead of using Roku-type hardware.

https://www.bee-link.com/beelink-mini-s-n5095-mini-pc

BIOS configuration was uneventful.

Hit Del while booting to enter BIOS setup

Menu "Main" - Set hwclock to UTC time

Menu "Advanced" "MAC whatever IPv4 Network Configuration" - Configured Enabled, Enable DHCP

"Security" "Secure Boot" "Disable"

"Boot" - Setup Prompt Timeout change from 1 to 3, Quiet Boot Disabled

"Save and Exit" - "Save and Reset"

Reboot, hit del again to enter setup again (can't save and do a pxeboot in the same step, don't know why, doesn't really matter in the long run)

"Save and Exit" Boot Override "UEFI PXE"

I have a netboot.xyz installation on the LAN so I can PXE boot for OS installations.

https://netboot.xyz/

An example of how to configure the ISC DHCP server for PXE based netboot.xyz:

https://gitlab.com/SpringCitySolutionsLLC/dhcp/-/blob/master/header.dhcpd.conf.dhcp11

Likewise, if you use OpenStack and its HEAT template system, you can install netboot.xyz on Zun container service using this example:

https://gitlab.com/SpringCitySolutionsLLC/openstack-scripts/-/blob/master/projects/infrastructure/netbootxyz/netbootxyz.yml

OS

The Ubuntu 20.04 install was mostly uneventful, aside from the usual annoyances revolving around timezones settings, avoiding DHCP incorrect autoconfiguration, etc.  It's the usual Ubuntu experience.

In the Netboot.xyz menu: "Linux Network Installs (64-bit)"

"Ubuntu"

"Ubuntu 20.04 LTS Focal Fossa (Legacy)"

Don't use: "Ubuntu 20.04 LTS Focal Fossa (Subiquity)" - Install seems to hang at "hdaudio hdaudioCOD2: Unable to bind the codec"

"Install"

Reasonable defaults as usual

Full name for the new user: Ubuntu

Password for ubuntu user is "the standard LAN password", doesn't matter the username will be deleted after ansible connects it to AD anyway.

Force timezone to "Central" I don't live in Chicago LOL

The only software to install is OpenSSH server

Note that upon bootup it looks like a failed boot but ctrl-alt-f1 etc will work, very annoying.

Super annoying that it autoconfigures the enp2s0 ethernet as DHCP with no option to change. You can crash out of the DHCP setting and enter manual config mode.  If that fails and it installs in DHCP mode (super annoying) then:

boot, log in as ubuntu, sudo vi /etc/netplan/01-netcfg.yaml and do something like this:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp2s0:
      dhcp4: no
      addresses: [10.10.20.71/16]
      gateway4: 10.10.1.1
      nameservers:
        addresses: [10.10.7.3,10.10.7.4]

Then a quick "sudo netplan apply" and "ip addr" to verify and of course ssh in over the LAN to verify.

sudo reboot now

There is some weird bug where Ubuntu looks like the boot failed but as soon as you hit C-A-F1 you see a login, who knows.  Weird text console bug at bootup doesn't seem to matter.

verify SSH works over the lan as the ubuntu user which ansible will bootstrap into an AD connection

sudo shutdown now

At this point I physically installed the new server in the data center rack.  Properly label ethernet cables on both sides using the BradyLabel model 41 (yeah, its a bit of a brag, I really like this label maker), update the port name so the Observium installation makes pretty graphs with the correct server name, all the usual tasks.

Here is a link to the Ansible playbook for rancher1.  There's nothing special or unusual about it, its just a very small desktop PC being configured into a server.

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/playbooks/rancher1.yml

At this point the server is completely integrated in my infrastructure, although no "K8S specific" software has been installed.  AD SSO works, NTP works, Elasticsearch logging and metrics work, Zabbix monitoring works, etc.

Rancher Suite K8S Adventure - Chapter 004 - Rancher RKE2 IP Addressing and DNS

Rancher Suite K8S Adventure - Chapter 004 - Rancher RKE2 IP Addressing and DNS

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

Each machine in the cluster needs an ip address, so I added them in Netbox (Its a web-based IPAM system) and physically labeled each machine.

rancher1.cedar.mulhollon.com = 10.10.20.71

rancher2.cedar.mulhollon.com = 10.10.20.72

rancher3.cedar.mulhollon.com = 10.10.20.73

Ansible does my DNS configuration in Active Directory as seen at this URL:

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/roles/activedirectory/tasks/rancher1.yml

The cluster overall needs an entry where rancher.cedar.mulhollon.com points to all the load balancers in the cluster.  This results in several problems:

I've never really found "the right way" to store a DNS entry like that in Netbox.  Then again, its not using an ip address, so does it really "need" to be stored in Netbox?

The other problem is Samba round robin DNS is not supported LOL.

https://wiki.samba.org/index.php/Samba_Internal_DNS_Back_End#Limitations

So, that's annoying.  It'll always return the same answer in the same order.  I think this will be "OK" for access when/if the first host crashes, but it will be very suboptimal for load balancing because essentially all incoming traffic will go to only one cluster machine.

In the long run I've been considering setting up a "nice" load balancer on the large Harvester cluster to do proper load balancing for the small cluster and vice versa, or something like that.

Rancher Suite K8S Adventure - Chapter 003 - Software Version Selections

Rancher Suite K8S Adventure - Chapter 003 - Software Version Selections

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

I've been unsatisfied with how other examples online of Rancher Suite installations select their software versions.  I do not personally approve of "just install whatever is git branch latest or main", I've seen that blow up pretty badly.  Everybody knows Helm only supports the previous K8S API version, although that's wrong, because way back in the 3.0.0 days, Helm changed to supporting the last n-3 versions of the K8S API, and who can really memorize anyway that Helm 3.8.x only supports 1.20.x to 1.23.x (honestly, I had to look it up...).  A few minutes of planning ahead, can save many hours of troubleshooting later on.

Anyway.  The process I used was to open up diagram.net and create a big "flowchart" of the nine software components I plan to use, then sequentially cycle thru a list of URLs trying to condense my version selections into one coherent set where everything is predicted to work.

https://www.suse.com/suse-rancher/support-matrix/all-supported-versions/rancher-v2-7-1/

https://www.suse.com/suse-rke2/support-matrix/all-supported-versions/rke2-v1-24/

https://www.suse.com/suse-harvester/support-matrix/all-supported-versions/harvester-v1-1-1/

https://helm.sh/docs/topics/version_skew/

(And many other URLs unfortunately not recorded)

The one exception to the above is "officially" Harvester v1.1.1 only works with the v2.6 branch of Rancher, but unofficially extensive online research indicates everyone says it works just great for them.  Presumably support for Harvester v1.1.1 will only improve with time as Rancher's v2.7 branch develops.

As a disclaimer, this is being written in very early February 2023, on a posting delay to later that month, and you might be reading this post months or even years later, so don't laugh too hard at the "ancient" software versions listed above.

Another disclaimer is this is merely the first version of the plan.  I'm sure changes will be required.

Rancher Suite K8S Adventure - Chapter 002 - Plan 1.0

Rancher Suite K8S Adventure - Chapter 002 - Plan 1.0

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

Starting conditions

I have two small OpenStack clusters running independently.  This way I can move everything from cluster one to cluster two, upgrade cluster one from version 'yoga' to version 'zed', then test and migrate everything to cluster one, then upgrade cluster two, etc.  It has proven to be operationally highly effective and efficient.  All I'm being wasteful about is running two copies of OpenStack Horizon which is pretty lightweight.  VMware had excellent incredibly reliable USB passthru and nothing else has that feature, so I have a couple tiny dedicated bare metal servers connected to various USB hardware (SDR devices, IoT microcontroller hardware, some home automation using ZWAVE, etc).  I have a TrueNAS NAS machine for slow bulk NFS and iSCSI storage that works pretty well.

First, set up a Rancher cluster.  Small Intel SBCs are now much cheaper and somewhat more readily available than slower lower capacity Raspberry Pi SBCs, which seems crazy to me, but I guess we live in interesting times.  My intent is to set up a three node RKE2 cluster and install Rancher on that microcluster.

Second, I have enough spare parts laying around to set up a small Harvester HCI cluster.  All I intend to buy is new, fresh, NVME for Longhorn cluster storage.  I can backup my Longhorn data to my existing TrueNAS server.  These ancient Intel NUCs are not high performance; but should be more than adequate for testing and experimentation.  I intend to apply "real world" container and VM workloads to this small cluster and document my experiences.

Third, I will migrate the workload off one of the two OpenStack clusters to the other OpenStack cluster and convert that hardware to a medium size Harvester cluster.  Given the experience in step two above, it should be easy to migrate the real world workload off the remaining OpenStack cluster to the new Harvester cluster.

Fourth, as the remaining OpenStack cluster should be empty now, I will convert that hardware into additional capacity for the now "large" sized Harvester cluster.

I would anticipate along the way I will have several systems to convert and document.  For example, currently the primary NTP servers for the LAN are the OpenStack servers.  I'm tentatively considering changing that to use the Rancher cluster hosts as the primary LAN NTP servers; everything on the LAN is configured with Ansible so this should be a half hour job?  Another example of a larger scale system is I do logging using an ElasticSearch cluster and MetricBeat/FileBeat and Kibana, etc, and I'm curious how well, or if, K8S integrates into an ElasticSearch logging infrastructure.  Speaking of everything on the LAN being configured with Ansible, I wonder how (or if?) I will be able to integrate the 'Rancher Suite' with Ansible.

Ending Conditions

One micro RKE2 cluster dedicated solely to running Rancher.  A small Harvester cluster mostly for experimentation although I may distribute some infrastructure workload on it for increased system reliability.  A 'large' Harvester cluster for real workload.

After you experience the operational flexibility and increased reliability of having multiple clusters, its hard to go back to having only one cluster.  Technically workload would easily fit on the 'large' Harvester cluster, but I will keep the small Harvester cluster around for experimentation purposes, etc.

Rancher Suite K8S Adventure - Chapter 001 - Goals

Rancher Suite K8S Adventure - Chapter 001 - Goals

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

Goal

The goal of this new project is to convert a small OpenStack cluster to the Suse 'Rancher Suite'.  AFAIK the integrated set of Suse K8S cluster projects doesn't have a formal name so I'm calling it the 'Rancher Suite'.

History

Here's how I see the historical epic of virtualization.  (epic rant incoming)  Around the turn of the century was the rise of VM virtualization, run a hypervisor on the bare metal, which emulated/virtualized a VM, which runs a OS that usually doesn't entirely realize it's a VM, then run your apps on the OS.  I had admin access on some corporate VMware clusters and a little later, on some large corporate OpenStack clusters.  Meanwhile in the mid 00s, a different approach was becoming popular; chroot jails for enhanced security had been a thing since at least the 90s, maybe before, and the Linux kernel was "threaded" sorta by the mid 00s such that LXC containers could run multiple parallel runtime instances of the OS on top of one running kernel.  This is not theoretical; in the late 00s I was running multiple virtual machines on LXC on some Linux servers and it worked quite well.  Starting with LXC-type containerization, the addition of overlay filesystems, and, admittedly, a lot of hype, then Docker was born.  Then Docker led to Docker-Swarm (Docker-Swarm is like 'Fight Club' in that the first rule of Docker-Swarm is nobody talks about Docker-Swarm).  Docker-Swarm plus a lot of cool/complicated stuff leads to Kubernetes, which everyone abbreviates to K8S.

So, how to square the circle of virtualization?  Do we run virtual VMs on hypervisors and put workload on top of that, or run containers on K8S clusters and put workload on top of that?  The 'Marketing Department' name for trying to mix VMs with Containers seems to be 'Hyper Converged Infrastructure'.

I've tried several HCI strategies in the past.  Running hand configured VMs that host Docker, Docker-Swarm, and small K8S clusters like the K3S project worked fine on VMware and OpenStack.  Likewise, as you'd expect, there are automation 'solutions' for VMware and OpenStack that eliminate the hand configuration aspect by being, more or less, very small shell scripts that install preconfigured images for you.  Another way around it is OpenStack has the Zun project, which provides Docker Engine drivers for OpenStack resources such that containers can natively connect to OpenStacks block storage and network infrastructure, and this works very well for me.  Going the opposite direction, although I've never personally worked with it, the kubevirt.io project lets you run VMs on top of K8S containers, at least as I understand it.

An interesting looking HCI system I want to try with this project, is the Suse 'Rancher Suite' of tightly integrated projects.  Rancher is a controller and orchestrator of K8S clusters, Longhorn is a cool distributed block storage for K8S (Analogy is a vSAN for containers...) Harvester is a linux bare metal OS designed to hold K8S clusters and VMs, RKE2 (and, I suppose, RKE1) is a really nice K8S implementation.  What's noteworthy about this list of projects from SUSE, is they're all very compatible with each other, so I will deploy them as an integrated system.  So I will move the entire infrastructure from OpenStack to Rancher Suite.  What could possibly go wrong?

Why blog this project?

I think Rancher Suite sounds really cool.  If we're honest with ourselves most technology selections are based primarily on this criteria, although there's usually a dense wrapper of rationalization surrounding the decision.

The usual self-promotion, of course.  Need a C2C or W2 contract systems engineer?

I'm dissatisfied with the design process seen in YouTube and blog posts of similar deployments.  Just install 'latest' or 'main' branch of all 15 software components and hope for the best.  Its a moving target.  Its like throwing a net at a flock of birds after they've already scattered.  Why doesn't SUSE sell/promote a package deal named 'SUSE cluster 2023' where it's all guaranteed to "just work" together?

Likewise I'm dissatisfied with the demarcation point seen in other creative expositions.  Too many end with the admin running 'kubectl get nodes' then concluding the demonstration.  Wait a minute, I have many containers and VMs that need to be imported into the cluster and there's numerous other day to day 'operational concerns' such as logging and monitoring and backups that have not been addressed.  I'm reminded of the bad old days of Linux OS reviews in the 90s when the review would end at the conclusion of the installation process; wait a minute, I need to do actual stuff AFTER the install; where's the review about anything more than five minutes after the install?  LOL some things never change, do they?

Finally I write these things to informally document.  I have IPAM using Netbox, and I keep runbooks in MatterMost, and I keep detailed documentation in Redmine just like any other civilized engineer; but sometimes informal prose will effectively jog my memory.

Ubiquiti Firmware Update Requirements

I solved a Ubiquiti networking equipment problem yesterday.

The Ubiquiti system is very powerful and versatile, yet marketed as "just plug it together", so there are hidden and unclear pitfalls.

The initial problem statement was something like "device firmware updates are reported as available, but update process fails, no other known issues".

Eventually I located and fixed two separate problems:

Each hardware device in a Ubiquiti network participates fully in the network, having IP address, routing, etc.  This... also includes DNS.  Due to some re-addressing work all the Ubiquiti devices had the former IP addresses for DNS servers.  This had no apparent impact on day to day performance, until firmware upgrades were attempted.  It didn't seem to matter if the firmware was cached on the controller or not.  I'd theorize the firmware is going out to the internet to verify firmware keys and refusing to boot if unable to verify authenticity.  The symptom was attempts at firmware updates would take seemingly forever (at least more than five minutes per attempt per device), then fail and roll back to the old firmware.  Anyway, regardless of hypothesis about "how" it works, every individual device on the Ubiquiti network needed to be manually updated to the correct, new, working DNS server address.

Generally speaking, in the Ubiquiti web interface, you select a device, "Settings", "Network", and edit "Preferred DNS" and "Alternate DNS".

There's always some give and take in a centralized configuration system, or in this case, push and pull.  Take for example, Puppet, which has a remote agent periodically pull configs from a central server, vs Ansible, where a central server pushes configs out to remote devices.  Sure, in a simplistic way, Ubiquiti pushes:  You configure stuff in the web interface such as perhaps DNS server info, then "Apply Changes" and in a disturbingly long several minute process the device goes offline, then "Getting Ready" status, then comes back online.  However, how does it work under the surface?  There is a system wide setting, mysteriously named "Override Inform Host", that is pushed to every Ubiquiti device, asking it to pull from a new controller IP address.  I've used this many times in the past when installing new controllers.  I've upgraded from old to new cloud key to a much faster Docker installation of the controller in the past.  Anyway, for whatever reason this site setting had the old IP address for the inform host.  Not entirely sure how/why the system was working, but it was working perfectly... except for no upgrades are possible.

In the Ubiquiti web interface, select "Settings" then "System" then scroll to the very bottom "Override Inform Host" and if thats enabled, it should be the IP address of the current (or new?) controller.

One of the many advantages of a Docker install of the controller is access to the "unifi/log/remote" directory.  There should be a file for each device's IP address and the logs are interesting to watch during a firmware upgrade attempt.

So, the lesson learned, is a Ubiquiti network can operate perfectly normally with an incorrect IP address for DNS servers in each device AND an incorrect IP address as the system-wide "Override Inform Host", but either mistake will prevent successful firmware upgrades.  Who could have guessed?

Anyway, in the end, it all worked!

I wrote some digital input to USB HID support software using a $6 microcontroller

I made a fun, simply, working, weekend project:

https://gitlab.com/SpringCitySolutionsLLC/usb-hid-circuitpython-rp2040

Given a video game like DCS World or Elite Dangerous, it's fun to make a "real world" control panel with real world switches hooked up to the game, and people have been doing this for many years with "hundred dollar" digital input to USB adapter / appliances.

However, technology has improved over the years, and now for only $6, anyone can buy a Raspberry Pi RP2040 microcontroller running "CircuitPython" which is a limited Python language wedged into a microcontroller under a MIT free software license.  I decided to use the same MIT software license for my software linked above, please use it and have fun!  The RP2040 supports custom USB HIDs so I can easily emulate a multi-button joystick in software, and the buttons of this emulated joystick can then follow the switch inputs of the microcontroller's GPIO ports.

The hardware costs of the entire project are $6 for the microcontroller (plus a USB cable, I guess) and the software costs of the entire project, including the software I wrote, are zero dollars completely open source software.

I think this was a relatively simple software task.  Surprisingly, there do not seem to be many competing examples of this kind of code.  I tried to write the code and docs as obvious and clear and simple as humanly possible, although USB HID is ... non trivial so there is some inherent and unavoidable complexity in even the most minimal system.  I believe most home hobbyist simulation panel constructors are much handier with a miter saw or drill press than with a fancy software engineering IDE, and I also think they have more experience flying real airplanes than writing avionics software, so I'm trying to respect the needs of my end users, by making the code as simple as possible.  Just read the source code top to bottom, there's nothing tricky about it.

The demonstration code I have simply connects three toggle switches to a virtual joystick's first three buttons.  Obviously this could be trivially extended to something with more switches like a 'Huey' UH1-H helicopter's C-1611 Signal Distribution Panel, so you could flip a switch and turn the in-game helicopter's various radios on and off, or any other control panel in the game, such as the engine fuel control panel, or the overhead panel with the cockpit light dimmer dials.  Or this could be applied to any other simulation-type game, perhaps control your landing gear in Elite-Dangerous.

CircuitPython provides a very enjoyable software development experience.  A bit smoother integration with VS Code or Sublime Text would be nice.