Rancher Suite K8S Adventure - Chapter 004 - Rancher RKE2 IP Addressing and DNS

Rancher Suite K8S Adventure - Chapter 004 - Rancher RKE2 IP Addressing and DNS

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

Each machine in the cluster needs an ip address, so I added them in Netbox (Its a web-based IPAM system) and physically labeled each machine.

rancher1.cedar.mulhollon.com = 10.10.20.71

rancher2.cedar.mulhollon.com = 10.10.20.72

rancher3.cedar.mulhollon.com = 10.10.20.73

Ansible does my DNS configuration in Active Directory as seen at this URL:

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/roles/activedirectory/tasks/rancher1.yml

The cluster overall needs an entry where rancher.cedar.mulhollon.com points to all the load balancers in the cluster.  This results in several problems:

I've never really found "the right way" to store a DNS entry like that in Netbox.  Then again, its not using an ip address, so does it really "need" to be stored in Netbox?

The other problem is Samba round robin DNS is not supported LOL.

https://wiki.samba.org/index.php/Samba_Internal_DNS_Back_End#Limitations

So, that's annoying.  It'll always return the same answer in the same order.  I think this will be "OK" for access when/if the first host crashes, but it will be very suboptimal for load balancing because essentially all incoming traffic will go to only one cluster machine.

In the long run I've been considering setting up a "nice" load balancer on the large Harvester cluster to do proper load balancing for the small cluster and vice versa, or something like that.

Rancher Suite K8S Adventure - Chapter 003 - Software Version Selections

Rancher Suite K8S Adventure - Chapter 003 - Software Version Selections

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

I've been unsatisfied with how other examples online of Rancher Suite installations select their software versions.  I do not personally approve of "just install whatever is git branch latest or main", I've seen that blow up pretty badly.  Everybody knows Helm only supports the previous K8S API version, although that's wrong, because way back in the 3.0.0 days, Helm changed to supporting the last n-3 versions of the K8S API, and who can really memorize anyway that Helm 3.8.x only supports 1.20.x to 1.23.x (honestly, I had to look it up...).  A few minutes of planning ahead, can save many hours of troubleshooting later on.

Anyway.  The process I used was to open up diagram.net and create a big "flowchart" of the nine software components I plan to use, then sequentially cycle thru a list of URLs trying to condense my version selections into one coherent set where everything is predicted to work.

https://www.suse.com/suse-rancher/support-matrix/all-supported-versions/rancher-v2-7-1/

https://www.suse.com/suse-rke2/support-matrix/all-supported-versions/rke2-v1-24/

https://www.suse.com/suse-harvester/support-matrix/all-supported-versions/harvester-v1-1-1/

https://helm.sh/docs/topics/version_skew/

(And many other URLs unfortunately not recorded)

The one exception to the above is "officially" Harvester v1.1.1 only works with the v2.6 branch of Rancher, but unofficially extensive online research indicates everyone says it works just great for them.  Presumably support for Harvester v1.1.1 will only improve with time as Rancher's v2.7 branch develops.

As a disclaimer, this is being written in very early February 2023, on a posting delay to later that month, and you might be reading this post months or even years later, so don't laugh too hard at the "ancient" software versions listed above.

Another disclaimer is this is merely the first version of the plan.  I'm sure changes will be required.

Rancher Suite K8S Adventure - Chapter 002 - Plan 1.0

Rancher Suite K8S Adventure - Chapter 002 - Plan 1.0

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

Starting conditions

I have two small OpenStack clusters running independently.  This way I can move everything from cluster one to cluster two, upgrade cluster one from version 'yoga' to version 'zed', then test and migrate everything to cluster one, then upgrade cluster two, etc.  It has proven to be operationally highly effective and efficient.  All I'm being wasteful about is running two copies of OpenStack Horizon which is pretty lightweight.  VMware had excellent incredibly reliable USB passthru and nothing else has that feature, so I have a couple tiny dedicated bare metal servers connected to various USB hardware (SDR devices, IoT microcontroller hardware, some home automation using ZWAVE, etc).  I have a TrueNAS NAS machine for slow bulk NFS and iSCSI storage that works pretty well.

First, set up a Rancher cluster.  Small Intel SBCs are now much cheaper and somewhat more readily available than slower lower capacity Raspberry Pi SBCs, which seems crazy to me, but I guess we live in interesting times.  My intent is to set up a three node RKE2 cluster and install Rancher on that microcluster.

Second, I have enough spare parts laying around to set up a small Harvester HCI cluster.  All I intend to buy is new, fresh, NVME for Longhorn cluster storage.  I can backup my Longhorn data to my existing TrueNAS server.  These ancient Intel NUCs are not high performance; but should be more than adequate for testing and experimentation.  I intend to apply "real world" container and VM workloads to this small cluster and document my experiences.

Third, I will migrate the workload off one of the two OpenStack clusters to the other OpenStack cluster and convert that hardware to a medium size Harvester cluster.  Given the experience in step two above, it should be easy to migrate the real world workload off the remaining OpenStack cluster to the new Harvester cluster.

Fourth, as the remaining OpenStack cluster should be empty now, I will convert that hardware into additional capacity for the now "large" sized Harvester cluster.

I would anticipate along the way I will have several systems to convert and document.  For example, currently the primary NTP servers for the LAN are the OpenStack servers.  I'm tentatively considering changing that to use the Rancher cluster hosts as the primary LAN NTP servers; everything on the LAN is configured with Ansible so this should be a half hour job?  Another example of a larger scale system is I do logging using an ElasticSearch cluster and MetricBeat/FileBeat and Kibana, etc, and I'm curious how well, or if, K8S integrates into an ElasticSearch logging infrastructure.  Speaking of everything on the LAN being configured with Ansible, I wonder how (or if?) I will be able to integrate the 'Rancher Suite' with Ansible.

Ending Conditions

One micro RKE2 cluster dedicated solely to running Rancher.  A small Harvester cluster mostly for experimentation although I may distribute some infrastructure workload on it for increased system reliability.  A 'large' Harvester cluster for real workload.

After you experience the operational flexibility and increased reliability of having multiple clusters, its hard to go back to having only one cluster.  Technically workload would easily fit on the 'large' Harvester cluster, but I will keep the small Harvester cluster around for experimentation purposes, etc.

Rancher Suite K8S Adventure - Chapter 001 - Goals

Rancher Suite K8S Adventure - Chapter 001 - Goals

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

Goal

The goal of this new project is to convert a small OpenStack cluster to the Suse 'Rancher Suite'.  AFAIK the integrated set of Suse K8S cluster projects doesn't have a formal name so I'm calling it the 'Rancher Suite'.

History

Here's how I see the historical epic of virtualization.  (epic rant incoming)  Around the turn of the century was the rise of VM virtualization, run a hypervisor on the bare metal, which emulated/virtualized a VM, which runs a OS that usually doesn't entirely realize it's a VM, then run your apps on the OS.  I had admin access on some corporate VMware clusters and a little later, on some large corporate OpenStack clusters.  Meanwhile in the mid 00s, a different approach was becoming popular; chroot jails for enhanced security had been a thing since at least the 90s, maybe before, and the Linux kernel was "threaded" sorta by the mid 00s such that LXC containers could run multiple parallel runtime instances of the OS on top of one running kernel.  This is not theoretical; in the late 00s I was running multiple virtual machines on LXC on some Linux servers and it worked quite well.  Starting with LXC-type containerization, the addition of overlay filesystems, and, admittedly, a lot of hype, then Docker was born.  Then Docker led to Docker-Swarm (Docker-Swarm is like 'Fight Club' in that the first rule of Docker-Swarm is nobody talks about Docker-Swarm).  Docker-Swarm plus a lot of cool/complicated stuff leads to Kubernetes, which everyone abbreviates to K8S.

So, how to square the circle of virtualization?  Do we run virtual VMs on hypervisors and put workload on top of that, or run containers on K8S clusters and put workload on top of that?  The 'Marketing Department' name for trying to mix VMs with Containers seems to be 'Hyper Converged Infrastructure'.

I've tried several HCI strategies in the past.  Running hand configured VMs that host Docker, Docker-Swarm, and small K8S clusters like the K3S project worked fine on VMware and OpenStack.  Likewise, as you'd expect, there are automation 'solutions' for VMware and OpenStack that eliminate the hand configuration aspect by being, more or less, very small shell scripts that install preconfigured images for you.  Another way around it is OpenStack has the Zun project, which provides Docker Engine drivers for OpenStack resources such that containers can natively connect to OpenStacks block storage and network infrastructure, and this works very well for me.  Going the opposite direction, although I've never personally worked with it, the kubevirt.io project lets you run VMs on top of K8S containers, at least as I understand it.

An interesting looking HCI system I want to try with this project, is the Suse 'Rancher Suite' of tightly integrated projects.  Rancher is a controller and orchestrator of K8S clusters, Longhorn is a cool distributed block storage for K8S (Analogy is a vSAN for containers...) Harvester is a linux bare metal OS designed to hold K8S clusters and VMs, RKE2 (and, I suppose, RKE1) is a really nice K8S implementation.  What's noteworthy about this list of projects from SUSE, is they're all very compatible with each other, so I will deploy them as an integrated system.  So I will move the entire infrastructure from OpenStack to Rancher Suite.  What could possibly go wrong?

Why blog this project?

I think Rancher Suite sounds really cool.  If we're honest with ourselves most technology selections are based primarily on this criteria, although there's usually a dense wrapper of rationalization surrounding the decision.

The usual self-promotion, of course.  Need a C2C or W2 contract systems engineer?

I'm dissatisfied with the design process seen in YouTube and blog posts of similar deployments.  Just install 'latest' or 'main' branch of all 15 software components and hope for the best.  Its a moving target.  Its like throwing a net at a flock of birds after they've already scattered.  Why doesn't SUSE sell/promote a package deal named 'SUSE cluster 2023' where it's all guaranteed to "just work" together?

Likewise I'm dissatisfied with the demarcation point seen in other creative expositions.  Too many end with the admin running 'kubectl get nodes' then concluding the demonstration.  Wait a minute, I have many containers and VMs that need to be imported into the cluster and there's numerous other day to day 'operational concerns' such as logging and monitoring and backups that have not been addressed.  I'm reminded of the bad old days of Linux OS reviews in the 90s when the review would end at the conclusion of the installation process; wait a minute, I need to do actual stuff AFTER the install; where's the review about anything more than five minutes after the install?  LOL some things never change, do they?

Finally I write these things to informally document.  I have IPAM using Netbox, and I keep runbooks in MatterMost, and I keep detailed documentation in Redmine just like any other civilized engineer; but sometimes informal prose will effectively jog my memory.

Ubiquiti Firmware Update Requirements

I solved a Ubiquiti networking equipment problem yesterday.

The Ubiquiti system is very powerful and versatile, yet marketed as "just plug it together", so there are hidden and unclear pitfalls.

The initial problem statement was something like "device firmware updates are reported as available, but update process fails, no other known issues".

Eventually I located and fixed two separate problems:

Each hardware device in a Ubiquiti network participates fully in the network, having IP address, routing, etc.  This... also includes DNS.  Due to some re-addressing work all the Ubiquiti devices had the former IP addresses for DNS servers.  This had no apparent impact on day to day performance, until firmware upgrades were attempted.  It didn't seem to matter if the firmware was cached on the controller or not.  I'd theorize the firmware is going out to the internet to verify firmware keys and refusing to boot if unable to verify authenticity.  The symptom was attempts at firmware updates would take seemingly forever (at least more than five minutes per attempt per device), then fail and roll back to the old firmware.  Anyway, regardless of hypothesis about "how" it works, every individual device on the Ubiquiti network needed to be manually updated to the correct, new, working DNS server address.

Generally speaking, in the Ubiquiti web interface, you select a device, "Settings", "Network", and edit "Preferred DNS" and "Alternate DNS".

There's always some give and take in a centralized configuration system, or in this case, push and pull.  Take for example, Puppet, which has a remote agent periodically pull configs from a central server, vs Ansible, where a central server pushes configs out to remote devices.  Sure, in a simplistic way, Ubiquiti pushes:  You configure stuff in the web interface such as perhaps DNS server info, then "Apply Changes" and in a disturbingly long several minute process the device goes offline, then "Getting Ready" status, then comes back online.  However, how does it work under the surface?  There is a system wide setting, mysteriously named "Override Inform Host", that is pushed to every Ubiquiti device, asking it to pull from a new controller IP address.  I've used this many times in the past when installing new controllers.  I've upgraded from old to new cloud key to a much faster Docker installation of the controller in the past.  Anyway, for whatever reason this site setting had the old IP address for the inform host.  Not entirely sure how/why the system was working, but it was working perfectly... except for no upgrades are possible.

In the Ubiquiti web interface, select "Settings" then "System" then scroll to the very bottom "Override Inform Host" and if thats enabled, it should be the IP address of the current (or new?) controller.

One of the many advantages of a Docker install of the controller is access to the "unifi/log/remote" directory.  There should be a file for each device's IP address and the logs are interesting to watch during a firmware upgrade attempt.

So, the lesson learned, is a Ubiquiti network can operate perfectly normally with an incorrect IP address for DNS servers in each device AND an incorrect IP address as the system-wide "Override Inform Host", but either mistake will prevent successful firmware upgrades.  Who could have guessed?

Anyway, in the end, it all worked!

I wrote some digital input to USB HID support software using a $6 microcontroller

I made a fun, simply, working, weekend project:

https://gitlab.com/SpringCitySolutionsLLC/usb-hid-circuitpython-rp2040

Given a video game like DCS World or Elite Dangerous, it's fun to make a "real world" control panel with real world switches hooked up to the game, and people have been doing this for many years with "hundred dollar" digital input to USB adapter / appliances.

However, technology has improved over the years, and now for only $6, anyone can buy a Raspberry Pi RP2040 microcontroller running "CircuitPython" which is a limited Python language wedged into a microcontroller under a MIT free software license.  I decided to use the same MIT software license for my software linked above, please use it and have fun!  The RP2040 supports custom USB HIDs so I can easily emulate a multi-button joystick in software, and the buttons of this emulated joystick can then follow the switch inputs of the microcontroller's GPIO ports.

The hardware costs of the entire project are $6 for the microcontroller (plus a USB cable, I guess) and the software costs of the entire project, including the software I wrote, are zero dollars completely open source software.

I think this was a relatively simple software task.  Surprisingly, there do not seem to be many competing examples of this kind of code.  I tried to write the code and docs as obvious and clear and simple as humanly possible, although USB HID is ... non trivial so there is some inherent and unavoidable complexity in even the most minimal system.  I believe most home hobbyist simulation panel constructors are much handier with a miter saw or drill press than with a fancy software engineering IDE, and I also think they have more experience flying real airplanes than writing avionics software, so I'm trying to respect the needs of my end users, by making the code as simple as possible.  Just read the source code top to bottom, there's nothing tricky about it.

The demonstration code I have simply connects three toggle switches to a virtual joystick's first three buttons.  Obviously this could be trivially extended to something with more switches like a 'Huey' UH1-H helicopter's C-1611 Signal Distribution Panel, so you could flip a switch and turn the in-game helicopter's various radios on and off, or any other control panel in the game, such as the engine fuel control panel, or the overhead panel with the cockpit light dimmer dials.  Or this could be applied to any other simulation-type game, perhaps control your landing gear in Elite-Dangerous.

CircuitPython provides a very enjoyable software development experience.  A bit smoother integration with VS Code or Sublime Text would be nice.

Solution for upgrading ISM43362-M3G-L44 wifi module firmware on a ST B-L4S5I-IOT01A dev board

Why I'm trying to upgrade the wifi module's firmware.

I have been experiencing problems with network offload on Zephyr RTOS using a ST B-L4S5I-IOT01A dev board which has an onboard Inventek ISM43362-M3G-L44 wifi module.  Its not completely dead, I have it successfully connecting to my wifi, the offload processing works well enough that I can ping the module and telnet to the wifi coprocessor, but there's something not "connecting" between the processors in the sense that I can't get telnet to the dev board shell or anything else on Zephyr to connect in a conceptual sense to the wifi.  Offload "works" in that the C language callbacks fire and I can report my DHCP provided wifi IP address, which is how I can ping it, but its like the application TCP/UDP sockets are refusing to connect to the wifi interface, even though they kind of are doing that; most aggravating.

For a variety of reasons, I decided to upgrade the wifi firmware.  I've read elsewhere about how Azure connectivity with this board will NOT work unless the wifi firmware is upgraded, so that is very interesting although I'm not using Azure in this application.  Another reason is the usual troubleshooting procedure of I've done everything easier and more of an obvious solution already.  My final justification is I have no idea what version of the wifi coprocessor firmware was shipped on the board, so at least after an upgrade I'd know for certain what version I'm running, which probably provides at least some troubleshooting value.  

I did not decide to upgrade the wifi firmware for fun or because I was bored, LOL.

Description of the wifi module

The wifi module is similar to an ancient Hayes-compatible phone modem, serial attached with "AT" command set, that connects to wifi networks instead of phone lines.  Also instead of connecting to the device using a serial port like a traditional phone modem, this module has the option of SPI connection for higher speed, and on this ST dev board, it connects via that SPI interface.  So in one way of looking at it, it's exactly like a phone modem, except when looking at it in another way, everything is different from a phone modem (LOL?).

Finding references, instructions, tools for the upgrade process

If you google for the wifi module and firmware, which is a pretty good strategy most of the time, one of the top results is this URL:

https://www.inventeksys.com/es-wifi-firmware/

Sadly, the most recent firmware on that page is v3.5.2.3 from Oct 10 2016.  That's not going to help.  I don't think the CPU on this dev board had been released back in 2016.

Further research led me to:

https://www.inventeksys.com/ism4336-m3g-l44-e-embedded-serial-to-wifi-module/

From that device page, you can find a link to another, separate and newer, firmware upgrade page for the device:

https://www.inventeksys.com/iwin/firmware/

The highest version available on THIS download page is 3.5.2.7, that's cool.

Next we have to upload the firmware into the wifi module.  If you have a serial connection, with a switch selectable USB port or similar like the Arduino shields provide, this task is pretty easy.  If you have a PCB standard JTAG pinout and a JTAG programmer this task is also pretty easy.  The ST board has none of the above.  What the ST provided upgrade solution does have, is a very small binary program provided by ST, that enables SPI passthrough so STM32CubeProgrammer or anything else that speaks ST/Link can program what it thinks is the STM32 flash, but its actually programming the flash in the wifi module.  What could possibly go wrong? 

The software to upload the firmware is on the ST website, in the "docs" section not "tools":

https://www.st.com/en/evaluation-tools/b-l4s5i-iot01a.html#documentation

The best instructions for the firmware upgrade seem to be in the upload tool's README file, although Inventek provides a link to a .pdf which has some alternative options:

http://www.inventeksys.com/iwin/wp-content/uploads/Firmware-Upgrades.pdf

How the upgrade process was supposed to work

I followed the README upgrade instructions from ST.  First you download the firmware zip from inventeksys.com, using the correct, newer download page, not the Google-provided older download page.  Then, unzip and change the filename from *.bin.rename to *.bin, because of crazy windows anti-virus software, the firmware has to be double wrapped to get past.  Then copy the renamed firmware into the bin directory of the ST provided firmware upgrader.  The firmware upgrader batch file uploads a tiny bin file that puts the dev board into SPI passthrough, and then uses the CLI for STM32CubeProgrammer to burn the firmware into what it thinks is the STM32L4 chip but is actually the passed-through wifi module's flash chip.  It's a rather short and simple windows batch file.

Something undocumented, but cool, is when the wifi module is in passthrough mode the blue LED turns on.  At least ... I think that's what the LED implies.  The STM32L4 firmware is a binary blob and I didn't run the debugger on it so who really knows?

What actually happened during the upgrade process

There is a slight problem, if you run update_Wifi.bat on a Windows 11 PC with the newest version of STM32CubeProgrammer and newest ST/Link firmware on the dev board, the dev board bootloader pass thru is downloaded to the STM32L4 and it runs successfully, OK so far.  Then the command to bulk erase the Inventek module's flash runs, and eventually times out while in the process of erasing the flash (whoops).  Then the program stage runs, tries to re-issue the erase command before writing the firmware, however the flash chip is still erasing from the previous bulk erase command, so it crashes the write process and never writes the firmware to the wifi.  So now we have a wifi module with erased firmware flash.  That's not good.

It gets somewhat worse.  If you reinstall MCUBoot and a Zephyr image, the MCUBoot will jump to the Zephyr image and silently fail and lock up the STM32L4 during initial bootup of Zephyr before it even jumps to main(), so time can be wasted trying to troubleshoot your reinstall, well, obviously I forgot to ... confirm the Zephyr image or something.  So the board seems bricked and everyone is unhappy.

Unbricking and successful wifi module firmware upgrade process

The board, however, is not bricked.  I would theorize the problem is older versions of STM32CubeProgrammer and/or the ST/Link firmware would not return from the erase command until the erase was fully complete and/or the default delay to wait for bulk erase completion before timing out was longer.  Also I don't really see the point of bulk erasing the wifi flash if the next step in the batch file, the write step, begins by erasing the flash anyway.

So use vim or whatever editor you use to edit this line in update_Wifi.bat to skip the bulk erase step by commenting out, prefix the line with a "rem":

rem %PROGRAMMERPATH% -c port=%COMPORT% br=115200 p=EVEN db=8 sb=1 fc=OFF -e all

Then re-run the uploader script and it seems to work quite well, instead of timing out and screwing everything up past the failed erase command, it skips the bulk erase that doesn't work anyway, and runs the next command in the uploader script which successfully erases the flash, writes the firmware, and does a verify read pass in about two and a half minutes.

After this, unplug and plug the board back in, re-upload the MCUBoot firmware, re-upload the Zephyr RTOS image, and it works exactly the same as before.  Unfortunately this firmware upgrade had zero effect on my Zephyr networking offload problems.  But at least the dev board isn't bricked anymore, and I know it has the latest wifi module firmware version installed.

Summary

If you're trying to upgrade the firmware on a Inventek ISM43362-M3G-L44 board which is SPI attached on a ST B-L4S5I-IOT01A dev board, you'll run into two problems.  There are two firmware download pages at Inventek and Google will send you to the older download page with half a decade old firmware that doesn't work with the ST uploader.  The other problem is the ST batch file to automate firmware upgrades will brick your board, although with some extremely minor editing the script will work and un-brick your board and successfully upgrade your wifi module firmware.

This firmware upgrade process, overall, was somewhat more exciting than is necessary.