Adventures of a Small Time OpenStack Sysadmin Chapter 010 - Bare Metal OS install on OpenStack hosts 1, 2, 3

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 010 - Bare Metal OS install on OpenStack hosts 1, 2, 3

The actual OS installation is uneventful.  Its just Ubuntu 20.04 LTS aka the "Subiquity" network install, using Netboot.xyz.  As part of the install process you get to pick a default user and password, I uncreatively entered the username "test".

However, there are tasks to perform between installing the OS and installing OpenStack software.

After installation, I like to log in and do a standard upgrade process. 

sudo apt-get update

sudo apt-get dist-upgrade

sudo apt-get autoremove

sudo apt-get clean 

For some weird reason, I configured a swap partition in LVM but Ubuntu adds its own swapfile in the filesystem.  I get it; if you're running RAID you want to have your swap protected against drive failure, and allow hot swapping.  But its a little slower and I already set swap up in a LVM LV anyway.  So, "swapoff /swap.img" "rm /swap.img", remove swap from /etc/fstab.  This shrank my root partition size from 34 or so gigs down to about 2 gigs.

Some OpenStack docs imply apparmor is incompatible with OpenStack, some internet posts claim it doesn't matter, and Kolla-Ansible takes care of it automatically as part of the "bootstrap-servers" stage of installation.  For better or worse I shut down apparmor, "systemctl stop apparmor" "systemctl disable apparmor" "systemctl mask apparmor".

There's kind of a standard process to configure networking on new Ubuntu servers.  I set the hostname to the FQDN (important for ActiveDirectory join later on).  I set up the host for Ansible, life is too short to configure NTP and DNS resolvers and standard VIM options and SSHD options and stuff like that by hand.  After Ansible is done with basic server configuration, I usually re-generate my SSH host keys (because I have some specific requirements).  Anyway the point of this paragraph is now I have a generic Ubuntu 20.04 host that's integrated fully into my network environment, but it doesn't actually "do" anything, yet.

Pre-OpenStack, I had some rando hosts acting as a NTP cluster and a nice raspberry-pi based GPS clock, which worked OK.  My idea is to set up NTP on every host in both OpenStack clusters and have everything connect to that new NTP cluster.  This works really well!  Because almost everything on my network is configured by Ansible, I was able to push out the NTP config changes to all devices with minimal effort.  So I set up, tested, and modified my Ansible configs to make the OpenStack cluster host NTP on the "bare metal" level (as opposed to running a virtual machine instance on each host or something)

I used to do DNS by having all user devices point to the Samba AD DCs, which have a forwarder to two virtual machines I had on VMware but I hoped having six hosts doing DNS on bare metal would work even better.  For the use of OpenStack Designate, I set up bind9 DNS on each OpenStack host, fully recursive PLUS authoritative for the OpenStack Designate domains I had not at that time set up yet.  This eventually ended up not working very well at all because the Active Directory internal DNS resolver in Samba on my domain controllers is not particularly smart and will NOT cooperate with NS records pointing to other subdomains in the existing domain for use with Designate, but I'm getting WAY ahead of the story here.  It seemed a good design at the time.

Previously, on VMware, I ran both my DHCP servers as FreeBSD images with high availability and all that.  Experimented with VMware Fault Tolerance but it's really kind of overkill for the benefit.  Initially I intended to run DHCP on bare metal on my six hosts, of course ISC-DHCPD only supports dual host clusters so I intended to set up DHCP on all six hosts, twice, once as a primary and once as a secondary, and then I could manually log in and run a script or something to set a host as primary or secondary DHCP as needed.  This all seems like a lot of work to do by hand, and it is, but via the magic of Ansible scripting it was really almost no work at all!  However, for various reasons, later on I changed this architecture quite a bit and now have four DHCP servers, which is a long story I'll get to later on.

I had the interesting idea to replace USB pass-thru on VMware with running Docker on bare metal on the OpenStack cluster.  No need for openwebrx or homeassistant to have anything to do with the innards of OpenStack just run on the bare metal.  I always stored my volumes in NFS so the docker containers don't care where they're running as long as the host has NFS access, which my entire network does...  This works GREAT and is very fast.  There is a tiny problem in that Kolla-Ansible does not seem to be 100% compatible with this solution.  But we will get to that part of the story later on.  In the short term, this worked really well, very fast and efficient!

There was a minor problem discovered later on, where I manage Docker hosts using Portainer, which has a default port number of 9001.  Which, on an OpenStack cluster, is the port number of Designate.  So I reconfigured Portainer to run just the OS hosts on port 19001 instead of 9001 and that worked.  The good news about Kolla-Ansible is it puts the APIs on the virtual IP address so port 9001 should be open, I think.  The bad news is Kolla-Ansible is not entirely compatible with bare metal Docker especially after installing Zun and friends. 

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 009 - Prepare OpenStack hosts 1, 2, 3

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 009 - Prepare OpenStack hosts 1, 2, 3

Today was BIOS prep day and a hardware checklist.

Here's how to enable network UEFI PXE boot on a SuperMicro SYS-E200-8D aka X10SDV using a pretty standard Netboot.XYZ and ISC-DHCPD installation.

First, note that PXE boot mostly does not do VLAN tagging without "much extra effort" so you need a simple access port, even if the eventual network design is all LAG-bonded together and VLAN-trunking-tagged up.

In the BIOS config screen after booting:

"Advanced" "PCIe/PCI/PnP Configuration" area:

"Onboard LAN OPROM" set to EFI

"Onboard Video OPROM" set to EFI

"Network Stack" set to Enabled

"Ipv4 PXE Support" set to Enabled

"Boot" area:

"Boot Mode Select" set to UEFI

Set the boot orders to UEFI Hard Disk then UEFI Network.

Depending on BIOS version there may be some option you can set along the lines of "UEFI NETWORK drive priorities" where you can push netboot to work off the second 1G ethernet interface instead of timing out on the first 1G interface, or you could just plug the cable into the first interface...

Anyway, save configs, reboot, hit F12 for a netboot.

Its nerve-wrackingly incredibly slow, like a five to ten minute process from turning on in the IPMI KVM to a Netboot-XYZ menu prompt... but it does eventually work.  As I've often noted, computers only get slower over time.  This is not like booting MS-DOS 3.3 from a 360K floppy drive back in '84 which only took about five seconds.  Computers only get slower as technology "progresses".

I like to boot the system rescue CD provided by Netboot.XYZ and run "memtester 93000M" or so (depends how much memory you have, LOL).  All the classic old programs for burnin testing like Memtest86+ require old fashioned pre-UEFI booting and AFAIK there are no FOSS UEFI compatible burnin tools anymore, which is sad.  None the less I ran memory tests and other tests to verify the hardware is good.

Here's a checklist for hardware work required before OS installation:

Update all the physical labels; chassis, cables, everything.

Change the IPMI hostname in "configuration" "network" "hostname"

Update the BIOS as per the above notes to PXE netboot off the "new" Netboot.XYZ infrastructure.

Test the Netboot.XYZ and use the opportunity to run various hardware burnin or stress tests to verify memory is working, CPU fan can handle 100% CPU use, etc.

Add everything in Netbox ranging from machine name to interface configurations and IP addresses.

Update all the DNS entries forward and reverse for all the interfaces in ActiveDirectory.  Which in my case is a Samba cluster, so its just a shell script full of samba-tool lines.

At the end of this hardware checklist everything should be as ready as possible for "bare metal" OS installation.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 008 - Hardware decommission the old VMware ESXi hosts

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 008 - Hardware decommission the old VMware ESXi hosts

Removing hardware from a working cluster is, in some ways, harder and more nerve wracking than installing hardware.  I only accidentally pulled one unrelated connection loose.  There's only two types of sysadmin out there, ones that will tell you they accidentally disconnect the wrong cable once in awhile and those that will lie when they claim the never done that.  Well that's what "maintenance windows" are for, along with automated monitoring systems like Zabbix, and documentation, and proper cable labeling from my good ole Brady BMP-41 which is older than my kids but still works great.  You can't control random human accidents, but you CAN control how professionally and quickly you respond and resolve them... And its not that hard to do a better than average job.  Whatever, its fixed now, anyway.

I thermal camera scanned the innards of the hosts, nothing unexpected seen, powered them down, take the hardware out of the rack, great opportunity to blow all the dust out, wiggle and tap down all the connectors, generally inspect for any problems, none found.

I did run in to a problem some years ago with the SuperMicro SYS-E200-8D ten gigabit ports specifically (oddly, not the 1G ports right next to them) and some really nice looking CAT-6, CAT-7, CAT-8 whatever fancy ethernet cables I bought specifically for my fancy 10 gig switch because its cool, or at least expensive, so I have to use only the coolest cables.  Turns out some combination of mechanical tolerances on the server jacks and the ethernet plugs conspired such that with a minimal force on the plugs in certain directions even just the weight of the cable itself hanging, they would electronically disconnect although they were physically latched and that would drop the ethernet link causing me much headache until I tracked down the problem.  Luckily I don't use token ring networks or there would be tokens lying all over my wanna-be-data-center floor (that's supposed to be a joke...).  Anyway I made a point of buying mechanically better, although cheaper cables (LOL) that work perfectly under any condition and whenever I find one of those old faulty cables I chop it in half like a venomous snake and toss the halves out.  Somehow years later, today I found one.  An old-style unreliable cable, not a snake, I mean.  So I was happy to dispose of that unreliable cable.  Obviously it worked for many years, but I don't trust them, at least a quarter of them randomly disconnected in the past and I don't need that headache.  Does anyone periodically inspect their ethernet cables, at least on unclassified LANs?  I don't, but maybe I should?  Who has time for that is a better question... 

Anyway, the old $5 USB boot flash drives were starting to fail after half a decade of continuous use, so I pulled them, I'll be booting off the internal SSD.  Interesting anecdote, this model of SuperMicro motherboard and BIOS can NOT boot off the M.2 NVME device but can boot off the SSD.  I never ran into that situation before.  Interesting.

The hosts were taken apart, cleaned, re-labeled, dusted, generally serviced and reconditioned, I even scanned them with my thermal camera before pulling the plug to see if there's anything overheating to worrisome extent, now that its all reconditioned I treat it as if it's new hardware, time for some burn in testing.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 007 - OpenStack Infrastructure Prep - Netboot.XYZ

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 007 - OpenStack Infrastructure Prep - Netboot.XYZ

So... my netboot / PXE infrastructure was not working.

In the old days, setting up VOIP phones and turning piles of P75 desktops into clusters before the turn of the century, I had some hand written ISC-DHCP server configs and hand written TFTP configs, so this is all "old stuff" to me.

The thing that trips up newer sysadmins about network booting, it the utter lack of transparency.  Configure some rando DHCP server with mysterious incantations, then configure some rando TFTP server with similar weird configurations, then toss strange never before seen files into an unusual directory hierarchy, change some BIOS options on the host that most mere mortals fear to change, slap the reset button, and either it works and you celebrate, or it just freezes up and that's pretty much all you know until you whip out the ethernet protocol analyzer, or find someone else who previously "been there done that".  Set up Active Directory, eh, that's hard and a lot of work but there's entire books written about it.  Set up network booting, and not just "set up" but it actually works?  Now that's hard core, as I see things.

The way its supposed to work, from a very high level, is your DHCP server sends some interesting option codes to let devices know it can do this stuff.  Various BIOS settings, traditionally undocumented but sometimes you'll be surprised, tell your device to netboot off the network after the hard drive boot fails or maybe when you press F12 while booting or something more random and nonstandard, or maybe netboot every time, or whatever.  Anyway, assuming your device is set up correctly (and how do you know?) when the DHCP server responds to netbooting devices, the device is told the ip address of a tftp server to follow up with later, and what file to grab from that TFTP server, probably depending on some option codes the device sent to the server to tell if you should boot EFI or old fashioned BIOS (or I suppose you could hard code which depending on device MAC address, or lay down the law, its 2022 we only boot UEFI now a days and forever more).  Then your TFTP server tosses the netbooting device some magic file that does absolutely everything else.  So a problem could be at the DHCP server level, the TFTP server level, or a higher level "everything else" application level.  Without a protocol analyzer and someone who knows how to use it, best of luck to you, you'll need it.  As that famous Australian electrical engineer YouTube guy says, "its all a bit how-ya-doin" and when it works, its nice, but its not fun to troubleshoot as an overall system.

Anyway, specifically, Netboot.XYZ is a pretty awesome "everything else" for a netboot system, that can netboot install various operating systems and testing images.  Its really cool.  As an additional and much appreciated effort the project provides extras like the magic files required on your TFTP server to bootstrap into running netboot.xyz, and a Docker image holding a TFTP server and the higher-level stuff for the netboot system (menus and stuff).

https://netboot.xyz/

The system has been around for awhile, and technology has changed over the years, and the docs as shipped for the Docker image with instructions to "cut and paste" were for files that are no longer used for netbooting purposes and no longer shipped in the docker container.  So netboots would fail and I'd see requests for file download in the docker-provided TFTP server logs to filenames that do not exist in the container, at least no longer exist currently.

So I figured this all out, reverse engineered how the system should work based on other netboot systems I've used in the past, simplified the formerly rather elaborate ISC-DHCP server config down to about five minimal lines, and it all works, both legacy and UEFI netbooting!  I later went to submit a bug and a github pull request to improve the docs for everyone else, ended up meeting a guy trying to do the same thing as me, but running into the same problem, walked thru how to debug the innards of a Docker container to demonstrate the problem and the fix I came up with (wasn't Docker invented to eliminate the need to do exactly what we end up doing all the time anyway?) and to summarize a long story, I met some really cool folks and everyone's netbooting is working now, both PXE and legacy, AFAIK, and the docs for Netboot.xyz are now up to date for anyone who wants to install the overall system.

Netboot.XYZ is definitely the finest netbooting software I've ever used or even seen, far exceeding the abilities of anything I ever handcoded into working temporarily.  And the people who develop it are super cool to work with.  Overall an excellent experience.  And its free FOSS software.  They do have an OpenCollective, so send them money at:

https://opencollective.com/netbootxyz/donate

So, after a long days work, I can install Ubuntu 20.04 (LTS) over the LAN.

And everyone lived happily ever after.   Oh wait, I still have to finish converting my VMware cluster over to OpenStack.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 007 - OpenStack Infrastructure Prep - The Easy Stuff

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 007 - OpenStack Infrastructure Prep - The Easy Stuff.

Happy 4th of July to my readers.

With respect to this being "The Easy Stuff" … You will find out what that means tomorrow, LOL.

There's a lot of infrastructural grunt work to do before spinning up a new cluster.

Netbox and the managed ethernet switches need somewhat different VLANs.

Netbox needs new IP address space and new addressing scheme (may as well start over the right way based on past experience...)

Ansible can be partially prepped ahead of time with playbooks and roles ready to eventually fill out and apply.

The network has a wiki acting as a jump page, any technical device or server or lightbulb or whatever that has a URL on the network has a link to it from the main wiki jump page.  So, I need a new page on the wiki for OpenStack, may as well put a couple links to online documentation for emergency purposes etc.

The Todoist runbook and scheduled maintenance scheduling tool can be pre-populated with the new hosts.

I can even add some DNS entries for future use.

I could do this work later after installation begins, but pre-configuring forces one to think about their design very hard before its a critical path.  So I can think for awhile about how I should set VXLAN up, or use GENEVE or what for the overlay network, and I have time to think very hard about it because this pre-work is not on the critical path.  So in a way, preconfiguring is a waste of time, but looking at it another way, its actually the most valuable and rewarding time spent.

Is that everything for infrastructure?  Oh wait I forgot, my netboot / PXE infrastructure is not working.  I'll tell you that long story tomorrow...

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 006 - Shut down vSAN

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 006 - Shut down vSAN

The reasoning behind the vSAN shutdown being you can’t have vSAN cluster failures while shutting down individual ESXi hosts if you’re not using vSAN anymore.

Historically performance over NFS was not as good because my NFS server was provisioned for capacity not speed (so HDDs and 1G ethernets, instead of all SDDs and 10G ethernet).  NFS performance is/was temporarily usable, in the past I successfully moved entire workload to NFS or shut it down during appliance firmware upgrades and appliance hardware upgrades.

If I had to do it all over again I would not have spent time completely wiping vSAN after all workloads were removed.  I did not want false alarms popping as ESXi hosts shut down so it seemed a wise investment at the time.

The main problem I ran into and had to Google for was vCenter will not let you shut down a vSAN, and WILL NOT TELL YOU WHY, if the cluster has High Availability HA running.  If you shut off HA then vCenter will let you shut off vSAN.

The secondary problem I ran into was I wanted to wipe the vSAN partitions just to make sure nothing weird tried to resurrect itself after a reboot, for whatever reason.  vCenter does not make it easy to wipe partitions but it is eventually possible with some Google search work.  This was probably a waste of time, in retrospect.

After shutting down at the cluster level and host level and partition level, I also removed my vSAN VLAN from the ethernet network, removed DNS entries for the former, now deleted, dedicated VMK interfaces, removed allocations for the VLAN and the ip addresses and space and device interfaces from Netbox.  "Paperwork" took longer than the technical work as is usually the case.

Enough deleting and removing; time to start designing and adding.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 005 - The Big Cleanup

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 005 - The Big Cleanup

The primary narrative about the big cleanup of the old VMware cluster, is it was circular and forgetful.

If I had to do a big cleanup again, I'd apply more effort ahead or time to more fully organize two checklists, then wait a couple days to think it all over in the background, finally implement all at once in a sequential organized manner, ideally with no interruptions.

The first checklist would be all the virtual machines to be eliminated or consolidated or worked around.  Over time its very easy with "free" cluster resources to implement an entire machine to, essentially, do as little as replace a symlink or cronjob on another more logically located and designed machine.  There is also the eternal dilemma of an experiment that isn't working completely; should I wipe it and start fresh later, or save its current config and wipe it until later, or try to patch it up into working, or throw some effort into doing it correctly now?  Finally we all know that in reality there will be virtual machines that cannot be completely replicated by existing Ansible scripts, even if that is the standard or at least the goal.  So:  Put in the time to document and automated it now?  or Hope for the best and fix it later, I mean, I put it together once, I'm always more skilled over time, so I'm sure I can do it again probably faster and easier?  Also need a dependency tree... If you plan to wipe Apache Guacamole much later, and it was relying on a dedicated OpenLDAP server to provide its config, then you can't wipe the OpenLDAP today it has to go after the Guac server goes (back in the olden days this was the easiest way to configure Guac, the new web config is great and very usable...).

The second checklist would be an inverse of the provisioning procedure.  People are usually pretty good about documenting their provisioning procedure, usually not so good at documenting their DEprovisioning procedure.  It's very easy when working on this as a secondary task to forget or skip over steps due to an interruption.  Remove (or reassign?) the IP allocation in netbox.  Remove the VM in netbox.  Remove the computer from Active Directory.  Remove the DNS, forward and reverse or maybe none, from Active Directory DNS.  Remove from Todoist runbooks and scheduled maintenance planner.  Remove backup scripts and maybe immediately trash the backups or place for safekeeping.  Remove the Ansible playbook, maybe remove a no longer in use Ansible role, maybe other stuff.  If there was a link to it on the wiki, remove the link or place a note that its down until later.  Zabbix monitoring will go crazy when the host disappears, remove it.  If it hosted Docker containers the Portainer monitoring system will get agitated when the host disappears.  If you plan to ever reimplement the service/system, it needs to be ADDED to the Todoist project planner, or even worst and more confusing, maybe it needs to be added to the Redmine or maybe Asana, because the only thing better than having one project planning software tool is spreading tasks across three depending on higher-level project needs.

Anyway, took a couple days, part time, to properly and completely "spring clean" everything that could be deleted, even if only temporarily.

This should pay dividends later on; I'm not going to put time and effort into moving something that doesn't need to be moved because its not there anymore.

Stay tuned for the next chapter!