Adventures of a Small Time OpenStack Sysadmin Chapter 001 - Intro

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 001 - Intro

"A long time ago in a galaxy far, far away..."

I personally set up a small VMware cluster of a mere six ESXi hosts a couple years ago.  I enjoyed the "cheap" ESXi-Experience as a VMware Users Group member for many years.  For those unfamiliar with the VMUG Eval-Experience or whatever its called now, for a very modest annual fee you get the "keys to the kingdom" of everything VMware has under a license only permitting educational use, and I learned a lot about VMware over the years.

I enjoyed the experience, I have to admit over years of experience that vSAN is the best SAN I've ever used, I scripted things that don't need to be scripted with Orchestrator, followed up on hardening suggestions from VMware Operations Manager, experimented with vCloud Director and Integrated Containers.  Log Insight is not necessarily better or worse than a FOSS ELK stack, but it is different.  Like many people I tried setting up NSX without understanding the basic concepts and thus obtained an education under fire of how NSX works; cool system, but a lot of mental gymnastics and work.  Upgraded vCenter many a time, had to reinstall it a couple times over the years, LOL.  Remediated quite a few ESXi hosts.  Like all the rest of you I was wondering how I'm going to use a flash-based web interface for vCenter when everyone removes flash support from their web browsers in a couple months, but VMware came thru seemingly at the last minute and everything worked perfectly on the new html5 web client.

Like everyone else who's honest about their cluster admin experience, there was plenty of comedy over the years.  Take a snapshot of a server before upgrading, upgrade, and wonder why there's a fraction of a terabyte snapshot file slowly filling up my NAS months later.  The usual fun with put host 2 in maint mode to shut it down and upgrade the memory, but instead pull the plug on host 3 and stick the memory in it instead, wonder why the cluster went nuts.  The odds of having five identical ethernet cables on six hosts across five VLANs all properly labeled and installed ... are about as low as you'd think, leading to fun with primary and backup links when upgrading the VSAN and vMotion interfaces to a new 10G ethernet switch.  Back in the old days (vers 5.x ?) you had to install special custom hardware drivers for 10G ethernet, which hilariously crashed ESXi when you upgraded ESXi far enough past those old drivers.  Much like how "everyone knows" that UPS hardware is less reliable than wall outlet power in the long run (well, depending where you live, LOL) VCHA sounds like a good idea but was always less reliable than a single VC install, I blew up VCHA quite a few times.  It seems that if something crazy could theoretically happen, it happened to me, I figured it out, and I fixed it.  I have this "VMware Sysadmin" stuff figured out.  Every time, eventually, everything always ended up working really well!

But all good things eventually end; VMware has been sold, and I'm not sure about the future, not as much money to be made in VMware consulting as I would have hoped (although you know where to reach me if you need skilled, experienced help with a VMware cluster).  Sounds like newer versions of ESXi will require a writable boot drive for logging that I don't want anyway and this cannot be wedged into my existing hardware without extensive effort.  The Eval-Experience program is a lot cheaper than list price, but it's still about half the annual cost of electricity to operate the cluster.  I am just done with VMware.    

I had some really positive experiences with OpenStack over the last decade or so.  Both at my home lab and when working for other people.  Eventually, I will start the project to convert my VMware cluster to OpenStack.  Eventually.  For a couple years now it has been "Eventually".  Meanwhile, other things to do.

Sometimes, "Eventually" happens faster than you'd expect.  One fine afternoon this winter, the power supply for host number three failed.  No big deal, that's why we keep spares.  About five minutes after replacing the power supply, I see in the IPMI KVM window, that host number three's boot device also failed some time in the past (no alarms in vCenter?), so now it can't boot ESXi.  No big deal, that's why we keep spares.  Next surprise discovery is the vastly underutilized yet very important netboot.xyz installation for PXE based operating system installation over the LAN is also not working.  I'm getting perturbed by that point.

I guess this is as good a point as any to initiate the big conversion project, replacing VMware with OpenStack?

Stay tuned for the next chapter!