Zephyr RTOS with MCUboot "USAGE FAULT" after "Jumping to the first image slot", fixed.

Subjectively, what is it like to develop embedded software?

When things go wrong, its days of feeling like your hands are wrapped in mittens while wearing a blindfold, with your primary source of documentation being a Douglas Adams novel like "Hitchhikers Guide to the Galaxy".  Or even worse a Charles Stross "Laundry Series" novel.  Then, after days of struggle, the problem is fixed!  Development rapidly proceeds at a supersonic pace.  Until impact with the next block, stuck for a couple days of zero progress.  Such is life when doing embedded / IOT software development.

Take for example, this recent three day struggle:

Consider that for more than a year, you've been using older, lower RAM STM32 dev boards like the disco_l475_iot1, a truly fine board, other than only having about 128K of ram and Zephyr needing about 127.9K to run everything other than your application code, which will only require another 128K of ram LOL.  Or a classic F767 board like the nucleo_f767zi, a wonderful board and a joy to use, other than being wired ethernet instead of wifi.  The good news is the b_l4s5i_iot01a is a massive upgrade over the disco_l475_iot1 it has around six times the ram... awesome... its the board every "starved for memory" disco_l475_iot1 developer dreams of... this should be a simple port job ... Right?  Right?  Just recompile and upload?  Its never that easy in embedded development...

Not to put the cart before the horse, however, the problem I ran into is, in a technical sense only, documented at:

https://docs.zephyrproject.org/latest/services/device_mgmt/dfu.html

Where it explains you need to set the zephyr,code-partition in your DTS overlay file using a line looking something like:

zephyr,code-partition = &slot0_partition;

However, old dev boards automatically or automagically had that set, no developer effort required.  To make it crystal clear, although it disagrees with the docs above, code for old boards (admittedly, using older versions of Zephyr...) works perfectly without setting a code-partition variable.  I just verified using the latest version of Zephyr that a venerable 'F767 compiles and boots perfectly with MCUboot, no need to manually set a code-partition setting.

What happens if you do not configure a code-partition and compile for a nice new l4s5i board?  Certainly, not any error messages upon compilation, not the slightest indication of any future problems incoming.  Run your imgtool to sign the new image with firmware keys and mark as confirmed so MCUboot will be happy, upload to your dev board at address 0x08020000 as you do on that platform, reboot, and I kid you not, this is all you have to troubleshoot the problem:

(the usual MCUboot startup messages, it recognizes my primary image as good, etc)
I: Jumping to the first image slot
E: ***** USAGE FAULT *****
E: Attempt to execute undefined instruction
(boring register dump here)
E: Faulting instruction address (r15/pc): 0x0801bca8
and it crashes out and halts the system.

Notice that the PC address where it crashes out is NOT in the zephyr firmware slot ranges, its in the MCUboot range of addresses.  Interesting!  So its "obviously" a MCUboot bug, amirite?  Not so fast...

I think I have a good MCUboot compilation and source tree, in fact if I change the "west" command to force the same zephyr source tree to compile MCUboot for my old boards, they work fine.  So, a code bug in MCUboot?  I donno, there's not much space in "jump to the first slot image" to have a bug. 

Clearly (sarcasm) it can't be a problem with my code that is configured as per my old notes and runs fine on my old boards.  MCUboot is crashing at an MCUboot program counter address ONLY on the brand new dev board and fine on the other hardware.

Try one of several dev boards?  Sure, why not.  Turns out none of them work, all fail the same way.  OK then.  So its not hardware, its not Zephyr, its crashing at a MCUboot address, its not my build system, that works fine when told to compile for other boards...  I've "proven" it has to be a MCUboot problem.  

I spent a couple days learning about the innards of MCUboot.  Fun, but not productive and it didn't fix my problem.

After some days of messing around trying everything else under the sun, as typical for embedded development, I finally checked the Zephyr docs for device management (as opposed to using my documentation and source code comments and working source code) and sometime after Zephyr version 2.2.1 and before Zephyr 3.0.0 the online Zephyr docs were changed to reflect the need for a code-partition setting (apparently only for SOME boards, as most of my boards don't require this, and certainly never required it in the past).

Add that code-partition config option and recompile and imgtool sign the binary, upload with STM32CubeProgrammer, and we're off and running, where I should have been three days ago.

Two interesting summary points:  Good luck finding anything explaining this using Google, because I certainly did not, all I had to work with was the crash address 0x0801bca8 and that isn't finding much.  Also, yes, weird as it sounds, I have proven you can crash MCUboot while its running, by giving it a Zephyr firmware image that compiled perfectly yet was incorrectly configured.

By posting this on my blog maybe Google will index it, then people who experience crashes in MCUboot at address 0x0801bca8 on the newest L4 STM32 boards while trying to run old code that runs fine on older boards, will find this explanation and fix...

I could also tell the typical embedded development story of how my spyware blocking "lie protection" software thought the Ninja build tool, used by Zephyr, is spyware so it would crash the build for fun, not bother to log any errors or other explanation, just kind of randomly crash the suspected spyware program while in mid-build.  That was so much fun.

So... yeah.  That's what people mean when they talk about the Embedded IoT software development experience.  On the other hand, it's more fun than the above blog post implies, really, it is.  It's just that the struggles are real.

Notes on Node-RED development on a Raspberry Pi using Visual Studio Code

Notes on Node-RED development on a Raspberry Pi using Visual Studio Code

In the past, I developed Node-RED nodes by locally editing using Visual Studio Code and exporting the directory via NFS to a Raspberry Pi with the appropriate hardware installed, and the node was added to Node-RED via symlinks.  It worked, but it was brittle and there can be permissions issues and its annoying if the Pi reboots and cannot connect to the fileserver then flows will not start because the nodes are not present, it's just so tedious.

My new, improved development environment involves VSC's "remote-ssh" extension.

As always, detailed notes make it easier to set things up.  There are so many small details from configuring git on the RaspPi to removing the "nano" editor to make it easier to do command line git commits.

So, here are my cut and paste notes that start with a newly installed RaspPi (or rephrased, OS and Node-RED previously installed) and end with a working VSC development environment for Node-RED:

On Desktop:

Install Visual Studio Code extension “Remote - SSH”

https://code.visualstudio.com/docs/remote/ssh

https://code.visualstudio.com/docs/remote/linux

https://code.visualstudio.com/docs/remote/troubleshooting#_installing-a-supported-ssh-client

https://code.visualstudio.com/docs/remote/troubleshooting#_improving-security-on-multi-user-servers

Install Windows OpenSSH client

https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse?tabs=gui

Move SSH keys around for key-based SSH authentication

On New Remote:

On windows scp the key from the windows host to the new remote

scp c:\users\vince\.ssh\id_rsa.pub pi@new_remote:

then while logged into the new remote run:

cat id_rsa.pub >> .ssh/authorized_keys

Then test a login from windows to the new remote

ssh pi@new_remote

Create a SSH key on the new remote and add the .ssh/id_rsa.pub to GitLab

ssh-keygen, then add the key to GitLab

Prepare git on the new remote

On the new remote:

sudo apt-get install git

sudo dpkg –purge nano

git config –global user.email “vince.mulhollon@springcitysolutions.com”

git config –global user.name “Vince Mulhollon”

Install the git repo for the project on the remote

git clone the applicable Node-RED node project into ~ directory

Add git repo to installed Node-RED

cd ~/.node-red

npm install ~/node_git_repo_from_above

Restart node-red

./node-red-restart

Install standard.js in a project

npm install standard –save-dev

Then run it with “npx standard”.

Install standard.js extension in VS Code

“View” “Extensions”

Search for vscode-standard

Click install

Remember to install the standard engine as a devDependencies and it works automatically.

Anyway I hope these detailed notes help somebody do remote Node-RED development on a Raspberry Pi using Visual Studio Code.

Adventures of a Small Time OpenStack Sysadmin Chapter 057 - ELK Stack

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 057 - ELK Stack

Why?

An ELK system for end user systems such as server syslog messages, ethernet switch logs, and similar things, is technically not an OpenStack service.  However, as the overall project intention is to replace the complete functionality of a VMware cluster, and VMware clusters have Log Insight to centralize logging, I describe how I set up an ELK stack to replace my VMware Log Insight installation.

The Plan

After some research, the plan is to add a Docker host to hold an ELK stack all-in-one Docker container.  Unlike running Zun containers natively on OpenStack, a dedicated host can connect to my large and backed up NFS NAS for log storage, by bind mounting the Docker container volumes.  So I will roll out yet another Ubuntu 20.04 instance with Docker installed, and integrate it fully into the LAN including Active Directory SSO, roaming home directories, Zabbix and Portainer monitoring, etc.

Create a New Virtual Server

I have a nice checklist in the Ansible repo, so all new server rollouts on the OpenStack clusters are consistent and easy, as detailed below.

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/ubuntu-server-20.04.txt

In Todoist, which is an online and mobile to-do tracking app, I create a task for the new server with a due date in the future to schedule upgrades. In two months the to-do task will reach the top of the queue and I will upgrade and examine this system.  Documenting and scheduling future upgrades using Todoist takes about two minutes. 

In my dockerized Netbox installation for IP allocation and management, I select an IP address, create the server entry in Netbox, etc.  So, looks like elk.cedar.mulhollon.com will be at IP address 10.10.7.23.  This takes about two minutes.

Logged into the OpenStack controller, I create a new HEAT orchestration template, the server templates are all similar, aside from obvious differences such as IP address, security groups, so this process is fast and easy with search and replace in the editor.  This takes about five minutes depending on how "unusual" the configuration is.  If I'm configuring my fourth identical Active Directory Domain Controller it only takes a bit more than a minute.  This ELK project required some thought and some new ideas; so this new (to me) "filebeat" protocol between the clients and the ELK server's Logstash uses TCP port 5044, I guess I'll add that to the security group for this server.  Also Elasticsearch is legendarily memory hungry so I boosted this instance to 8 gigs of ram.  Flavors in OpenStack are so annoying, I wish I could do the VMware thing and simply type in any random amount of ram I feel like, without having to pre-define it as a flavor beforehand.  Computers eliminate some busywork, create more busywork, kind of a physics law of the conservation of mass, or conservation of mass of busywork...  Once started, the stack create process takes quite awhile in the background, while I do other things.  I would estimate I had about ten minutes of things to think about when designing my ELK container.

https://gitlab.com/SpringCitySolutionsLLC/openstack-scripts/-/blob/master/projects/infrastructure/elk/elk.yml

I do Active Directory DNS, by having Ansible run samba-tool to add forward and reverse DNS entries for each thing on the network, so I added a file roles/activedirectory/tasks/elk.yml and search and replaced the correct values.  Don't forget to add to roles/activedirectory/tasks/main.yml and of course run ansible-playbook ./playbook/activedirectory.yml.  This takes about two minutes of actual work, the script takes longer to run, but whatever.

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/roles/activedirectory/tasks/elk.yml

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/roles/activedirectory/tasks/main.yml

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/playbooks/activedirectory.yml

There's some "behind the scenes" configuration in Ansible that's abstracted away by adding the new Ubuntu image to the Ansible file inventory/ubuntu3.  Its a one line job.  Takes one minute.

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/inventory/ubuntu3

For this specific server, Ansible needs a playbook script file created, playbook/elk.yml.  Generally I pick one that's pretty close and edit it.  To start with this is a generic Docker host so I copy one and change some names.  Takes one minute.

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/playbooks/elk.yml

I use Active Directory and SMB file sharing on my network so I need to create a roles/samba/files/smb.conf.elk.cedar.mulhollon.com file to configure which directories are exported as shares.  I'd like to pretend I put great effort into configuring custom shares to export the server's logs and such all under reasonable security precautions and so forth; but most of the time "its just another docker host" so copy a similar predecessor and change some hostnames.  Yeah, I know, there's still commented out config options from back when Samba 4.5 was new, I've been doing this for awhile and could modernize the config files, sometime, in my infinite spare time...  Anyway setting up Samba for a new host takes about one minute.

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/roles/samba/files/smb.conf.elk.cedar.mulhollon.com

For some years I've been using Ansible to maintain my /etc/sshd/known_hosts file across my LAN.  It's all scripted up, requires minimal effort, just add another hostname to the script's list of hosts.  So I edit roles/ssh/files/ssh_known_hosts.sh to add the new server.  Takes one minute.

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/roles/ssh/files/ssh_known_hosts.sh

By now, OpenStack HEAT Orchestration should have completed the installation of my new server, so this paragraph is about prepping the new server to run the Ansible playbook on it later on which does all the "real work" of configuring the new server.  I configure HEAT to use my Ansible ssh key for initial public-key login, so from the Ansible user's login, ssh ubuntu@elk lets me log in.  I have to do some minor manual sshd config work for Ansible related purposes, and OpenStack cloud init always messes up the domain name for the new server for a variety of obscure reasons, so I need to manually "sudo hostnamectl set-hostname elk.cedar.mulhollon.com" which is how Ubuntu 20.04 does it (seemingly every unix-alike OS and every version of that OS has a different protocol).  The version of Ubuntu loaded into Glance on OpenStack is recent, but there are always patches that are even newer, may as well start from as clean and recently upgraded system as possible, so I spend a couple minutes running the usual "apt-get get update" "apt-get dist-upgrade" "apt-get clean" routine.  Finally, a quick reboot and the new server is ready for Ansible to configure it.  This generally takes about fifteen minutes, almost all of which is spent waiting for upgrading processes and rebooting delays, probably three minutes actual human labor.

After the new, clean, bare, unconfigured Ubuntu server completes its reboot, I "ansible-playbook ./playbooks/elk.yml" then Ansible does the vast majority of work required to integrate and harmonize with my existing network.  It would probably be a couple hours work to do manually, especially integrating with Active Directory using Samba, and it would be a very long error-prone checklist for a human to follow, but Ansible scripts never make mistakes.  There are a very small handful of manual tasks to perform after Ansible is done.  I could automatically install the latest Zabbix Agent V2 but I still consider it experimental until I get used to it, and as such I run a script that Ansible placed there ready for me to use to install it; I will eventually automate Zabbix Agent 2, after I am fully chill with it's use and behavior, seems OK so far...  Also given that I reconfigured the crypto options for SSH I create new SSH host keys (again using a script I wrote that Ansible places there ready for my use).  I generally get rid of the default "ubuntu" user because I have full SSO via Active Directory.  Also I feel weird about embedding my Active Directory "administrator" password in Ansible, so my entire Active Directory integration is fully automated with the exception of running a quick "net ads join -U administrator" and entering my domain's administrator password, although please remember that command line is NOT how to join a new Domain Controller to an existing domain, that's a similar but different one-liner.  Active Directory integration on Linux is sometimes sketchy, I've never found a way around rebooting to make everything about it work on a new install, so another, final, reboot of the new server.  This task overall is maybe 15 wall clock minutes, mostly watching automation do its thing, but I'd budget about four minutes of actual human labor.

Now that the new ELK server is integrated with my LAN, I need to work the opposite direction and integrate my LAN with the new ELK server, which is mostly accomplished by Active Directory but I do need to run roles/ssh/files/ssh_known_hosts.sh to pull the NEW ssh host keys off the ELK server and then I run the playbooks for OTHER hosts using the "—tags ssh" option to only update ssh configs on the other servers.  This is about one minutes work, its just running two scripts.

Usually, while Ansible is distributing the new SSH host keys, I fill time by messing around with Active Directory "ADUC tool" to enter a plain text description of the new server and enable trust delegation for SSO purposes.  Takes probably five minutes total to log into AD and mess around, after which Ansible is usually done with updating other server's SSH known host key list.

My next step is verifying SSO works.  Can I log into my new server and see my roaming NFS home directory without re-entering my password assuming I am already logged into a different server?  All my docker hosts share a NFS share that holds (and eventually backs up) the docker volumes, can I access it?  This testing takes only two minutes just to try and poke around.

Just a couple final integration tasks remain.  I use Zabbix to monitor operating systems so I configure Zabbix to connect to the new server, and I wait to verify good live data arrives in Zabbix.  I also use Portainer for remote control and monitoring at the Docker application level, so I need to install the agent for Portainer on the new host (its a docker container, as you'd expect) then add the new server as a docker host, verify it operates.  This probably takes ten minutes total.

The final task in rolling out a new server is git commit the OpenStack orchestration template and the Ansible playbook and other files.  This probably takes two minutes.

Overall using the power of OpenStack and Ansible, the time required to spin up a new usable server can be broken down into:

Documentation and Design 20 minutes

Operations "manual" labor 7 minutes

Integration and Testing 18 minutes

In the "bad old days" the operations category would have been "half a day" to scare up some hardware, burnin test it, verify the BIOS settings, slowly watch an OS installation progress bar creep across the screen, install the hardware in some permanent location.  You still have to do all that, once, for the cluster hardware, but once its done the additional labor to spin up a new server drops to, as seen above, seven minutes.  Which is quite an improvement from "half a day".

Install ELK stack on the new virtual server

I am using the "sebp" combined stack to spin up an ELK:

https://elk-docker.readthedocs.io/

https://hub.docker.com/r/sebp/elk/

Unusual Server Configuration Requirement

Another advantage of setting up a Docker host for the ELK stack is I have more control over the Docker environment that I would have with an OpenStack Zun container.  I have to make a custom mmap count limit setting as per: 

https://elk-docker.readthedocs.io/#prerequisites

and:

https://www.elastic.co/guide/en/elasticsearch/reference/5.0/vm-max-map-count.html#vm-max-map-count

I ran sysctl vm.max_map_count on the server as configured, and the default seems to be 65530 instead of the desired 262144.

Well, OK, fine, whatever, I can fix that.

In the short term I created a file /etc/sysctl.d/elk.conf containing one line

vm.max_map_count=262144

and run "service procps restart" (The documentation in /etc/sysctl.d/README.sysctl has a bug, the reload option doesn't exist LOL but restart works fine, when I get around to it, I will file a simple documentation-fix bug).

then I ran sysctl vm.max_map_count and now it shows the correct, larger, configuration.  Cool.

I documented that oddity in the Todoist task for this server.  The Todoist tasks act as a "runbook" to document exactly whats required to replicate a server installation, and usually there's not much oddity to document because Ansible Playbooks will take care of everything.  

In the long term, I created an issue in GitLab to add a "hardware" role for configuration challenges like this.

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/issues/9

Who knows, maybe by the time you read this, I will have already implemented this in Ansible?

Open many more TCP and UDP ports

I had to add some more ports to the security groups for the Orchestration Template.  No big deal, just edit and run the update.  It doesn't wipe and rebuild, it reasonably intelligently modifies in place.

Docker Run Script

My docker run script for the new ELK looks like this:

docker run \
  -d \
  --name elk \
  --restart=always \
  --log-driver local \
  --log-opt max-size=1m \
  --log-opt max-file=3 \
  -e TZ="US/Central" \
  -v /net/freenas/mnt/freenas-pool/docker/elk/elasticsearch:/var/lib/elasticsearch \
  -v /net/freenas/mnt/freenas-pool/docker/elk/backups:/var/backups \
  -p 5044:5044/tcp \
  -p 5601:5601/tcp \
  -p 9200:9200/tcp \
  -p 9300:9300/tcp \
  -p 9600:9600/tcp \
  sebp/elk:8.3.3

Filebeat

Back in the "old days" when I was getting started with ELK, we ran logstash on our servers and that pumped into Elasticsearch.  The modern solution seems to be using various *beat applications to pump data into Logstash which then pumps into Elasticsearch.  In the end I configured this differently, but whatever, in the narrative I set up Filebeat at this time, and someday in the future I might use it. 

Looks like the exact version of Filebeat is important for ELK, so I can't run filebeat locally because every little system would have a different version, and of course hardware devices like my managed ethernet switches will never run Filebeat as their firmware only supports syslog.  Therefore I will run a Docker Filebeat, on the same ELK server, of the exact matching version, and use multiple syslog inputs (for each specific syslog RFC format) to feed logs into ELK, and it looks like the highest shared matching version for both this specific ELK stack and Filebeat at the time of posting is:

https://www.docker.elastic.co/r/beats/filebeat-oss:8.3.3

Here are some Filebeat links for reference:

https://www.elastic.co/guide/en/beats/filebeat/8.3/filebeat-overview.html

https://www.elastic.co/guide/en/beats/filebeat/8.3/filebeat-input-syslog.html

https://www.elastic.co/guide/en/beats/filebeat/8.3/running-on-docker.html

My filebeat.docker.yml file looks like this:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
filebeat:
  inputs:
    -
      type: syslog
      format: rfc3164
      protocol.udp:
        host: "0.0.0.0:23164"
    -
      type: syslog
      format: rfc3164
      protocol.tcp:
        host: "0.0.0.0:23164"
-
      type: syslog
      format: rfc5424
      protocol.udp:
        host: "0.0.0.0:25424"
    -
      type: syslog
      format: rfc5424
      protocol.tcp:
        host: "0.0.0.0:25424"
output.elasticsearch:
  hosts: elk.cedar.mulhollon.com:9200

Note that I output directly into the elasticsearch which has no security theater on, by default.  The typical port for beats connected to logstash has some security theater on by default and it would be a bit of work to apply the self signed SSL cert; its just not worth the effort.  They are both running on the same server so a MITM attack seems unlikely, and the entire point of the Filebeat container is to import unsecured raw UDP logs so implementing security theater, or even real live SSL certs, between the Filebeat and the ELK would be a waste of effort.  I might still do that for the LOLs someday in my infinite spare time, just to have the experience of having done it.

My docker run script for Filebeat looks like this:

docker run \
  -d \
  --name filebeat \
  --restart=always \
  --log-driver local \
  --log-opt max-size=1m \
  --log-opt max-file=3 \
  -v /net/freenas/mnt/freenas-pool/docker/filebeat/config/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro \
 -p 23164:23164 \
 -p 23164:23164/udp \
 -p 25424:25424 \
 -p 25424:25424/udp \
  docker.elastic.co/beats/filebeat-oss:8.3.3

Configure Servers to Send Logs to ELK

To set up FreeBSD to send logs to ELK, see ansible roles/syslog/files/syslog.freebsd

*.* @elk.cedar.mulhollon.com:25424

Note the RFC5424 option in the RC file:

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/roles/syslog/files/rc.conf.d.syslogd.freebsd

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/roles/syslog/files/syslog.freebsd

To set up Ubuntu or anything using syslog-NG, see ansible roles/syslog/files/syslog-ng.conf.ubuntu which has a destination section similar to:

destination d_net { 
  syslog(
    "elk.cedar.mulhollon.com"
    port(25424)
    transport(udp)
  ); 
};

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/blob/master/roles/syslog/files/syslog-ng.conf.ubuntu

Conclusion

Obviously I had to do some set up in ELK such as adding filebeat* as my data view source, although note that I'm not trying to write an ELK tutorial.  Its pretty easy to create some searches and dashboards in ELK.  Anyway, in summary, it works, Cool!

Obviously its possible to make this MUCH fancier using SSL secured TCP transport instead of simple UDP, I could write entire posts about interesting ELK query and dashboard creation, it would be fun to follow up with setting up filebeat on individual servers to pump data into ELK, or converting the existing Filebeat gateway from pumping directly into Elasticsearch and use Logstash instead, but this is an excellent start to an ELK stack.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 056 - Prometheus

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 056 - Prometheus

During plan 3.0 I decided to either use or lose Prometheus.

Some observations about Prometheus

It works very well in my experimentation.

It replicates my Zabbix infrastructure without providing any additional value.

I need Zabbix to monitor the rest of my infrastructure, which is larger than my openstack cluster.  So I can't replace Zabbix with Prometheus (at least at this time, who knows in the future?)

As such I decided to remove Prometheus.

Kolla-Ansible is not an orchestration system, Ansible is merely a very fancy scripting language and set of libraries.  So removal of the /etc/kolla/globals.d/prometheus.yml file and running a kolla-ansible deploy will NOT remove the Prometheus installation although it will configure the entire rest of the system to NOT use Prometheus anymore in the future.

The solution to that problem, is to deploy, test that everything is working other than Prometheus, run something like "docker ps | grep prometheus" note a long list of about a dozen large containers providing the now-orphaned Prometheus service, then manually run many "docker stop prometheus-whatever" commands to shut down all the Prometheus containers.  The final step is a quick "kolla-ansible prune-images" with the really-really sure option to wipe the cached docker images for prometheus which will save a couple bytes of storage.

Tomorrows post will depend on what I do next to my home lab in my spare time.  I'm caught up to real time after a mere 56 posts.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 055 - Kolla-Ansible installation on Cluster 1 aka hosts 1, 2, and 3

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 055 - Kolla-Ansible installation on Cluster 1 aka hosts 1, 2, and 3

References

https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html

https://docs.openstack.org/kolla-ansible/yoga/reference/index.html

Pre-Kolla-Ansible Preparation

Ansible installs the OS packages.  Begin following along with the web instructions for Kolla-Ansible (linked above) at the VENV stage.

The new controller 1 will be on OS3 as root.

Python Prep

python3 -m venv /root/kolla-ansible

source /root/kolla-ansible/bin/activate

pip install -U pip

pip install 'ansible>=4,<6'

Kolla-Ansible Installation

This is installing Kolla-Ansible, not using Kolla-Ansible to install an OpenStack cluster (which will be done later)

pip install git+https://opendev.org/openstack/kolla-ansible@stable/yoga

mkdir /etc/kolla

chown root:root /etc/kolla

cp /root/kolla-ansible/share/kolla-ansible/etc_examples/kolla/* /etc/kolla

cp /root/kolla-ansible/share/kolla-ansible/inventory/* .

kolla-ansible install-deps

ssh-keygen to create a ssh key for root, and add to gitlab, so I can access the repos which store my configs and scripts and templates.

git clone the openstack-scripts repo and the glance loader repo.  Feel free to adapt these to your own needs.  I intentionally make these repos public to be seen and used.

mkdir /etc/kolla/globals.d

cp /root/openstack-scripts/backup/whatever/globals.d/* /etc/kolla/globals.d and edit them if necessary

mkdir /etc/kolla/config

cp -R /root/openstack-scripts/backup/whatever/config/* /etc/kolla/config and edit them if necessary

The online docs recommend setting some ansible config options; the instructions are for a non-VENV install, so I put my config in /root/ansible.cfg instead.

Edit /root/multinode, notice this inventory is for Cluster 1 which uses hosts 1, 2, and 3.

Final Pre-Deployment Config

kolla-genpwd and examine /etc/kolla/passwords.yml and note that some will need changing.

Be sure to edit /etc/kolla/passwords.yml line "keystone_admin_password" as you're not going to like the admittedly highly secure autogenerated password.

Also need to edit /etc/kolla/passwords.yml and edit the line kibana_password as you're not going to like the autogenerated kibana password.

Be sure to edit or verify /etc/kolla/config/ml2_conf.ini to set the network_vlan_ranges variable to a reasonable range of VLAN ids, such as bond12:1:1000 (I'm only using 10,20,30 thru 60 on interface bond12)

Run the Swift disk labeler on ALL swift disks on each host

apt install docker because the swift ring generator requires docker

Make sure /etc/kolla/config/swift is empty... for now.

Run the ring maker script in openstack-scripts to make the rings.

Configure Individual Kolla-Ansible Product Globals.d Files

For every service in the product reference, there will be a .yml file in /etc/kolla/globals.d for example neutron.yml to configure the OpenStack Neutron service.  There are no changes to the as-shipped globals.yml file other than one line for various bug reasons (explained later on).  My backups of these files work for me.  You may find my configurations inspirational or at least amusing as a starting point for your own cluster.  Some typical starting points for configuration:

kolla_ansible.yml

distro, vip addrs, and keepalived virt router ID

central_logging.yml

enable_central_logging: "yes"

That will eventually provide kibana on port 5601

Also need to edit /etc/kolla/passwords.yml and edit the line kibana_password as you're not going to like the autogenerated kibana password.

cinder.yml

Enable LVM backend, use the ssd volume group, and swift as backup driver.

I go back and forth on using swift or a shared NFS mount for backups; probably 51% better off with swift and tools like rclone.

glance.yml

Disable file backend, enable swift backend

heat.yml

Empty initial config

horizon.yml

Empty initial config

keystone.yml

Empty initial config

neutron.yml

network_interface: "whatever the bind interface is for the dual 10G on Prod VLAN"

neutron_external_interface: "whatever the bind interface is for the dual 1G"

kolla_internal_vip_address: "10.10.20.62" aka controller2.cedar.mulhollon.com

keepalive_virtual_router_id: "cluster number, 2 in this case"

Note that Kolla-Ansible uses the openvswitch agent whereas all my experience is with linuxbridge.

nova.yml

Empty initial config

swift.yml

I have to set up swift rings by hand as per:

https://docs.openstack.org/kolla-ansible/yoga/reference/storage/swift-guide.html

Note that the kolla-ansible docs provide an opaque process using docker to generate rings, with a pointer to the swift docs as an explanation, whereas the swift docs use a completely different method.  So that's confusing.

Work around the bootstrapping bug

Because I'm not modifying the globals.yml file and am doing all configuration in individual yml files in global.d, that triggers a bug:

https://bugs.launchpad.net/kolla-ansible/+bug/1970638

Instead of setting a dummy variable, I make ONE edit to /etc/globals.yml to set the base distro to ubuntu.  Now bootstrapping works...

Bootstrap

Make sure the venv is activated (if no (kolla-ansible)  in the prompt, source /root/kolla-ansible/bin/activate)

kolla-ansible -i ./multinode bootstrap-servers

Pre-Deployment Checks

Make sure the venv is activated (if no (kolla-ansible)  in the prompt, source /root/kolla-ansible/bin/activate)

kolla-ansible -i ./multinode prechecks

Deployment

Make sure the venv is activated (if no (kolla-ansible)  in the prompt, source /root/kolla-ansible/bin/activate)

kolla-ansible -i ./multinode deploy

Generate the admin-openrc.sh file:

kolla-ansible post-deploy

. /etc/kolla/admin-openrc.sh

Now copy /etc/kolla/admin-openrc.sh where-ever you need it.

Possibly create demonstration data:

/root/kolla-ansible/share/kolla-ansible/init-runonce

Or more likely just use the HEAT templates.

Prepare the CLI

If on OS3, Make sure the venv is activated (if no (kolla-ansible)  in the prompt, source /root/kolla-ansible/bin/activate)

pip install python-openstackclient -c https://releases.openstack.org/constraints/upper/yoga

Or, more likely:

run the install-cli script from openstack-scripts repo.

Local Configuration

Run network scripts to create provider nets and ip pools

Run the keypair script to upload ssh keys

Run the flavor script to upload the flavors

Run the glance-loader repo scripts to upload some usable install images

Run the heat scripts to set up all the projects

Use the web ui to add myself and admin user to all the projects with roles of admin for both

Create the /etc/kolla/admin-project-name openrc scripts for each individual project.  Look at how individual scripts in the projects handle the issue.  There are probably more elegant options to do this without individual files.

Run the individual heat project scripts to set up individual projects (had previously set up all the projects as a group)

Test the backup script in openstack-scripts

Run the heat scripts to set up some test instances.

Conclusion

I completed this step in a long afternoon.  It took hours using Kolla-Ansible to go twice as far as I got when configuring OpenStack by hand over the course of about two weeks.

After this monstrous long post, tomorrow will be a short post about Prometheus.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 054 - Bare Metal Install on hosts 1, 2, 3, and the docker USB server.

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 054 - Bare Metal Install on hosts 1, 2, 3, and the docker USB server.

https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html

https://docs.openstack.org/kolla-ansible/yoga/reference/index.html

Its the usual networking dance where netbooting is done on the non-LAG 1G ethernets and after a successful OS installation the networking is completely modified on the host and in the ethernet switch for LAG'd ethernets and VLANs.

Netboot and Install OS Ubuntu 20.04 LTS "Subiquity".

Leave the M2 alone, will configure it later, after the installer.

Remember to make the root partition on the controller 200G instead of 100G.

Username test, standard LAN password.

sudo apt-get update

sudo apt-get dist-upgrade

sudo apt-get autoremove

sudo apt-get clean

swapoff /swap.img, rm /swap.img, get rid of /swap.img file in /etc/fstab because I'm using a swap LVM partition not a swap file.

Configure the M2 on each host, using the labelswift script:

https://gitlab.com/SpringCitySolutionsLLC/openstack-scripts/-/blob/master/labelswift/label_swift.sh

Copy over the netplan setting up bond interfaces and VLANs and configure the ethernet switch.  After copying the file over, probably safest to log into the console via the IPMI KVM when altering the network.  Don't forget to ping test long packets to verify MTU settings...

https://gitlab.com/SpringCitySolutionsLLC/openstack-scripts/-/blob/master/backups/os3.cedar.mulhollon.com/netplan/netplan.yaml

Get LAN Ansible up and running to do basic configuration.  Fundamentally its just another Ubuntu server.  Cloud configs use "ubuntu" as a default username and for whatever reason when I set up physical servers I use the username "test".  Other than that, pretty uneventful.

Remember after setting up Zabbix to let it autodiscover drives and interfaces for a half hour or so after initial setup, but you NEED to shut off the autodiscovery rules in Zabbix for the OpenStack hosts before installing Kolla-Ansible or Zabbix will be flooded with virtual devices.

The docker server is a physical hardware Ubuntu install, netbooted on an Intel NUC mini-server.  The weird USB hardware devices plug into the NUC, the NUC hosts Ubuntu and Docker, the Docker containers run on NFS so nothing is stored locally on the docker server thus there really isn't anything to backup on the docker server itself.  There's not much else to say about the docker server.  It only exists because VMware USB passthrough is/was quite reliable but there is no such functionality on OpenStack.

Tomorrow will be a long post about Kolla-Ansible installation on Cluster 1.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 053 - Plan 3.0

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 053 - Plan 3.0

Here is a list of required tasks and optimistic goals for the Plan 3.0 era which involves redeploying the Plan 1.0 Cluster 1 hardware into a new Kolla-Ansible deployed cluster 1.  At the conclusion of Plan 3.0 there will be two mostly-identical Kolla-Ansible clusters.  I'm not sure there will be a Plan 4.0.  I will be able to upgrade the clusters one at a time and via the power of load balancing and redundancy its no big deal if I have upgrade "issues" on one small cluster at a time.  Kolla-Ansible seems pretty reliable and relatively low stress WRT upgrades-in-place... so far.

Main Task

The main task of Plan 3.0 is Cluster 1 will be reinstalled using Kolla-Ansible instead of hand rolling like the old Plan 1.0 installation, with the following minor exceptions from how cluster 2 was set up during the Plan 2.0 era:

Bare Metal

Root partition on the controller should be more like 200G than 100G based on experience.

At the end of the Plan 2.0 era, my 100G root partition utilization looked like this:

Compute node os4: 23.9 gigs

Compute node os5: 13.1 gigs

Controller aka host os6: 82.8 gigs (on a 100G root partition, eeeeeeek!)

So I would feel more confident with a 200G root partition on the controller.   The compute nodes seem stable, low % use, and may as well save their disk space for Cinder to use.

Bare Metal

Configure Zabbix BEFORE deploying Kolla-Ansible, then disable the automatic creation in Zabbix of drive and interface graphing, because using the cluster will spam Zabbix full of unusable drives and interfaces and polling that slows it down.

Ansible on Bare Metal

During the era of Plan 2.0, I improved my LAN-Ansible configuration for bare metal on OS4, OS5, and OS6.  Using my system-wide LAN Ansible to set my default "vim" editor options and similar tasks has not interfered with whatever Kolla-Ansible is doing to set up Lib-Kuryr and Cinder and whatever else Kolla-Ansible does to spawn an OpenStack cluster.  I have to be "reasonably" careful going forward but this idea of two Ansibles is now a production meme rather than an experimental meme.

Keystone

I can federate the two cluster's Keystones together, so they imply...

Cinder

At some point I would like to get NFS working with Cinder.  I have three NFS servers, one "real" backed up NAS, and two test/experimental smaller NFS servers.

Cinder

Push Cinder traffic off the management VLAN and onto the pre-existing Storage VLAN.

Swift

Push Swift traffic off the management VLAN and onto the pre-existing Storage VLAN.  I would imagine this would require new rings to be deployed etc so do this before trying to do anything important with Swift.

Upgrade Kolla-Ansible

I've upgraded Kolla-Ansible in the Yoga series (not across releases, yet) and I'd like to document the process.  Its mostly painless?

Neutron

Push Overlay traffic off the management VLAN and onto the pre-existing Overlay VLAN previously used for VMware and NSX and so on.

Designate

Modify DNS such that os1 domains on os2 cluster will be secondary DNS relationship with each other.  They will back each other up rather than ignore each other.

Mistral

Mistral is so cool and works so well, I want to go out of my way to find a use case for it.  I intend to completely automate provisioning down to one push of a button.  In the Plan 1.0 era, provisioning was completely manual followed by Ansible playbook-based configuration.  In the Plan 2.0 era, provisioning was handed by Heat Orchestration Templates and after that completed, Ansible playbook configuration was run, so its kind of one button, pause, second button.  My hope for Plan 3.0 is to use Mistral to automate the complete provisioning process down to one button press, possibly to even include updating status in my Netbox IPAM system.

Prometheus

Use it or lose it.  By the end of the Plan 2.0 era I have Zabbix monitoring everything at bare metal OS level and below which includes IPMI hardware sensors, and Centralized Logging more or less successfully and efficiently funneling all the Docker container logs into a working Kibana.  I also have Prometheus installed and am not using Prometheus in any fashion; so find a use for it or wipe it to save the disk space / cpu cycles.

Backups

Backups in general should be a focus of the Plan 3.0 cycle.

ELK for endusers

At some point in this conversion process from VMware to OpenStack I have to admit I need to replace VMware's LogInsight, which combined cluster and enduser logs, with something, and currently I intend to use a simple dockerized ELK stack for enduser only.  This will be just another HEAT Orchestration Template in the server project, and some Ansible changes to point syslog outputs to the ELK stack.  The only reason this was not done in Plan 2.0 era was simple hardware capacity, which I do have in the Plan 3.0 era.

Neutron

Plan 1.0 had each instance dynamically assigning IP addresses from the pool.  Plan 2.0 mostly orchestrates the same address regardless which cluster an instance is running on today.  In Plan 3.0 I still intend to keep pools on each cluster, only for testing and experimenting use, trying out new OS images or similar, "production" will continue the Plan 2.0 tradition of using "real" ip address assignments rather than pool assignments.

*aaS

I'm basically getting rid of almost everything *aaS provided by OpenStack as per previous discussion.

Bare metal Docker

I have three Docker containers that need to talk to bare metal USB interfaces connected to obscure hardware, the explanation and details are a long story...  I was running docker on OS2, OS3 for these containers in the Plan 1.0 era, but Kolla-Ansible wants to run its own Docker and something about it does not cooperate with my Portainer monitoring and something about Kolla's installation of Zun and Kuryr makes manual installed docker containers not work, its all so tedious sometimes.  "In the old days" on VMware, I completely successfully relied on VMware's USB passthrough service, but the OpenStack Nova does not do USB passthrough AFAIK.  So as part of Plan 3.0 I set up a tiny Intel NUC bare metal hardware server with all the USB ports stuffed full of interesting hardware devices and I run my bare metal Docker containers on that physical server.  Sometimes to go cloudy, or cloudier, you have to un-cloud some unusual workloads; weird but true.

Tomorrow, reinstall bare metal OS on hosts 1, 2, 3.

Stay tuned for the next chapter!