Adventures of a Small Time OpenStack Sysadmin Chapter 017 - OpenStack Swift Object Store

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 017 - OpenStack Swift Object Store

My primary reference for installing Swift is here:

https://docs.openstack.org/swift/yoga/install/

Note there is quite a difference between disk partitioning and mounting for hand installed Swift and Kolla-Ansible installed Swift.  For hand installed swift, format a disk on /srv/node/swift as XFS format as per the online docs, just expect to do it completely differently for Plan 2.0 Kolla-Ansible installation later on.

The controller instructions for Ubuntu on page:

https://docs.openstack.org/swift/yoga/install/controller-install-ubuntu.html

Suggest to install a Python2.7 library that no longer exists, simply specify python3-keystoneclient instead.  After figuring it out on my own and attempting to file a documentation improvement bug report, I found someone else already filed a doc-improvement bug a mere two years ago.  Such is life when administering OpenStack sometimes the best doc are to read all five hundred open bug reports before starting installation...

https://bugs.launchpad.net/swift/+bug/1893769

When I was testing and experimenting, partially following along with the docs at this URL:

https://docs.openstack.org/swift/yoga/install/verify.html

I ran into a situation where the above link recommends using demo-openrc but that user didn't have access to see status, so I ended up using admin-openrc instead.  I messed around with things enough that I'm not sure who did it wrong, myself or the docs, so I didn't file a bug, and as these blogposts are being written as a retrospective about the past based on my extensive project documentation notes, by the implementation of Plan 3.0 I no longer have access to a hand installed Swift so I can't replicate.  None the less, for those playing along at home if demo-openrc does not work due to permissions issues, try admin-openrc because that will possibly work and certainly can't make the situation worse.

I encountered some MTU mismatch problems when experimenting with Swift.  Container creates will work because that fits in one packet, object upload timeouts, its all very tedious but typical of a MTU mismatch, a problem I've seen before.  If you're following along at home this is the first time the system is getting pushed hard enough to run into MTU problems.  It's easy to diagnose with some simple long ping packet commands, and then force your interface and switch MTUs to working values.  I could complain and file a bug on NetGear firmware from 2017 but I'm not even using that old managed switch anymore thus I can't replicate it, but it was something "hilarious" like the web UI for the switch permitted me to set a MTU as high as 9198 bytes but experimental tests with the ping command shows no packets longer than 9018 bytes would actually pass (or something like that) so I set it all to 9000, switch port, interface, everything, and later ended up replacing that switch anyway so I no longer have access.  Anyway, the moral of the story is always test different length ping packets during initial testing, it'll save you time in the long run.

The list of Swift-associated projects is very cool to browse.

https://docs.openstack.org/swift/yoga/associated_projects.html

rclone is not in the list although I plan to make use of it later.  In the past I've successfully used rclone in non-Swift related situations to do cloudy file transfer stuff, so when I get around to setting up rclone swift access, I expect it'll work at least as well as past situations.

https://rclone.org/swift/

I submitted a doc improvement bug to the swift-launchpad as you can see at this link:

https://bugs.launchpad.net/swift/+bug/1981617

Tomorrow is Glance day.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 016 - OpenStack Horizon UI

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 016 - OpenStack Horizon UI

Horizon is a pretty easy install.  Note that the docs on page:

https://docs.openstack.org/horizon/yoga/install/install-ubuntu.html

Refer to OPENSTACK_KEYSTONE_URL on port 80, and if you've been following along in the instructions, you set it up on port 5000, and there is a note in the docs on that topic.

I note that Kolla-Ansible also installs Keystone on port 5000.  So its just kind of odd to see a suggestion of running Keystone on port 80, I have not seen that anywhere else.

On identical hardware, Kolla-Ansible's Horizon is vastly faster.  I suspect something to do with different WSGI configurations.

You will need the "Horizon Plugin Registry" later on when installing add-on services.

https://docs.openstack.org/horizon/yoga/install/plugin-registry.html

When I first set up Horizon and Keystone I placed pretty much the entire LAN into one project, which is slow to use and slow to page thru multiple screens in Horizon.  My solution for Plan 2.0 of breaking everything into smaller projects makes Horizon much more usable.  My architecture advice would be if you find yourself with multiple pages of Anything in Horizon, you probably need to break your projects down into smaller components, for ease of use reasons if nothing else.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 015 - OpenStack Designate DNS Service

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 015 - OpenStack Designate DNS Service

My strategy in setting up Designate before setting up storage or compute services was to get something VERY simple working with the complete system from database to web UI.  Then, if I have any problems with something complicated like volume storage, I can rule out system-wide issues such as mysql not working or whatever "infrastructure-level" problems.  This strategy worked very well.

https://docs.openstack.org/designate/yoga/install/

https://docs.openstack.org/designate/yoga/install/install-ubuntu.html

The installation was almost flawless.  I had to restart the designate-api and designate-central one extra time before updating my DNS pools, not mentioned in the docs.  Odd, but not a big deal.

There was a minor problem discovered later on, where I manage Docker hosts remotely using Portainer, which has a default port number of 9001.  Which, on an OpenStack cluster, is the default port number of the Designate API.  So I reconfigured Portainer and its agent to connect only to the OpenStack hosts on port 19001 instead of 9001 and that worked great.

The good news about Kolla-Ansible is it puts the APIs on the virtual IP address so port 9001 should be open on the bare metal.  The bad news is Kolla-Ansible is not entirely compatible with bare metal Docker especially after installing Zun and friends.  See Plan 3.0 for how I host bare metal Docker hosts requiring hardware access / pass-thru.

Samba Internal DNS Resolver is not very smart, and unfortunately being an AD domain, all my hosts point to it for DNS resolution.  All the Internal Resolver can do is respond back with any configured A or PTR record if the request is in its domain, OR pass the request to its configured forwarders if its not in a configured domain.  Which, superficially, sounds like a good start to a DNS server, but as always the devil is in the details.  I set up my Designate to configure it's zones on my bind9 resolver "forwarders" which are downstream of my Samba Internal DNS Resolver.  Yes I know, not ideal DNS architecture to merge authoritative and resolution DNS on the same system but if you're careful, it'll work very well.  Anyway, my plan was to delegate NS records in Samba Active Directory such that subdomains of the domain set up in AD would be forwarded to the resolvers, which are also the Designate authoritative servers.  If the Samba Internal DNS resolver were as smart as a Bind DNS server, it would indeed work using that configuration.  But ... it is not, so it does not work, you can put a protocol analyzer on the network then feel unhappy, or just trust me, it does not work.  You can configure NS records in the AD side of Samba, but the built in Internal DNS resolver in Samba simply ignores those NS records.  Later on, sometime around Plan 2.0, I implemented the very common alternative of simply using multiple domains, cedar.mulhollon.com remained my Samba domain, and openstack.mulhollon.net became my OpenStack Designate domain, but that's a story for a much later blogpost.

Somehow the docs completely skip installation of the CLI client for Designate.  On Ubuntu installation of the client is a one liner:

# apt-get install python3-designateclient

https://docs.openstack.org/neutron/yoga/admin/config-dns-int.html

Integrating Designate with Neutron.

Although it makes sense and is obvious in retrospect, this link has a table describing which Neutron components support which Designate attribute, and it boils down to the only component pairs that can't be linked are dns_name and a network, which makes a certain amount of sense.  I was kind of hoping that if you link a hostname to a network that would hypothetically be a sensible way for a sysadmin to define a wildcard A record for a domain, but that's not how its done.

Note that I discovered later on, in my HEAT orchestration experiments around Plan 2.0 timeframe, that if you design a HEAT template with DNS attributes and try to apply it to a Neutron resource that does NOT have Designate installed and configured, the HEAT template will fail.  So that's kind of annoying.  I had hoped and expected that Neutron would support entering the unused data in the database and later on after configuring Designate it would process the backlog or at least start to work on new configurations, but that is not the case, unfortunately.  So if you're EVER going to use Designate in the future, you should probably set it up and integrate it with Neutron very early in your overall cluster installation process.

Another thing that does not work until AFTER Designate is integrated with Neutron is integrating Designate with your external dns service.

Its kind of buried in the fine print, but remember you need to restart neutron-server after integrating with Designate.  You don't have to power cycle and reboot everything, remember this is not Windows, just restart neutron-server service and it's all good.

https://docs.openstack.org/neutron/yoga/admin/config-dns-int-ext-serv.html

This link is probably the most important link on the page, you have to read and wrap your mind around all the use-cases in order to successfully use Designate with Neutron.

This link explains some interesting behavior you will probably run into which might otherwise be confusing.  For example, if you consider a Use-Case-1 situation, if you try to configure a Neutron Port with a dns_name that does not match a Nova Instance hostname, you will probably become frustrated.  AFAIK if you want to IPv4 IPv6 dual stack and only NAT the IPv4 then Use-Case-3a is your only practically viable option.  Of course my OpenStack experience as end-user and later as an admin cover quite a span of time, so its possible times have changed and I should probably closely re-read this link.

On the other hand if you're just doing "typical normal stuff" then Designate does reliably do pretty much exactly what you'd expect it to do, which is awesome.  It is nice to have this link to read when you're trying to implement something well off the beaten path.

https://opendev.org/openstack/designate-dashboard

There is no documentation about the Designate Horizon web UI plugin, however it was pretty easy to figure out what to do on the OpenStack Controller:

apt-get install python3-designate-dashboard

systemctl reload apache2.service

My overall experience with Designate is it's pretty cool and easy to use and reliable.  I have always been disappointed that virtualization services like VMware do not have a boxed, prepackaged DNS solution similar to Designate.  VMware does have a half dozen different orchestration and automation solutions and most of them could be manipulated into a custom solution to automatically set up DNS, of course.

My project notes are grouped by service.  Because of that, setting up designate-dashboard and linking with Neutron above is out of order; we install Horizon tomorrow and Neutron four days later.  There are some circular loops when setting up an OpenStack cluster, at least if you set it up by hand like Plan 1.0.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 014 - OpenStack Placement Inventory Service

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 014 - OpenStack Placement Inventory Service

https://docs.openstack.org/placement/yoga/install/

This is a short blog post.  Sorry about that.  There is just not much to say about the installation of Placement on OpenStack.  It "just works", the documentation was complete and accurate.

Sometimes no news is good news.

In comparison, I will have a LOT more to say tomorrow about OpenStack Designate.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 013 - OpenStack Barbican Key Management Service

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 013 - OpenStack Barbican Key Management Service

https://docs.openstack.org/barbican/yoga/install/

Barbican is a pretty cool idea.  Take a large number of physical crypto storage hardware solutions, or even just a plain text file, and wrap it with all the Keystone access control stuff WRT users and projects.  My experience, so far, is it works very well.

Note that I configured with "enabled_secretstore_plugins = store_crypto" which just uses the crypto.simple software only crypto system.  Still better than nothing.

The hand installed Barbican following the official installation guides came up with secret HREF uris listing localhost instead of a real IP address, which is weird, later on the Kolla-Ansible installation of Barbican worked perfectly out of the box, although that's getting way ahead of the story.

To test the hand installed Barbican I wrote some testing scripts.  Later those were modified for a Kolla-Ansible installation, but they may still be useful for a hand installation, and can be found here at this public GitLab repository:

https://gitlab.com/SpringCitySolutionsLLC/openstack-scripts/-/tree/master/demos/barbican

As of writing this blogpost I have not written demonstration scripts for Barbican's Container functions or its Consumer functions, although I probably will sooner or later.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 012 - OpenStack Keystone Identity Service Installation

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 012 - OpenStack Keystone Identity Service Installation

https://docs.openstack.org/keystone/yoga/install/

Ansible installs/upgrades the package, everything else is manual.

Installation of this service was uneventful.  Lots of cut and paste from the installation guide into the console.

I used the CLI client to create the sample domain, projects, roles, etc.

It takes awhile to wrap your head around the right way to use Keystone's features.  At first I just used random designs from the docs.  Eventually I set up one giant project, at least I set it up correctly, and put everything "production" into that giant project.  That makes the Web UI very slow and clumsy to operate.

Not to jump too far ahead, but in the end I set up projects in Keystone more or less along role and project lines.  I have the following projects at this time:

admin: test experiments while logged in as admin, which you shouldn't do normally anyway.

enduser: literal enduser instances, hosts that I log into on a day to day basis to do "stuff".  Things you'd have an Apache Guacamole entry to access.

infrastructure: instances and things that are required but rarely directly touched, think of DHCP servers or DNS servers or similar.  If a user would directly touch it, like Booksonic or Apache Guacamole, it belongs in "server" project.

iot: this is an enduser project with multiple images.  More or less the entire Eclipse IoT software suite.

rutherford: another enduser project.  Kind of a demonstration of computational physics simulations mixed with a demonstration of OpenStack.  More detail is beyond the scope of this blog post, or even this series.  Watch for an interesting Youtube video in the future on the Spring City Solutions LLC channel!

server: this is server things endusers directly connect to, like Apache Guacamole.  If its important and required but end users don't log in directly, like DHCP, that would belong in the "infrastructure" project.  Think of Emby server, or minecraft server, or Booksonic server.

Looking back, from the time of writing this all the way back to the earliest days, this project organization strategy has worked very well.

The docs emphasize making your own roles in Keystone, but this seems a non-productive activity.  Nice to know its possible and nice to know how if you need to, but under normal conditions, why?

For troubleshooting experience, everyone involved with Keystone should replicate the work in the Verification section of the chapter, just to have the experience of having obtained a token.  Never know when this troubleshooting experience will be necessary.

https://docs.openstack.org/keystone/yoga/install/keystone-verify-ubuntu.html

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 011 - Installing the OpenStack Environment

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 011 - Installing the OpenStack Environment.

The relevant chapter from The OpenStack Install Guide:

https://docs.openstack.org/install-guide/environment.html

Most of this was installed one time, by hand, by me, then I added it to my Ansible 

and let Ansible take care of the other hosts in the cluster. Life is too short to configure by hand, let Ansible take care of it. I created for myself kind of a low-budget homemade Kolla-Ansible, and it worked pretty well.

As a checklist and commentary of each step in the process:

Ansible takes care of the apt repo.

Ansible installs SQL packages, but have to restart and finalize manually.

Ansible installs message queue, but have to add user and permissions manually.

Ansible installs memcached, but have to restart manually if config changes.

Ansible installs etcd and config file, but have to enable and restart manually. Note the human readable name is default, not controller as per docs. That seems to work?

As a side note, Openstack docs should emphasize testing MTU on networks as OS behaves very weirdly during MTU mismatch situations. MySQL, especially goes insane if you set up for a 9000 byte MTU but its actually only 1500 or even worse, 1450 bytes later on. Old timers who've run into MTU mismatch problems recognize and fix it pretty easily, but sysadmins whom have not run into MTU problems always struggle on their first. I'd almost suggest new sysadmins intentionally set up MTU settings wrong to gain the experience.

Stay tuned for the next chapter!