Adventures of a Small Time OpenStack Sysadmin Chapter 006 - Shut down vSAN

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 006 - Shut down vSAN

The reasoning behind the vSAN shutdown being you can’t have vSAN cluster failures while shutting down individual ESXi hosts if you’re not using vSAN anymore.

Historically performance over NFS was not as good because my NFS server was provisioned for capacity not speed (so HDDs and 1G ethernets, instead of all SDDs and 10G ethernet).  NFS performance is/was temporarily usable, in the past I successfully moved entire workload to NFS or shut it down during appliance firmware upgrades and appliance hardware upgrades.

If I had to do it all over again I would not have spent time completely wiping vSAN after all workloads were removed.  I did not want false alarms popping as ESXi hosts shut down so it seemed a wise investment at the time.

The main problem I ran into and had to Google for was vCenter will not let you shut down a vSAN, and WILL NOT TELL YOU WHY, if the cluster has High Availability HA running.  If you shut off HA then vCenter will let you shut off vSAN.

The secondary problem I ran into was I wanted to wipe the vSAN partitions just to make sure nothing weird tried to resurrect itself after a reboot, for whatever reason.  vCenter does not make it easy to wipe partitions but it is eventually possible with some Google search work.  This was probably a waste of time, in retrospect.

After shutting down at the cluster level and host level and partition level, I also removed my vSAN VLAN from the ethernet network, removed DNS entries for the former, now deleted, dedicated VMK interfaces, removed allocations for the VLAN and the ip addresses and space and device interfaces from Netbox.  "Paperwork" took longer than the technical work as is usually the case.

Enough deleting and removing; time to start designing and adding.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 005 - The Big Cleanup

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 005 - The Big Cleanup

The primary narrative about the big cleanup of the old VMware cluster, is it was circular and forgetful.

If I had to do a big cleanup again, I'd apply more effort ahead or time to more fully organize two checklists, then wait a couple days to think it all over in the background, finally implement all at once in a sequential organized manner, ideally with no interruptions.

The first checklist would be all the virtual machines to be eliminated or consolidated or worked around.  Over time its very easy with "free" cluster resources to implement an entire machine to, essentially, do as little as replace a symlink or cronjob on another more logically located and designed machine.  There is also the eternal dilemma of an experiment that isn't working completely; should I wipe it and start fresh later, or save its current config and wipe it until later, or try to patch it up into working, or throw some effort into doing it correctly now?  Finally we all know that in reality there will be virtual machines that cannot be completely replicated by existing Ansible scripts, even if that is the standard or at least the goal.  So:  Put in the time to document and automated it now?  or Hope for the best and fix it later, I mean, I put it together once, I'm always more skilled over time, so I'm sure I can do it again probably faster and easier?  Also need a dependency tree... If you plan to wipe Apache Guacamole much later, and it was relying on a dedicated OpenLDAP server to provide its config, then you can't wipe the OpenLDAP today it has to go after the Guac server goes (back in the olden days this was the easiest way to configure Guac, the new web config is great and very usable...).

The second checklist would be an inverse of the provisioning procedure.  People are usually pretty good about documenting their provisioning procedure, usually not so good at documenting their DEprovisioning procedure.  It's very easy when working on this as a secondary task to forget or skip over steps due to an interruption.  Remove (or reassign?) the IP allocation in netbox.  Remove the VM in netbox.  Remove the computer from Active Directory.  Remove the DNS, forward and reverse or maybe none, from Active Directory DNS.  Remove from Todoist runbooks and scheduled maintenance planner.  Remove backup scripts and maybe immediately trash the backups or place for safekeeping.  Remove the Ansible playbook, maybe remove a no longer in use Ansible role, maybe other stuff.  If there was a link to it on the wiki, remove the link or place a note that its down until later.  Zabbix monitoring will go crazy when the host disappears, remove it.  If it hosted Docker containers the Portainer monitoring system will get agitated when the host disappears.  If you plan to ever reimplement the service/system, it needs to be ADDED to the Todoist project planner, or even worst and more confusing, maybe it needs to be added to the Redmine or maybe Asana, because the only thing better than having one project planning software tool is spreading tasks across three depending on higher-level project needs.

Anyway, took a couple days, part time, to properly and completely "spring clean" everything that could be deleted, even if only temporarily.

This should pay dividends later on; I'm not going to put time and effort into moving something that doesn't need to be moved because its not there anymore.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 004 - Ending Conditions

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 004 - Ending Conditions

To end up where you need to be, first you have to figure out where you need to go.

Originally I planned on one big OS cluster, as seen in previous posts.  You probably noticed the intro to all the posts in this blog series, mentions I have two small OS clusters instead of one large cluster.  I'm writing this blog series some months after the "fun" so some of the chronology is out of order, and nothing motivates OpenStack administrators to use Kolla-Ansible to install OS clusters, quite like trying to use the online "Installation Tutorials".  So my idea is to always have two clusters, and I can alternate between a cluster being in a relatively stable "production" status and temporarily being in a "dev" or "upgrading" status, and keep my images on the move to either load balance across the clusters or push everything to one to deploy a new one.  Its all automated so its not any work, for me anyway, just watts of CPU power.  In summary, the original plan gradually evolved from a single six host cluster (which was the VMware cluster architecture for many years) to a dual three host cluster.  I wanted everything managed in my existing Ansible; seems I will be multi-Ansible soon, with Kolla-Ansible for the OS clusters, and my own internal system.

My plan for storage was pretty simple, Cinder on LVM on the "bulk" terabyte SSDs on each host, and Swift could use the entire M.2 NVME SSD on each host to store objects, especially backups.  Also I could cross store backups such that cluster2 could have a copy of all the backups in cluster1's Swift store, certainly make cluster "disaster recovery" simple if each cluster has all backups all the time.  I still plan to do multi-backend Cinder connecting to the NAS over NFS, or, I suppose, iSCSI.  I have some images for fileservers and such exporting iSCSI attached storage at the image level from the NAS and that works well, at least iSCSI works on FreeBSD (LOL, more on that later).

I didn't really understand networking on OpenStack until I extensively experimented with it.  Looking thru a lens of "its like NSX, but its not NSX" helps.  I originally decided to set up my provider network ports as non-tagged non-vlan, and that worked, although it was a HUGE headache later on, for reasons I will explain later, and the second cluster VLAN definitely attaches provider networks with VLAN tagging such the cluster has access to everything... this became VERY important later on (more detail in a later post...).

Generally I planned my goal as standardizing everything on FreeBSD for "real" workloads and Ubuntu for unix-like workloads (basically, Ubuntu only as hosts for Docker containers, at least until I install and figure out OpenStack Zun).  I was pretty successful in this.  Its fun to have images on your VMware cluster of msdos with Samba network access or Devuan or GUIX or old installs of GNU Hurd, but that was mostly for experimenting and I can experiment on OpenStack AFTER I get the clusters operational.  So, for a little while, I planned to compress everything into working on FreeBSD 13.1 or if necessary Ubuntu 20.04, and this temporary "compression" was entirely one hundred percent successful.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 003 - Starting Conditions

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 003 - Starting Conditions

To get where you're trying to go, first you have to figure out where you are.

The good news is, I'm the guy whom stood up the VMware cluster, so I have a pretty good idea how it works.  And I like having good documentation, so I have a local installation of Netbox, which is the best IPAM solution I've ever seen, or even dreamed of.

There are six identical ESXi hosts, all a couple years old, the "famous" SuperMicro SYS-E200-8D model that is so popular in home labs around the world.  96 gigs of RAM, each, because in my short term, wild and experimental VMware NSX era, NSX was incredibly memory hungry, to the point where you'd think the world's memory manufacturers bribed VMware to find some way to use more memory.  Like what is it even doing with all those gigabytes of RAM on each host?

As for networking, each ESXi host has an IPMI port with a pretty good web accessible HTML KVM, and dual one-gig ethernet ports and dual ten-gig ethernet ports.  The VMware networking concept, at least pre-NSX, is to set up distributed switches across the hosts and uplink each switch to VLANs, preferably across all the ethernet ports, to different switches.  So I have a one-gig ethernet switch with 12 connections and a ten-gig ethernet switch (which was admittedly expensive some years ago...) with 12 connections.  All the ports are configured the same way and all identically trunked together although I tried to follow VMware guidelines to prefer vmotion vmk to use to one dedicated 10G port and prefer vSAN vmk to use a different dedicated 10G port, but in theory (and in practice, a couple times...) it was possible to yank out any three of the four ethernet ports and the system would keep working, although perhaps slowly.  Well, it was possible to REALLY fool VMware in the old days by admin down ports or blocking VLANs on the switch side, but in general it was pretty bulletproof.  NSX was somewhat less bulletproof due to extreme complexity, but I had given up on NSX many years ago.  OpenStack does things quite a bit different than VMware, will get to that later...

As for storage, there's vSAN across all six hosts, all SSD, ESXi hosts, which for years has always been one hundred percent reliable, and I've never personally experienced data loss or frankly even much of  a problem with vSAN.  I also have a huge iX Systems hardware NAS of many terabytes, and a couple smaller TrueNAS boxes based on Intel NUCs from a couple years ago with a fraction of a TB each.  VMware works reliably over NFS although obviously the vSAN is enormously faster.  Over the years I upgraded the storage on each host, started out with cheap small HDD and moved to small SSD and later larger SSDs as prices fell over the years.  Those SSDs will be used for the bare metal OS and Cinder storage.  Each host also has a M.2 NVME internal SSD that was used for vSAN cache and intend to use for OpenStack Swift.  Storage will also be a lot different than VMware for OpenStack Cinder...

Generally the easiest way to load balance and backup and replicate and generally administer Docker workloads on VMware was to set up a host for every project (or even, every container!) and let vMotion and DRS and HA do its magic.  With Ansible to automate the Linux side and products like Orchestrator to automate the VMware side it only takes minutes to spin up a new Active Directory connected Docker host on the old VMware system, which I can likely replicate on OpenStack using Heat (or Magnum with really small Docker Swarms?) or I could move to Zun on OpenStack, eventually.

The biggest changes were infrastructural in nature.  Aside from previously mentioned Zun to handle Docker, I would be using Heat instead of Orchestrator, and 'something' in place of Log Insight.  Probably a homemade ELK stack?  In the old days you'd install an ELK stack on bare metal, or a bare metal image anyway, just like you'd install something like Apache Guacamole on bare metal in the old days, but now there are Docker containers for seemingly all services.  So somehow this conversion project is already morphing into a larger conversion; not just drag and drop existing images, but change entire software architectures to use more Docker containerization and so forth.  Not exactly the first IT project in history to experience massive scope creep over time... 

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 002 - The Plan, v 1.0

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 002 - The Plan, v 1.0

First, I need an operations and logistical plan.  What's going to go where, when, the usual project planning puzzle.

Before the big hardware crash, I had some experience with OpenStack and researched my options, so I was not going in blind.  Also, if you're creative enough, and have enough resources at hand, its hard to get yourself boxed in.  At least so far in life, I've always managed to figure my way out of whatever I got myself into.  So far, LOL.

The existing VMware cluster has six hosts, and vSAN really needs a minimum of three hosts, and the cluster workload was somewhat less than fifty percent, probably less than a third would be "tolerable" for awhile.  This would imply I could split the six host cluster, convert half of it, the first three hosts, to OpenStack, move everything from VMware to OpenStack, convert and add the other half of the cluster to the larger OpenStack, and I'd be done.  

No big deal, probably a long weekend's work.  With respect to installing OpenStack, I'm sure the experience and docs have improved over the years so a couple "apt-get install" lines on a fresh Ubuntu install and I'll be running.  Looking back, I was sooooooo optimistic.  "In the old days" when I was experimenting with OpenStack many years ago this new-fangled "Kolla-Ansible" project was too new to use, so I figured I'd use the openstack repos and just install components one at a time by hand, just like the good ole days, with some help from my existing Ansible infrastructure to replicate across all the hosts and keep the cluster identical.

So in more detail, my plan, version 1.0, looked like this:

Decommission as many legacy cruft/junk/obsolete VMs as possible.  If its not there anymore, I don't have to move it nor worry about making it work.  All clusters accumulate old junk; what a glorious time for a spring cleaning marathon?

Shut down vSAN, for safety's sake, and run everything off the slow NAS over NFS.  I've done this before and have some HUGE storage servers that run off this NAS; its just a matter of some Storage vMotion moves, then shut off the vSAN.  NFS to a spinning rust HDD NAS is slow, a lot slower than vSAN on an all SSD cluster with 10G ethernet, but its "fast enough for awhile".

Infrastructure prep before starting.  Why is my netboot.xyz infrastructure not working for PXE netboot/installation of software, I've certainly installed ESXi over the network before, along with other OS.  Get a head start on documenting everything in Netbox, tidy up the ethernet switch VLANs and stuff, prep Ansible for future hosts and services...

Shut down ESXi hosts 1, 2, and 3 safely and cleanly.  Clean, dust, relabel the hardware, update the docs in Netbox and in the Active Directory DNS (which is actually a cluster of Samba servers acting as DCs, works great for many years now).

Bare metal Ubuntu installs on hosts 1, 2, and 3.  Docs imply OpenStack "Yoga" version works best on Ubuntu 20.04.  I plan to have Bind DNS running on all bare metal hosts for OS Designate to dynamically configure DNS for the cluster, will be interesting to see how that interacts with the existing Active Directory install (spoiler, it was challenging and needed a lot of reworking...)  Originally I planned to move my DHCP servers to bare metal installs, that plan changed along the way also.  One plan that worked pretty well was setting up the entire OpenStack cluster as a giant NTP cluster.  My innovative and creative solution to OpenStack not handling USB passthru like VMware, was to just set up Docker on one of the hosts and not virtualize those USB-requiring applications at all.  This step also includes proving out the LAN, and in retrospect my MTU testing was not careful enough, leading to some considerable trouble later on.

OpenStack has a wildly different project architecture than VMware.  VMware sets up simple hypervisors on all hosts, then runs "the cool stuff" as virtualized hosts, so vCenter lives as just another image.  OpenStack kind of inverts that architecture, so you need a controller host (or, ideally, cluster) that runs bare metal MySQL as a database, or RabbitMQ or similar, and the hypervisors ONLY run production workload.  So there's some infrastructure to set up, those applications along with memcached, etcd, etc.

At this point I planned to follow the online OpenStack "Installation Tutorial" documentation.  I was a little nervous that most of the docs referenced Ubuntu 18 or even Ubuntu 14...  I designed a sensible dependency tree, Keystone first, Swift before Glance, Placement before Nova, etc.  I figured I'd set up the basics now, and experiment with more advanced features like Magnum and Mistral and Zun and Trove later.

"Move everything from VMware to OpenStack".  Sounds simple.  In retrospect, as usual, the last ten percent of any effort takes ninety percent of the time, recursively...

Shut down and clean up and re-install the last three hosts, 4, 5, and 6.  This will be the end of the VMware cluster, don't have to be so careful, plus with previous experience, this should be pretty smooth.

Add hosts 4, 5, and 6 to the existing cluster consisting of hosts 1, 2, and 3.  I have some experience messing around with OS Swift so I know it'll take some effort but its quite possible.

Add "cool new extra" services to the larger capacity full size OpenStack.  Maybe try out Trove for databases instead of spawning off more Docker containers, that sort of thing.  At the time, I planned on setting up an ELK stack, which I've done before, to replace Log Insight.

I expected to do one last sweep thru the entire system to update docs, update hardware labels, verify and maybe even test new backup strategies.

The plan sounded great...  However, IT work is similar to my military experience, in that no matter how well designed a plan is, the plan never survives contact with the enemy.  The mental effort of making a plan provides the "virtual experience" to improve the odds of success, so its not wasted time.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 001 - Intro

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 001 - Intro

"A long time ago in a galaxy far, far away..."

I personally set up a small VMware cluster of a mere six ESXi hosts a couple years ago.  I enjoyed the "cheap" ESXi-Experience as a VMware Users Group member for many years.  For those unfamiliar with the VMUG Eval-Experience or whatever its called now, for a very modest annual fee you get the "keys to the kingdom" of everything VMware has under a license only permitting educational use, and I learned a lot about VMware over the years.

I enjoyed the experience, I have to admit over years of experience that vSAN is the best SAN I've ever used, I scripted things that don't need to be scripted with Orchestrator, followed up on hardening suggestions from VMware Operations Manager, experimented with vCloud Director and Integrated Containers.  Log Insight is not necessarily better or worse than a FOSS ELK stack, but it is different.  Like many people I tried setting up NSX without understanding the basic concepts and thus obtained an education under fire of how NSX works; cool system, but a lot of mental gymnastics and work.  Upgraded vCenter many a time, had to reinstall it a couple times over the years, LOL.  Remediated quite a few ESXi hosts.  Like all the rest of you I was wondering how I'm going to use a flash-based web interface for vCenter when everyone removes flash support from their web browsers in a couple months, but VMware came thru seemingly at the last minute and everything worked perfectly on the new html5 web client.

Like everyone else who's honest about their cluster admin experience, there was plenty of comedy over the years.  Take a snapshot of a server before upgrading, upgrade, and wonder why there's a fraction of a terabyte snapshot file slowly filling up my NAS months later.  The usual fun with put host 2 in maint mode to shut it down and upgrade the memory, but instead pull the plug on host 3 and stick the memory in it instead, wonder why the cluster went nuts.  The odds of having five identical ethernet cables on six hosts across five VLANs all properly labeled and installed ... are about as low as you'd think, leading to fun with primary and backup links when upgrading the VSAN and vMotion interfaces to a new 10G ethernet switch.  Back in the old days (vers 5.x ?) you had to install special custom hardware drivers for 10G ethernet, which hilariously crashed ESXi when you upgraded ESXi far enough past those old drivers.  Much like how "everyone knows" that UPS hardware is less reliable than wall outlet power in the long run (well, depending where you live, LOL) VCHA sounds like a good idea but was always less reliable than a single VC install, I blew up VCHA quite a few times.  It seems that if something crazy could theoretically happen, it happened to me, I figured it out, and I fixed it.  I have this "VMware Sysadmin" stuff figured out.  Every time, eventually, everything always ended up working really well!

But all good things eventually end; VMware has been sold, and I'm not sure about the future, not as much money to be made in VMware consulting as I would have hoped (although you know where to reach me if you need skilled, experienced help with a VMware cluster).  Sounds like newer versions of ESXi will require a writable boot drive for logging that I don't want anyway and this cannot be wedged into my existing hardware without extensive effort.  The Eval-Experience program is a lot cheaper than list price, but it's still about half the annual cost of electricity to operate the cluster.  I am just done with VMware.    

I had some really positive experiences with OpenStack over the last decade or so.  Both at my home lab and when working for other people.  Eventually, I will start the project to convert my VMware cluster to OpenStack.  Eventually.  For a couple years now it has been "Eventually".  Meanwhile, other things to do.

Sometimes, "Eventually" happens faster than you'd expect.  One fine afternoon this winter, the power supply for host number three failed.  No big deal, that's why we keep spares.  About five minutes after replacing the power supply, I see in the IPMI KVM window, that host number three's boot device also failed some time in the past (no alarms in vCenter?), so now it can't boot ESXi.  No big deal, that's why we keep spares.  Next surprise discovery is the vastly underutilized yet very important netboot.xyz installation for PXE based operating system installation over the LAN is also not working.  I'm getting perturbed by that point.

I guess this is as good a point as any to initiate the big conversion project, replacing VMware with OpenStack?

Stay tuned for the next chapter!

Flutter in Action Chapter 9 "Async Dart and infinite Scrolling"

Completed taking notes on "Flutter in Action" chapter 9 "Async Dart and Infinite Scrolling".

https://www.manning.com/books/flutter-in-action

To quote the author, "Features all rely on each other to explain in a circular manner."  That is the fundamental problem with async programming, you either get all of it and its all crystal clear, or get none of it and its all unclear.

The chapter starts with an introduction to Future as the base of async programming in Dart.  Give it a .then and a function to call when the data is eventually ready.  Give it a .catchError to process failure arriving rather than good data arriving.

Next up is a discussion of async to turn a function async, and await to make little parts inside it synchronous again.  Supposed to be easier or more clear to read than using Futures.

Error handling in async programming can be done with try/catch blocks or placing a .catchError on a Future.

The chapter repeats a couple times what's required for a working Observer/Stream pattern, which is good because its important.  The first is a sink for events, Sink being the abstract class and mostly using StreamController as the sink.  The second requirement is streams which are properties on the sink.  Final requirement is subscribers / observers / listeners which wait for notification by listening to the stream.  This section has great examples which build up from simple examples to rather complicated multi-subscriber systems using StreamController.broadcast() using an example of a hamburger restaurant.  I especially like how the cartoon illustrations gradually have more complex source code added to them to describe the async operation of the restaurant's kitchen.  Very clear and detailed explanation of the Observer Pattern.

Higher order streams are analogous to higher order functions; streams which return streams.  A bit complicated and even a simple example is a page of dense code.  It would have been nice to arrange the chapter to cover a minimal set of circular dependencies and later cover this complicated feature.  Then again the usual problem that its already a 342 page long book so this topic had to be fit somewhere.

StreamBuilder turns a stream of data into widgets, similar to a template library.  It is a close cousin of the FutureBuilder.  The book glosses over how to use and set it up, probably a page length limitation.  If the book had the luxury of ten additional pages to cover this one page topic, it would probably look like the tutorials I'll link below:

https://www.woolha.com/tutorials/flutter-using-streambuilder-widget-examples

https://api.flutter.dev/flutter/widgets/StreamBuilder-class.html

I think the topic lends itself well to video presentation as seen in the video on the API page above.  I like that video.

Infinitely scrollable lists are made with CustomScrollView.  It wants a source of Slivers and some type of scroll physics.  Sadly the chapter lists scroll physics as a topic to be covered on the very first page of the chapter but its not discussed; the book already has 342 pages... probably got edited out to save space.  Slivers are widgets with a weird lifecycle, they are created and deleted based on if they're in the user's view of the list, which makes them fast and efficient.  Regular widgets would be slower and use more memory.  Begs the question of why Flutter does not make caching decisions automatically for the programmer; why not let the framework decide instead of hard coding it into the source code?

The book unfortunately misses a great chance to compare/contrast a ListView vs the CustomScrollView.  Main differences are ListViews don't sliver and their widgets like Row/Container for arrangement, whereas CustomScrollView does sliver and has its grid passed separate from its slivers.  Supposedly you can do things like appbar animations while scrolling with CustomScrollView although I've never done this myself.  That would likely be too advanced of a topic for the current length of the book.

I think the best way to read this chapter is to close that circularity problem; read it all, get the big picture locked down, THEN research in detail, like my example above of doing a compare and contrast online of ListView vs CustomScrollView or digging in deep for an hour or two of online docs and videos about how to use StreamBuilder.  Async programming is a big topic; someone could probably make an entire book out of going into this chapter in great detail; I'd buy that book.  On a side note I have a pretty good book about async programming on Python so I know its a book length topic...  When "Python Concurrency with asyncio" is officially published in a couple months I might review it.  I've been reading that MEAP book as chapters have been released and its a good book.  Anyway Dart / Flutter could use a similar book which would expand this chapter to book length.  I see the Python async book is already up to 376 pages in pre-publishing which is longer than this entire Flutter book, LOL.

Next up, chapter 10, "Working with data: HTTP, Firestore, and JSON".