Adventures of a Small Time OpenStack Sysadmin Chapter 004 - Ending Conditions

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 004 - Ending Conditions

To end up where you need to be, first you have to figure out where you need to go.

Originally I planned on one big OS cluster, as seen in previous posts.  You probably noticed the intro to all the posts in this blog series, mentions I have two small OS clusters instead of one large cluster.  I'm writing this blog series some months after the "fun" so some of the chronology is out of order, and nothing motivates OpenStack administrators to use Kolla-Ansible to install OS clusters, quite like trying to use the online "Installation Tutorials".  So my idea is to always have two clusters, and I can alternate between a cluster being in a relatively stable "production" status and temporarily being in a "dev" or "upgrading" status, and keep my images on the move to either load balance across the clusters or push everything to one to deploy a new one.  Its all automated so its not any work, for me anyway, just watts of CPU power.  In summary, the original plan gradually evolved from a single six host cluster (which was the VMware cluster architecture for many years) to a dual three host cluster.  I wanted everything managed in my existing Ansible; seems I will be multi-Ansible soon, with Kolla-Ansible for the OS clusters, and my own internal system.

My plan for storage was pretty simple, Cinder on LVM on the "bulk" terabyte SSDs on each host, and Swift could use the entire M.2 NVME SSD on each host to store objects, especially backups.  Also I could cross store backups such that cluster2 could have a copy of all the backups in cluster1's Swift store, certainly make cluster "disaster recovery" simple if each cluster has all backups all the time.  I still plan to do multi-backend Cinder connecting to the NAS over NFS, or, I suppose, iSCSI.  I have some images for fileservers and such exporting iSCSI attached storage at the image level from the NAS and that works well, at least iSCSI works on FreeBSD (LOL, more on that later).

I didn't really understand networking on OpenStack until I extensively experimented with it.  Looking thru a lens of "its like NSX, but its not NSX" helps.  I originally decided to set up my provider network ports as non-tagged non-vlan, and that worked, although it was a HUGE headache later on, for reasons I will explain later, and the second cluster VLAN definitely attaches provider networks with VLAN tagging such the cluster has access to everything... this became VERY important later on (more detail in a later post...).

Generally I planned my goal as standardizing everything on FreeBSD for "real" workloads and Ubuntu for unix-like workloads (basically, Ubuntu only as hosts for Docker containers, at least until I install and figure out OpenStack Zun).  I was pretty successful in this.  Its fun to have images on your VMware cluster of msdos with Samba network access or Devuan or GUIX or old installs of GNU Hurd, but that was mostly for experimenting and I can experiment on OpenStack AFTER I get the clusters operational.  So, for a little while, I planned to compress everything into working on FreeBSD 13.1 or if necessary Ubuntu 20.04, and this temporary "compression" was entirely one hundred percent successful.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 003 - Starting Conditions

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 003 - Starting Conditions

To get where you're trying to go, first you have to figure out where you are.

The good news is, I'm the guy whom stood up the VMware cluster, so I have a pretty good idea how it works.  And I like having good documentation, so I have a local installation of Netbox, which is the best IPAM solution I've ever seen, or even dreamed of.

There are six identical ESXi hosts, all a couple years old, the "famous" SuperMicro SYS-E200-8D model that is so popular in home labs around the world.  96 gigs of RAM, each, because in my short term, wild and experimental VMware NSX era, NSX was incredibly memory hungry, to the point where you'd think the world's memory manufacturers bribed VMware to find some way to use more memory.  Like what is it even doing with all those gigabytes of RAM on each host?

As for networking, each ESXi host has an IPMI port with a pretty good web accessible HTML KVM, and dual one-gig ethernet ports and dual ten-gig ethernet ports.  The VMware networking concept, at least pre-NSX, is to set up distributed switches across the hosts and uplink each switch to VLANs, preferably across all the ethernet ports, to different switches.  So I have a one-gig ethernet switch with 12 connections and a ten-gig ethernet switch (which was admittedly expensive some years ago...) with 12 connections.  All the ports are configured the same way and all identically trunked together although I tried to follow VMware guidelines to prefer vmotion vmk to use to one dedicated 10G port and prefer vSAN vmk to use a different dedicated 10G port, but in theory (and in practice, a couple times...) it was possible to yank out any three of the four ethernet ports and the system would keep working, although perhaps slowly.  Well, it was possible to REALLY fool VMware in the old days by admin down ports or blocking VLANs on the switch side, but in general it was pretty bulletproof.  NSX was somewhat less bulletproof due to extreme complexity, but I had given up on NSX many years ago.  OpenStack does things quite a bit different than VMware, will get to that later...

As for storage, there's vSAN across all six hosts, all SSD, ESXi hosts, which for years has always been one hundred percent reliable, and I've never personally experienced data loss or frankly even much of  a problem with vSAN.  I also have a huge iX Systems hardware NAS of many terabytes, and a couple smaller TrueNAS boxes based on Intel NUCs from a couple years ago with a fraction of a TB each.  VMware works reliably over NFS although obviously the vSAN is enormously faster.  Over the years I upgraded the storage on each host, started out with cheap small HDD and moved to small SSD and later larger SSDs as prices fell over the years.  Those SSDs will be used for the bare metal OS and Cinder storage.  Each host also has a M.2 NVME internal SSD that was used for vSAN cache and intend to use for OpenStack Swift.  Storage will also be a lot different than VMware for OpenStack Cinder...

Generally the easiest way to load balance and backup and replicate and generally administer Docker workloads on VMware was to set up a host for every project (or even, every container!) and let vMotion and DRS and HA do its magic.  With Ansible to automate the Linux side and products like Orchestrator to automate the VMware side it only takes minutes to spin up a new Active Directory connected Docker host on the old VMware system, which I can likely replicate on OpenStack using Heat (or Magnum with really small Docker Swarms?) or I could move to Zun on OpenStack, eventually.

The biggest changes were infrastructural in nature.  Aside from previously mentioned Zun to handle Docker, I would be using Heat instead of Orchestrator, and 'something' in place of Log Insight.  Probably a homemade ELK stack?  In the old days you'd install an ELK stack on bare metal, or a bare metal image anyway, just like you'd install something like Apache Guacamole on bare metal in the old days, but now there are Docker containers for seemingly all services.  So somehow this conversion project is already morphing into a larger conversion; not just drag and drop existing images, but change entire software architectures to use more Docker containerization and so forth.  Not exactly the first IT project in history to experience massive scope creep over time... 

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 002 - The Plan, v 1.0

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 002 - The Plan, v 1.0

First, I need an operations and logistical plan.  What's going to go where, when, the usual project planning puzzle.

Before the big hardware crash, I had some experience with OpenStack and researched my options, so I was not going in blind.  Also, if you're creative enough, and have enough resources at hand, its hard to get yourself boxed in.  At least so far in life, I've always managed to figure my way out of whatever I got myself into.  So far, LOL.

The existing VMware cluster has six hosts, and vSAN really needs a minimum of three hosts, and the cluster workload was somewhat less than fifty percent, probably less than a third would be "tolerable" for awhile.  This would imply I could split the six host cluster, convert half of it, the first three hosts, to OpenStack, move everything from VMware to OpenStack, convert and add the other half of the cluster to the larger OpenStack, and I'd be done.  

No big deal, probably a long weekend's work.  With respect to installing OpenStack, I'm sure the experience and docs have improved over the years so a couple "apt-get install" lines on a fresh Ubuntu install and I'll be running.  Looking back, I was sooooooo optimistic.  "In the old days" when I was experimenting with OpenStack many years ago this new-fangled "Kolla-Ansible" project was too new to use, so I figured I'd use the openstack repos and just install components one at a time by hand, just like the good ole days, with some help from my existing Ansible infrastructure to replicate across all the hosts and keep the cluster identical.

So in more detail, my plan, version 1.0, looked like this:

Decommission as many legacy cruft/junk/obsolete VMs as possible.  If its not there anymore, I don't have to move it nor worry about making it work.  All clusters accumulate old junk; what a glorious time for a spring cleaning marathon?

Shut down vSAN, for safety's sake, and run everything off the slow NAS over NFS.  I've done this before and have some HUGE storage servers that run off this NAS; its just a matter of some Storage vMotion moves, then shut off the vSAN.  NFS to a spinning rust HDD NAS is slow, a lot slower than vSAN on an all SSD cluster with 10G ethernet, but its "fast enough for awhile".

Infrastructure prep before starting.  Why is my netboot.xyz infrastructure not working for PXE netboot/installation of software, I've certainly installed ESXi over the network before, along with other OS.  Get a head start on documenting everything in Netbox, tidy up the ethernet switch VLANs and stuff, prep Ansible for future hosts and services...

Shut down ESXi hosts 1, 2, and 3 safely and cleanly.  Clean, dust, relabel the hardware, update the docs in Netbox and in the Active Directory DNS (which is actually a cluster of Samba servers acting as DCs, works great for many years now).

Bare metal Ubuntu installs on hosts 1, 2, and 3.  Docs imply OpenStack "Yoga" version works best on Ubuntu 20.04.  I plan to have Bind DNS running on all bare metal hosts for OS Designate to dynamically configure DNS for the cluster, will be interesting to see how that interacts with the existing Active Directory install (spoiler, it was challenging and needed a lot of reworking...)  Originally I planned to move my DHCP servers to bare metal installs, that plan changed along the way also.  One plan that worked pretty well was setting up the entire OpenStack cluster as a giant NTP cluster.  My innovative and creative solution to OpenStack not handling USB passthru like VMware, was to just set up Docker on one of the hosts and not virtualize those USB-requiring applications at all.  This step also includes proving out the LAN, and in retrospect my MTU testing was not careful enough, leading to some considerable trouble later on.

OpenStack has a wildly different project architecture than VMware.  VMware sets up simple hypervisors on all hosts, then runs "the cool stuff" as virtualized hosts, so vCenter lives as just another image.  OpenStack kind of inverts that architecture, so you need a controller host (or, ideally, cluster) that runs bare metal MySQL as a database, or RabbitMQ or similar, and the hypervisors ONLY run production workload.  So there's some infrastructure to set up, those applications along with memcached, etcd, etc.

At this point I planned to follow the online OpenStack "Installation Tutorial" documentation.  I was a little nervous that most of the docs referenced Ubuntu 18 or even Ubuntu 14...  I designed a sensible dependency tree, Keystone first, Swift before Glance, Placement before Nova, etc.  I figured I'd set up the basics now, and experiment with more advanced features like Magnum and Mistral and Zun and Trove later.

"Move everything from VMware to OpenStack".  Sounds simple.  In retrospect, as usual, the last ten percent of any effort takes ninety percent of the time, recursively...

Shut down and clean up and re-install the last three hosts, 4, 5, and 6.  This will be the end of the VMware cluster, don't have to be so careful, plus with previous experience, this should be pretty smooth.

Add hosts 4, 5, and 6 to the existing cluster consisting of hosts 1, 2, and 3.  I have some experience messing around with OS Swift so I know it'll take some effort but its quite possible.

Add "cool new extra" services to the larger capacity full size OpenStack.  Maybe try out Trove for databases instead of spawning off more Docker containers, that sort of thing.  At the time, I planned on setting up an ELK stack, which I've done before, to replace Log Insight.

I expected to do one last sweep thru the entire system to update docs, update hardware labels, verify and maybe even test new backup strategies.

The plan sounded great...  However, IT work is similar to my military experience, in that no matter how well designed a plan is, the plan never survives contact with the enemy.  The mental effort of making a plan provides the "virtual experience" to improve the odds of success, so its not wasted time.

Stay tuned for the next chapter!

Adventures of a Small Time OpenStack Sysadmin Chapter 001 - Intro

Adventures of a Small Time OpenStack Sysadmin relate the experience of converting a small VMware cluster into two small OpenStack clusters, and the adventures and friends I made along the way.

Adventures of a Small Time OpenStack Sysadmin Chapter 001 - Intro

"A long time ago in a galaxy far, far away..."

I personally set up a small VMware cluster of a mere six ESXi hosts a couple years ago.  I enjoyed the "cheap" ESXi-Experience as a VMware Users Group member for many years.  For those unfamiliar with the VMUG Eval-Experience or whatever its called now, for a very modest annual fee you get the "keys to the kingdom" of everything VMware has under a license only permitting educational use, and I learned a lot about VMware over the years.

I enjoyed the experience, I have to admit over years of experience that vSAN is the best SAN I've ever used, I scripted things that don't need to be scripted with Orchestrator, followed up on hardening suggestions from VMware Operations Manager, experimented with vCloud Director and Integrated Containers.  Log Insight is not necessarily better or worse than a FOSS ELK stack, but it is different.  Like many people I tried setting up NSX without understanding the basic concepts and thus obtained an education under fire of how NSX works; cool system, but a lot of mental gymnastics and work.  Upgraded vCenter many a time, had to reinstall it a couple times over the years, LOL.  Remediated quite a few ESXi hosts.  Like all the rest of you I was wondering how I'm going to use a flash-based web interface for vCenter when everyone removes flash support from their web browsers in a couple months, but VMware came thru seemingly at the last minute and everything worked perfectly on the new html5 web client.

Like everyone else who's honest about their cluster admin experience, there was plenty of comedy over the years.  Take a snapshot of a server before upgrading, upgrade, and wonder why there's a fraction of a terabyte snapshot file slowly filling up my NAS months later.  The usual fun with put host 2 in maint mode to shut it down and upgrade the memory, but instead pull the plug on host 3 and stick the memory in it instead, wonder why the cluster went nuts.  The odds of having five identical ethernet cables on six hosts across five VLANs all properly labeled and installed ... are about as low as you'd think, leading to fun with primary and backup links when upgrading the VSAN and vMotion interfaces to a new 10G ethernet switch.  Back in the old days (vers 5.x ?) you had to install special custom hardware drivers for 10G ethernet, which hilariously crashed ESXi when you upgraded ESXi far enough past those old drivers.  Much like how "everyone knows" that UPS hardware is less reliable than wall outlet power in the long run (well, depending where you live, LOL) VCHA sounds like a good idea but was always less reliable than a single VC install, I blew up VCHA quite a few times.  It seems that if something crazy could theoretically happen, it happened to me, I figured it out, and I fixed it.  I have this "VMware Sysadmin" stuff figured out.  Every time, eventually, everything always ended up working really well!

But all good things eventually end; VMware has been sold, and I'm not sure about the future, not as much money to be made in VMware consulting as I would have hoped (although you know where to reach me if you need skilled, experienced help with a VMware cluster).  Sounds like newer versions of ESXi will require a writable boot drive for logging that I don't want anyway and this cannot be wedged into my existing hardware without extensive effort.  The Eval-Experience program is a lot cheaper than list price, but it's still about half the annual cost of electricity to operate the cluster.  I am just done with VMware.    

I had some really positive experiences with OpenStack over the last decade or so.  Both at my home lab and when working for other people.  Eventually, I will start the project to convert my VMware cluster to OpenStack.  Eventually.  For a couple years now it has been "Eventually".  Meanwhile, other things to do.

Sometimes, "Eventually" happens faster than you'd expect.  One fine afternoon this winter, the power supply for host number three failed.  No big deal, that's why we keep spares.  About five minutes after replacing the power supply, I see in the IPMI KVM window, that host number three's boot device also failed some time in the past (no alarms in vCenter?), so now it can't boot ESXi.  No big deal, that's why we keep spares.  Next surprise discovery is the vastly underutilized yet very important netboot.xyz installation for PXE based operating system installation over the LAN is also not working.  I'm getting perturbed by that point.

I guess this is as good a point as any to initiate the big conversion project, replacing VMware with OpenStack?

Stay tuned for the next chapter!

Flutter in Action Chapter 9 "Async Dart and infinite Scrolling"

Completed taking notes on "Flutter in Action" chapter 9 "Async Dart and Infinite Scrolling".

https://www.manning.com/books/flutter-in-action

To quote the author, "Features all rely on each other to explain in a circular manner."  That is the fundamental problem with async programming, you either get all of it and its all crystal clear, or get none of it and its all unclear.

The chapter starts with an introduction to Future as the base of async programming in Dart.  Give it a .then and a function to call when the data is eventually ready.  Give it a .catchError to process failure arriving rather than good data arriving.

Next up is a discussion of async to turn a function async, and await to make little parts inside it synchronous again.  Supposed to be easier or more clear to read than using Futures.

Error handling in async programming can be done with try/catch blocks or placing a .catchError on a Future.

The chapter repeats a couple times what's required for a working Observer/Stream pattern, which is good because its important.  The first is a sink for events, Sink being the abstract class and mostly using StreamController as the sink.  The second requirement is streams which are properties on the sink.  Final requirement is subscribers / observers / listeners which wait for notification by listening to the stream.  This section has great examples which build up from simple examples to rather complicated multi-subscriber systems using StreamController.broadcast() using an example of a hamburger restaurant.  I especially like how the cartoon illustrations gradually have more complex source code added to them to describe the async operation of the restaurant's kitchen.  Very clear and detailed explanation of the Observer Pattern.

Higher order streams are analogous to higher order functions; streams which return streams.  A bit complicated and even a simple example is a page of dense code.  It would have been nice to arrange the chapter to cover a minimal set of circular dependencies and later cover this complicated feature.  Then again the usual problem that its already a 342 page long book so this topic had to be fit somewhere.

StreamBuilder turns a stream of data into widgets, similar to a template library.  It is a close cousin of the FutureBuilder.  The book glosses over how to use and set it up, probably a page length limitation.  If the book had the luxury of ten additional pages to cover this one page topic, it would probably look like the tutorials I'll link below:

https://www.woolha.com/tutorials/flutter-using-streambuilder-widget-examples

https://api.flutter.dev/flutter/widgets/StreamBuilder-class.html

I think the topic lends itself well to video presentation as seen in the video on the API page above.  I like that video.

Infinitely scrollable lists are made with CustomScrollView.  It wants a source of Slivers and some type of scroll physics.  Sadly the chapter lists scroll physics as a topic to be covered on the very first page of the chapter but its not discussed; the book already has 342 pages... probably got edited out to save space.  Slivers are widgets with a weird lifecycle, they are created and deleted based on if they're in the user's view of the list, which makes them fast and efficient.  Regular widgets would be slower and use more memory.  Begs the question of why Flutter does not make caching decisions automatically for the programmer; why not let the framework decide instead of hard coding it into the source code?

The book unfortunately misses a great chance to compare/contrast a ListView vs the CustomScrollView.  Main differences are ListViews don't sliver and their widgets like Row/Container for arrangement, whereas CustomScrollView does sliver and has its grid passed separate from its slivers.  Supposedly you can do things like appbar animations while scrolling with CustomScrollView although I've never done this myself.  That would likely be too advanced of a topic for the current length of the book.

I think the best way to read this chapter is to close that circularity problem; read it all, get the big picture locked down, THEN research in detail, like my example above of doing a compare and contrast online of ListView vs CustomScrollView or digging in deep for an hour or two of online docs and videos about how to use StreamBuilder.  Async programming is a big topic; someone could probably make an entire book out of going into this chapter in great detail; I'd buy that book.  On a side note I have a pretty good book about async programming on Python so I know its a book length topic...  When "Python Concurrency with asyncio" is officially published in a couple months I might review it.  I've been reading that MEAP book as chapters have been released and its a good book.  Anyway Dart / Flutter could use a similar book which would expand this chapter to book length.  I see the Python async book is already up to 376 pages in pre-publishing which is longer than this entire Flutter book, LOL.

Next up, chapter 10, "Working with data: HTTP, Firestore, and JSON".

Node-RED development

A long time ago, I developed some Node-RED nodes for "STEM" and "Amateur Radio" hardware, written in node.js and Python, under a FOSS license for the general public to use.

A couple months ago I noticed a popular electronic hobbyist magazine linked to my code for the relay board.  Which is really cool, the more people using Node-RED to automate and control using flow based programming technologies, the better.

I wrote that old code for my own personal use; but released it to the general public, may as well let everyone play along.  However, if people are interested, I could apply real resources and real effort toward building better Node-RED nodes.  And people DO seem interested.

I decided to set up a project, Patreon funded, to fund further Node-RED node development for "STEM" and "Amateur Radio" related hardware, mostly on Raspberry Pi type computer hardware.  I intend to produce nodes that follow all the standards in full, are well documented in written and video format, are very easy to use, FOSS licensed, and generally of the highest possible quality.  

And so far, so good!  I've cleaned up the relay board a bit, added several example flows, and recorded some pretty good Youtube documentation videos.

The link to the project main page is:

https://www.springcitysolutions.com/nodered

The link to the project Patreon page is:

https://www.patreon.com/springcitysolutions_nodered

I intend to post detailed daily logs of Node-RED project related work on the Patreon page along with downloads and special posts for Patrons and similar add-ons.

I'll only very occasionally post major news about the project on the main company blog.

Thank You in advance for your support!

Flutter in Action Chapter 8 "Flutter State Management"

Completed taking notes on "Flutter in Action" chapter 8 "Flutter State Management".

https://www.manning.com/books/flutter-in-action

The book lists three closely related sub-chapters, but there's actually four subchapters.

Ch08a "StatefulWidget":

This is a pretty good explanation of the StatefulWidget lifecycle and how the widget tree, element tree, and state object work together over time.  Its clearer than the explanation on api.flutter.dev.

There are two small issues with this chapter.  It glosses over the big picture "why", and it avoids talking about keys and why you need them in lists.

The big picture "why" is that the Flutter element tree is, very simplistically, a cache between the programmer controlled widget tree and the pixels on the screen from the render tree, intended to make it faster, or to use that speed to make it cheaper to manipulate the widget tree as much as the programmer feels like.  So when you change the behavior of the app as the programmer, by changing the widget tree and causing a rebuild, the app renders faster because most of it was unchanged and remains in the element and render trees.  The cache-like nature has its issues; sometimes change flows from the widget tree to the element tree like when something new is created, sometimes change flows from the element tree to the widget tree (kinda) like when something is deleted from the widget tree and the element tree needs to garbage collect and discover its pointing to something that no longer exists and needs deletion.

That deletion situation is the reason for the mysterious keys that are skipped over in this sub-chapter of the book.  For something like a list where somebody might store a little state in the widget (probably a bad idea) like the stereotypical example from the Academind video series of using a random color as a background color for artistic reasons, after a widget tree rebuild the process that reconnects old elements to new widgets can get confused if you delete, for example, the 7th of 10 list items and your state may properly delete the 7th state object, but the remaining widgets might see the 10th background color removed instead of the 7th background colored widget because the re-linker has trouble figuring out which pairs of widgets and elements are now broken by that deletion.  So for that, we can give the widget a unique key to remember forever and share with the element tree and then when it "checks backwards" it can very trivially match keys very quickly.  See, if you were displaying an ebook one line at a time, and there's nothing special about any given line widget, then deleting any old random state object wouldn't matter how the remaining unchanged elements pair up with the identical widgets.

Could add a paragraph around the start of the subchapter something like: Very handwavy, Flutter is fast because it has layers of caches between the programmer generated widgets and the screen pixels, one layer of cache being the Element tree, and caches always have cache coherency problems, so keys are sometimes needed in unusual situations to help old elements find their matching new widgets after the widget tree rebuilds.

Its not really a technical complaint in that if you read the rest of the book this topic makes sense, but in a literary sense I feel technical book chapters should be like self contained essays with some attempt to answer the "why" in each and every chapter.  Yet I can already imagine the authors argument, that would cost a couple pages, and its already a 340 page book.

Ch08b "InheritedWidget"

There's an issue in the Flutter ecosystem where people get confused about small s state as in generic program state vs capital letter S State which is a specific object in Flutter.

There's some leakage between subchapter b and subchapter c where b discusses three state management paradigms and subchapter c discusses what the author considers the most useful "serious" state management paradigm.

One state management paradigm in Flutter is "just kinda pass stuff around to your neighbors and hope for the best".  On very small projects that works great, but is the spaghetti code equivalent of 1980s BASIC having goto statements randomly laying all over.  It often works the first time you write it, but any changes later will cause headaches.  I'm not sure this paradigm has a formal Flutter ecosystem name other than "please don't do this".

A somewhat better state management paradigm is "vanilla lifting state up" which does a disservice to the good name and flavor of vanilla in that it's the second worst way to handle app state.  All state is put in the root or top or peak of the tree and everything that could ever be useful needs to be passed down manually thru every layer then thru to every leaf.  This opens up hilarious debugging problems where there's a problem in layer 12 because layer 6 is intermittently not passing to layer 7, or even worse, layer 6 accidentally on purpose modifies whatever its passing down instead of pushing changes to the top and letting them percolate back down.

The third state management paradigm is the focus of most of chapt 8 subchapter b, which is using Flutter's own InheritedWidget system, which replicates how Theme, Mediaquery, and Scaffolds work.  Essentially instead of complicated business state flowing thru the tree, the complication is focused at the top where there's a three (or four) part process where the child widget gets passed into the container stateful widget, passed into the State object via widget.child, then passed via build method into the private container, then the private container passes it to its superclass, all without touching the actual child widget.  The agony at initial setup seems worth it as now the entire tree can access state by something similar to AppState.of(context).yourMethodHere(); which is relatively clean and its not necessarily global in that you could have multiple "near the top of the tree" states, but then you'll just mix up which data is in which place, so in practice it amounts to a global state solution.  It has a cost in that changing anything with any of the setState methods will cause a rebuild of "all" the StatefulWidget underneath, but that's supposed to be very fast in Flutter and that's why we have the Element Tree as a middle cache to speed it up, etc.  At least its no longer necessary to copy everything correctly from layer 6 to layer 7 to make layer 12 work, layer 12 simply looks its way up the tree until it finds the top and the data it needs.  Everything has a tradeoff.

I have to compliment the author, this "subchapter b" for InheritedWidgets is possibly the clearest explanation I've ever read of how they work and how to implement this state management paradigm / pattern.  Where do I get my $49.99 worth out of this book, well just point right here.

Ch08c "State Mgmt focusing on Google BLoC"

Before getting started with BLoC, the author points out there are sixteen examples of state management alternatives in Flutter at:

https://fluttersamples.com/

So subchapter c is all about the BLoC aka "business logic components".  As a side note, DartConf 2018 is available on youtube, about a quarter of the videos seem to discuss state management in some form, and its all worth watching at:

https://www.youtube.com/playlist?list=PLOU2XLYxmsIIJr3vjxggY7yGcGO7i9BK5

I would describe BLoC theory as separate UI and logic as much as possible and minimize the code in UI and connect the two with streams.

The book explains the good idea of creating a helper class that holds all your BLoCs and then implement something looking kinda like InheritedWidgets where you'll gain access to your BLoCs downstream with something like _someBloc=AppStateContainer.of(context).blocProvider.someBloc;

Actually running streams in and out of the BLoCs is explained in the forth and final subchapter of the book.

Ch08d "Intro to Streams with an emphasis on BLoCs"

This is a short, straightforward, minimal, theory free intro with examples of using streams to implement the flow of data between BLoCs and UI code.  As heard many times thru the book, wait until chapter 9 async, then everything will make more sense.  If you just want data to flow in streams as minimally as possible, these couple pages very clearly explain how to do so, enough to implement BLoC style state management paradigm.

There's about a dozen other ways to do flutter app state management that are not mentioned in the book.  But the book is already 350 pages long, and this chapter is a good intro to the general problem and some simpler solutions.

Next up, chapter 9, "Async Dart and Infinite Scrolling".